For OpenDRIVE related scripts and tutorial, visit OpenDRIVE output.
Introduction
This project presents a programming interface and automated testing infrastructure for closed-loop autonomous vehicle simulation tests. For more information, please refer to our paper.
Setting up the project
A Windows machine is recommended, though this project will also run on an Ubuntu machine. Download and install Unity (tested to work on Unity 2019.2). Open the project as a Unity project. There are two scenes of interest:
-
Assets\Scenes\ProgrammingSampleUdacityTrainingis the training scene, which can be used to generate training data.0 -
Assets\Scenes\ProgrammingSampleUdacityAutonomousis the autonomous scene where the autonomous vehicle can be controlled by an external program. Online events are recorded. The scenes have all relevant scripts attached to objects contained therein. An important object is theScriptPlaceHolder. This contains the environment program that is loaded, which includes the set-up of the environment, test variables and actors.
Project Structure
- The entire folder can be opened in Unity as a Unity project.
- The
Assetsfolder contains the scripts, scenes and relevant 3D resources. -
Traffic Systemincludes prefabs for road and traffic building. -
Pedestrian Systemincludes prefabs for building a pedestrian system. - The
Udacityfolder contains relevant scripts and prefabs from Udacity's self driving simulator project.
Usage Guide
Initial set-up.
Open the project in Unity (you will need to clone the entire project at https://gitlab.mpi-sws.org/mathur/ProgrammingDMEnvironment).
Open the scene ProgrammingSampleUdacityAutonomous (located in Assets/Scenes).
In the scene, click on the ScriptPlaceHolder object.
Then, in the inspector, you will have many scripts that control the environment, but especially note:
-
Visualize: the support that generates the desired test/scenario -
StraightRoad: specifies the desired road network and test TheStraightRoadshould have the box by the name checked.
This image shows the Unity editor, with key features highlighted in green.
If you want to run a different road network script, first remove the StraightRoad script by right-clicking and selecting the Remove Component option.
To add a new script, simply drag it over to the inspector.
Some example scripts are provided in https://gitlab.mpi-sws.org/mathur/ProgrammingDMEnvironment/tree/master/Assets/Scripts/Program/Examples .
Creating Road Environments
Creating a simple straight road segments
The following example creates a straight road of variable number of lanes and length, then adds an autonomous vehicle with a collision monitor.
public class StraightRoad : EnvironmentProgramBaseClass
{
//Add the ProDM vars here
VarInterval<int> length;
VarEnum<int> numLanes;
//Environment variables
VarInterval<float> lightIntensity;
ProDMColor color;
VarInterval<float> ambientIntensity;
VarInterval<float> fogDensity;
void Start() //needs to be called in the start because the setup in the RoadElementVisualization happens in the Awake()
{
fogDensity = new VarInterval<float>(val: 0, min: 0, max: 0.3f, desc: "FOG_DENSITY");
lightIntensity = new VarInterval<float>(val: 1, min: 0, max: 10);
color = new ProDMColor(new VarInterval<float>(val: 1f, min: 0f, max: 1f), new VarInterval<float>(val: 0.9f, min: 0f, max: 1f),
new VarInterval<float>(val: 0.8f, min: 0f, max: 1f), 1f);
ambientIntensity = new VarInterval<float>(val: 1.5f, min: 0f, max: 2.5f, desc: "AMBIENT_LIGHT_INTENSITY");
//define the ProDM variables
length = new VarInterval<int>(val: 100, changeable: true, min: 20, max: 4000);
List<int> numLanesOptions = new List<int> { 2, 4, 6 };
numLanes = new VarEnum<int>(numLanesOptions, 0);
Test();
}
//create the visualization of the road network
//NOTE: The code to make the road network should go here.
public override void MakeView()
{
//create list of items to be visualized later
List<object> list = new List<object>();
//create a straight road segment
RoadSegment cRoad1 = new RoadSegment(length, numLanes, nameof(cRoad1), Orientation.VERTICAL);
list.Add(cRoad1);
//place autonomous car on the road segment, and attach a collision monior
AutonomousVehicle autonomouscar = new AutonomousVehicle(nameof(autonomouscar), cRoad1, 0.11f, LANE.PLUS_ONE, type: TestActorType.EGO_VEHICLE_AUTONOMOUS, monitor: "CollisionMonitor");
list.Add(autonomouscar);
//visualize the scene
PlaceHolderForVisualizer.Visualize(list);
}
public override void MakeEnvironment()
{
//Environment settings
LightSettings light = new LightSettings(intensity: lightIntensity, direction: Vector3.down, color: color, ambientIntensity: ambientIntensity);
FogSettings fog = new FogSettings(density: fogDensity);
EnvironmentSettings env = new EnvironmentSettings(light, fog);
PlaceHolderForVisualizer.Visualize(env);
}
}
The output is as:
We provide the option to add trees or residential zoning to different sides of the road and corners of the intersections. This line of code modifies the straight road to have trees on either side:
RoadSegment cRoad1 = new RoadSegment(length, numLanes, nameof(cRoad1), Orientation.VERTICAL, zone1: Zoning.TREES, zone2: Zoning.TREES);Making more complicated networks
The following example creates a more complicated road network with intersections and AI vehicles.
public class ConnectedRoads : EnvironmentProgramBaseClass
{
//Add the ProDM vars here
VarInterval<int> length;
VarEnum<int> numLanes;
//Environment variables
VarInterval<float> lightIntensity;
ProDMColor color;
VarInterval<float> ambientIntensity;
VarInterval<float> fogDensity;
void Start() //needs to be called in the start because the setup in the RoadElementVisualization happens in the Awake()
{
fogDensity = new VarInterval<float>(val: 0, min: 0, max: 0.3f, desc: "FOG_DENSITY");
lightIntensity = new VarInterval<float>(val: 1, min: 0, max: 10);
color = new ProDMColor(new VarInterval<float>(val: 1f, min: 0f, max: 1f), new VarInterval<float>(val: 0.9f, min: 0f, max: 1f),
new VarInterval<float>(val: 0.8f, min: 0f, max: 1f), 1f);
ambientIntensity = new VarInterval<float>(val: 1.5f, min: 0f, max: 2.5f, desc: "AMBIENT_LIGHT_INTENSITY");
//define the ProDM variables
length = new VarInterval<int>(val: 20, changeable: true, min: 10, max: 40);
List<int> numLanesOptions = new List<int> { 2, 4, 6 };
numLanes = new VarEnum<int>(numLanesOptions, 0);
Test();
}
//create the visualization of the road network
//NOTE: The code to make the road network should go here.
public override void MakeView()
{
// make the T-intersection
TIntersection int1 = new TIntersection(numLanes: numLanes, zone1: Zoning.TREES, zone2: Zoning.TREES, zone3: Zoning.RESIDENTIAL_ZONE);
PlaceHolderForVisualizer.Visualize(int1);
// make straight roads to connect to the Intersection
RoadSegment cRoad1 = new RoadSegment(lengthOfRoad: length, numLanes: numLanes, zone1: Zoning.RESIDENTIAL_ZONE, zone2: Zoning.TREES);
RoadSegment cRoad3 = new RoadSegment(lengthOfRoad: length, numLanes: numLanes, zone1: Zoning.RESIDENTIAL_ZONE, zone2: Zoning.TREES);
PlaceHolderForVisualizer.Visualize(cRoad1);
PlaceHolderForVisualizer.Visualize(cRoad3);
// add straight roads to a list
List<RoadElement> cRoadList = new List<RoadElement>
{
cRoad1,
cRoad3
};
// make an encapsulating object of straight roads
RoadElement cRoads = new RoadElement(cRoadList);
PlaceHolderForVisualizer.Visualize(cRoads);
// Connect the encapsulated object to T-Interesection
// first specify the connection mapping
Dictionary<PhysicalConnection, PhysicalConnection> connMap1 = new Dictionary<PhysicalConnection, PhysicalConnection>
{
{ cRoad1._connections.AtIndex(0), int1._connections.AtIndex(0) },
{ cRoad3._connections.AtIndex(1), int1._connections.AtIndex(2) }
};
// then make the connection
RoadElement finalconnection = cRoads.ConnectTo(r2: int1, connectionMap: connMap1);
PlaceHolderForVisualizer.Visualize(finalconnection);
PlaceHolderForVisualizer.connectRoadElements(connMap1, cRoads, finalconnection, int1, cRoads, true);
//place autonomous vehicle and attach collision monitor
AutonomousVehicle autonomouscar = new AutonomousVehicle(nameof(autonomouscar), cRoad1, 0.11f, LANE.PLUS_ONE, monitor: "CollisionMonitor", type: TestActorType.EGO_VEHICLE_AUTONOMOUS);
PlaceHolderForVisualizer.Visualize(autonomouscar);
//place AI vehicles in the scene
AIVehicle car1 = new AIVehicle(nameof(car1), cRoad3, 0.15f, LANE.MINUS_ONE, Color.blue);
PlaceHolderForVisualizer.Visualize(car1);
AIVehicle car2 = new AIVehicle(nameof(car2), cRoad3, 0.36f, LANE.PLUS_ONE, Color.blue);
PlaceHolderForVisualizer.Visualize(car2);
AIVehicle car3 = new AIVehicle(nameof(car3), cRoad1, 0.3f, LANE.PLUS_ONE, Color.blue);
PlaceHolderForVisualizer.Visualize(car3);
AIVehicle car4 = new AIVehicle(nameof(car4), cRoad1, 0.7f, LANE.MINUS_ONE, Color.blue);
PlaceHolderForVisualizer.Visualize(car4);
AIVehicle car5 = new AIVehicle(nameof(car5), cRoad1, 0.5f, LANE.PLUS_ONE, Color.blue);
PlaceHolderForVisualizer.Visualize(car5);
AIVehicle car6 = new AIVehicle(nameof(car6), cRoad1, 0.5f, LANE.MINUS_ONE, Color.blue);
PlaceHolderForVisualizer.Visualize(car6);
}
public override void MakeEnvironment()
{
//Environment settings
LightSettings light = new LightSettings(intensity: lightIntensity, direction: Vector3.down, color: color, ambientIntensity: ambientIntensity);
FogSettings fog = new FogSettings(density: fogDensity);
EnvironmentSettings env = new EnvironmentSettings(light, fog);
PlaceHolderForVisualizer.Visualize(env);
}
}The output is as:
Further complicated scenes can be found in the Assets\Scripts\Programs\RoadSegmentation folder.
Environment Settings
Environment settings control the light intensity, fog, etc. of the simulation environment.
Adding Test Actors
Autonomous Vehicle
This test actor can be controlled manually (A-S-W-D keys on keyboard) or by connecting some autonomous controller.
To use the training scene, ensure the type is TestActorType.EGO_VEHICLE_TRAINING and for the test scene, the type should be TestActorType.EGO_VEHICLE_AUTONOMOUS
The collision monitor for the autonomous vehicle is located in https://gitlab.mpi-sws.org/mathur/ProgrammingDMEnvironment/tree/master/Assets/Scripts/Presenter/LiveChecks/Monitors
Pedestrians
Pedestrians currently come in one design. Behaviors are setup to allow the pedestrian to cross the road after a specified time, or when a specified actor of interest comes within a certain euclidian distance. These behaviors are located in https://gitlab.mpi-sws.org/mathur/ProgrammingDMEnvironment/tree/master/Assets/Scripts/Presenter/LiveChecks/Behaviors
The following line of code creates a pedestrian that crosses the road after a specified time:
Pedestrian pedestrian1 = new Pedestrian(nameof(pedestrian1), cRoad1, 0.2f, timeDelay: 5f, walkingSpeed: walkingSpeed, behavior: "PedestrianCrossingDelay");The following line of code creates a pedestrian that crosses the road when a test actor (in this case, the autonomous car) comes within a certain Euclidian distance:
Pedestrian pedestrian1 = new Pedestrian(nameof(pedestrian1), cRoad1, normalizedDistance:distance, actorOfInterest:autonomouscar, euclideanDistance: euclideanDist, walkingSpeed:walkingSpeed, behavior: "PedestrianCrossingBehavior01");AI Vehicles
AI vehicles are other traffic vehicles. They have their own built-in behaviors as a default, although more complex behaviors can be specified.
The model of AI vehicle can be altered by specifying in its instantiation:
AIVehicle car1 = new AIVehicle(nameof(car1), cRoad1, 0.15f, LANE.MINUS_ONE, Color.blue, model: "AICar5");Generating Tests
Timed Test
Call the following function to run a test that lasts for 100 seconds (one iteration):
Test(timeCutOff: 100f);Iterative Tests
The following program creates an iterative test. Explanations of all of the modes of iterative testing are further described below.
public class RandomSampling_100iterations : EnvironmentProgramBaseClass
{
//Add the ProDM vars here
VarInterval<int> cr_length;
VarEnum<int> cr_numLanes;
VarInterval<float> lightIntensity;
ProDMColor color;
VarInterval<float> ambientIntensity;
VarInterval<float> fogDensity;
// Parameterize using two ProDMFloats
VarInterval<float> walkingSpeed;
VarInterval<float> euclideanDist;
void Start() //needs to be called in the start because the setup in the RoadElementVisualization happens in the Awake()
{
fogDensity = new VarInterval<float>(val: 0, min: 0, max: 0.3f, desc: "FOG_DENSITY");
lightIntensity = new VarInterval<float>(val: 1, min: 0, max: 10);
color = new ProDMColor(new VarInterval<float>(val: 1f, min: 0f, max: 1f), new VarInterval<float>(val: 0.9f, min: 0f, max: 1f),
new VarInterval<float>(val: 0.8f, min: 0f, max: 1f), 1f);
//PlaceHolderForVisualizer.Visualize(nameof(color), color);
ambientIntensity = new VarInterval<float>(val: 1.5f, min: 0f, max: 2.5f, desc: "AMBIENT_LIGHT_INTENSITY");
//define the ProDM variables
cr_length = new VarInterval<int>(val: 100, changeable: true, min: 20, max: 4000);
List<int> numLanesOptions = new List<int> { 2, 4 };
cr_numLanes = new VarEnum<int>(numLanesOptions, 0);
walkingSpeed = new VarInterval<float>(val: 4f, min: 2f, max: 10f);
euclideanDist = new VarInterval<float>(val: 40f, min: 30f, max: 60f);
List<VarInterval<float>> rangedFloats = new List<VarInterval<float>>();
rangedFloats.Add(walkingSpeed);
rangedFloats.Add(euclideanDist);
TestIterative(rangedFloats:rangedFloats, iterations: 100, singleTestTime:15f, setVar: AutomatedVarSetMethod.RANDOM);
}
//create the visualization of the road network
//NOTE: The code to make the road network should go here.
public override void MakeView()
{
List<object> list = new List<object>();
RoadSegment cRoad1 = new RoadSegment(cr_length, cr_numLanes, orientation: Orientation.VERTICAL);
cRoad1._zone1 = Zoning.TREES;
cRoad1._zone2 = Zoning.TREES;
list.Add(cRoad1);
AutonomousVehicle autonomouscar = new AutonomousVehicle(nameof(autonomouscar), cRoad1, 0.11f, LANE.PLUS_ONE, monitor: "CollisionMonitor", type: TestActorType.EGO_VEHICLE_AUTONOMOUS);
list.Add(autonomouscar);
Pedestrian pedestrian = new Pedestrian(nameof(pedestrian), cRoad1, 0.2f, actorOfInterest: autonomouscar, euclideanDistance: euclideanDist, walkingSpeed: walkingSpeed, behavior: "PedestrianCrossingBehavior01");
list.Add(pedestrian);
PlaceHolderForVisualizer.Visualize(list);
}
public override void MakeEnvironment()
{
LightSettings light = new LightSettings(intensity: lightIntensity, direction: Vector3.down, color: color, ambientIntensity: ambientIntensity);
FogSettings fog = new FogSettings(density: fogDensity);
EnvironmentSettings env = new EnvironmentSettings(light, fog);
PlaceHolderForVisualizer.Visualize(env);
}
}
Sampling
The following code will set up 100 test iterations that each last 15 seconds. rangedFloats contains the parameterized variables that will be tested.
TestIterative(rangedFloats: rangedFloats, iterations: 100, singleTestTime: 15f, setVar: AutomatedVarSetMethod.RANDOM);Using AutomatedVarSetMethod.RANDOM sets each rangedFloat randomly, while using AutomatedVarSetMethod.LOW_DISCREPENCY will use the Halton sequence to select the rangedFloat values for each iteration;
Energy Maximization
Energy maximization will find teh maximum collision energy using a simulated annealing algorithm to test the rangedFloats. The sample size dictates how many numbers from the range of each rangedFloat value to test, and the repetitions says how many times the simulated annealing algorithm will run. In this case, 20 samples will be selected, with 5 rounds of simulated annealing performed, for a total of 100 iterations total, with each test lasting 15 seconds.
TestIterative(testVars: rangedFloats, repetitions: 5, sampleSize: 20, singleTestTime: 15f, actorMonitorName: "CollisionMonitor", setVar: AutomatedVarSetMethod.OPTIMIZE_ENERGY_HALTON);Using AutomatedVarSetMethod.OPTIMIZE_ENERGY_RANDOM allows for random selection of each sample, while using AutomatedVarSetMethod.OPTIMIZE_ENERGY_HALTON will use the Halton sequence to select each sample;
A variant of the energy maximization uses the modes OPTIMIZE_ENERGY_HALTON_MAX and OPTIMIZE_ENERGY_RANDOM_MAX. This mode will first sample an initial n samples, then select the k highest energies and perform m repetitions of simulated annealing on each of the k samples.
For example, the following code will first take 80 samples, choose the top 5, and ask for 4 repetitions of each, for 100 total test iterations.
TestIterative(testVars: rangedFloats, repetitions: 4, sampleSize: 80, numTopEnergies: 5 singleTestTime: 15f, actorMonitorName: "CollisionMonitor", setVar: AutomatedVarSetMethod.OPTIMIZE_ENERGY_HALTON_MAX);For all of these tests, you must specify the name of the monitor script that should be maximized. All of the above examples use the "CollisionMonitor" that is attached to the autonomous vehicle.
External packages used
This project uses:
- Road & Traffic System
- UniRx
- Udacity's self driving simulator
- Car driving behavior cloning. This project can be easily used for training and running the autonomous vehicle.