carlaILTrainer

Carla Imitation Learning Trainer

The main project belongs to Carla Team, and I just wrote a training system for the main project. Here you can find more information about the original project.

Please check out issue and issue to find out why this code was published.

TODO:

Run all branches , but just back propagate on one ( use a mask for that). @merryHunter solved this in the python file.
Use the Tensorflow queues for data loading
Compatibility fix for carla checkpoint loading
Fix the loss function (Use only one loss)
Balance the dataset according to labels in training mode
Test the whole system with the recent benchmark

Contribution:

Please feel free to pull a request. Any features and any changes are welcome.

All Contributors:

Mojtaba Va'lipour: @mvpcom
Ivan Chernuha : @merryHunter

Paper Results:

This is an auto table generator for the paper writing purpose. Tables are based on the CoRL 2017 Carla Team Paper. This can help me to have a gorgeous table easier and with a lot of fewer efforts. Currently, Text, LaTex, and HTML are supported. Before you are going to run this code, you have to prepare the results for three different runs for each town in Carla. You can see an example in the following passage.

Output Support: Text, Latex, HTML

Example: python calc_CORL2017_Tables.py --path "./CarlaPaper_ReExperiment/" -v -n "CoRL-2017 Carla Paper"

"./CarlaPaper_ReExperiment/" contains the following folders:

CarlaPaper_ReExperiment/
├── CarlaPaperModel_Test01_CoRL2017_Town01
│   ├── log_201807010542
│   ├── log_201807011830
│   ├── log_201807020224
│   ├── measurements.csv
│   ├── metrics.json
│   ├── res
│   └── summary.csv
├── CarlaPaperModel_Test01_CoRL2017_Town02
│   ├── log_201807031155
│   ├── log_201807032256
│   ├── log_201807050552
│   ├── measurements.csv
│   └── summary.csv
├── CarlaPaperModel_Test02_CoRL2017_Town01
│   ├── log_201807020250
│   ├── log_201807020739
│   ├── log_201807021521
│   ├── measurements.csv
│   └── summary.csv
├── CarlaPaperModel_Test02_CoRL2017_Town02
│   ├── log_201807050555
│   ├── measurements.csv
│   └── summary.csv
├── CarlaPaperModel_Test03_CoRL2017_Town01
│   ├── log_201807022050
│   ├── log_201807030243
│   ├── log_201807030738
│   ├── measurements.csv
│   └── summary.csv
├── CarlaPaperModel_Test03_CoRL2017_Town02
│   ├── log_201807052015
│   ├── log_201807060744
│   ├── measurements.csv
│   └── summary.csv
├── results.html
└── results.laTex

P.S: results.html and results.laTex are produced by this code after execution. These are a little bit different because of different configuration and different Carla version. In these results, bikes were in the test only. Please share your code and enhancements by a pull request with the world if you use this in your papers and add a feature.

Example Output:

MODEL: CoRL-2017 Carla Paper_ReExperiment

Tasks	Training Conditions	New Town	New Weather	New Town & Weather
Straight	0.98 +/- 0.0	0.97 +/- 0.0	0.97 +/- 0.01	0.95 +/- 0.0
One Turn	0.94 +/- 0.01	0.68 +/- 0.01	0.94 +/- 0.02	0.7 +/- 0.0
Navigation	0.89 +/- 0.0	0.42 +/- 0.0	0.83 +/- 0.01	0.46 +/- 0.02
Nav. Dynamic	0.88 +/- 0.0	0.42 +/- 0.02	0.8 +/- 0.03	0.46 +/- 0.03

Infractions	Training Conditions	New Town	New Weather	New Town & Weather
Opposite Lane	16.78 +/- 0.04	2.89 +/- 0.0	8.13 +/- 0.13	3.89 +/- 0.5
Sidewalk	7.46 +/- 1.32	2.65 +/- 0.34	8.13 +/- 0.13	2.26 +/- 0.2
Collision-static	16.78 +/- 0.04	8.67 +/- 0.0	8.13 +/- 0.13	12.73 +/- 0.01
Collision-car	16.78 +/- 0.04	8.67 +/- 0.0	8.13 +/- 0.13	12.73 +/- 0.01
Collision-pedestrian	16.78 +/- 0.04	8.67 +/- 0.0	8.13 +/- 0.13	12.73 +/- 0.01

Infractions	Training Conditions	New Town	New Weather	New Town & Weather
Opposite Lane	13.09 +/- 2.68	1.03 +/- 0.11	16.86 +/- 0.09	1.35 +/- 0.1
Sidewalk	11.22 +/- 0.1	0.9 +/- 0.05	16.86 +/- 0.09	0.95 +/- 0.04
Collision-static	33.66 +/- 0.3	17.68 +/- 0.26	16.86 +/- 0.09	26.88 +/- 0.7
Collision-car	33.66 +/- 0.3	17.68 +/- 0.26	16.86 +/- 0.09	26.88 +/- 0.7
Collision-pedestrian	33.66 +/- 0.3	17.68 +/- 0.26	16.86 +/- 0.09	26.88 +/- 0.7

Infractions	Training Conditions	New Town	New Weather	New Town & Weather
Opposite Lane	17.4 +/- 3.7	2.26 +/- 0.05	23.47 +/- 8.47	3.12 +/- 0.17
Sidewalk	16.27 +/- 4.14	0.62 +/- 0.03	12.71 +/- 3.75	0.68 +/- 0.03
Collision-static	66.63 +/- 0.2	24.88 +/- 0.5	35.12 +/- 0.28	40.43 +/- 0.45
Collision-car	66.63 +/- 0.2	24.88 +/- 0.5	35.12 +/- 0.28	40.43 +/- 0.45
Collision-pedestrian	66.63 +/- 0.2	24.88 +/- 0.5	35.12 +/- 0.28	40.43 +/- 0.45

Infractions	Training Conditions	New Town	New Weather	New Town & Weather
Opposite Lane	20.17 +/- 2.74	1.5 +/- 0.18	24.95 +/- 12.95	1.66 +/- 0.04
Sidewalk	13.05 +/- 2.98	0.65 +/- 0.06	4.26 +/- 0.87	0.68 +/- 0.07
Collision-static	4.51 +/- 1.07	0.38 +/- 0.05	2.15 +/- 0.55	0.41 +/- 0.05
Collision-car	1.19 +/- 0.14	0.28 +/- 0.03	1.48 +/- 0.28	0.27 +/- 0.02
Collision-pedestrian	14.14 +/- 5.64	1.66 +/- 0.15	7.35 +/- 0.97	1.83 +/- 0.14

Number of Infractions	Training Conditions	New Town	New Weather	New Town & Weather
Opposite Lane	0.0 +/- 0.0	3.0 +/- 0.0	0.0 +/- 0.0	3.33 +/- 0.47
Sidewalk	2.33 +/- 0.47	3.33 +/- 0.47	0.0 +/- 0.0	5.67 +/- 0.47
Collision-static	0.0 +/- 0.0	0.0 +/- 0.0	0.0 +/- 0.0	0.0 +/- 0.0
Collision-car	0.0 +/- 0.0	0.0 +/- 0.0	0.0 +/- 0.0	0.0 +/- 0.0
Collision-pedestrian	0.0 +/- 0.0	0.0 +/- 0.0	0.0 +/- 0.0	0.0 +/- 0.0

Number of Infractions	Training Conditions	New Town	New Weather	New Town & Weather
Opposite Lane	2.67 +/- 0.47	17.33 +/- 1.7	0.33 +/- 0.47	20.0 +/- 1.41
Sidewalk	3.0 +/- 0.0	19.67 +/- 1.25	0.33 +/- 0.47	28.33 +/- 0.94
Collision-static	0.0 +/- 0.0	0.0 +/- 0.0	0.0 +/- 0.0	0.0 +/- 0.0
Collision-car	0.0 +/- 0.0	0.0 +/- 0.0	0.0 +/- 0.0	0.0 +/- 0.0
Collision-pedestrian	0.0 +/- 0.0	0.0 +/- 0.0	0.0 +/- 0.0	0.0 +/- 0.0

Number of Infractions	Training Conditions	New Town	New Weather	New Town & Weather
Opposite Lane	4.0 +/- 0.82	11.0 +/- 0.0	1.67 +/- 0.47	13.0 +/- 0.82
Sidewalk	4.33 +/- 0.94	40.33 +/- 1.25	3.0 +/- 0.82	59.33 +/- 1.7
Collision-static	0.0 +/- 0.0	0.0 +/- 0.0	0.0 +/- 0.0	0.0 +/- 0.0
Collision-car	0.0 +/- 0.0	0.0 +/- 0.0	0.0 +/- 0.0	0.0 +/- 0.0
Collision-pedestrian	0.0 +/- 0.0	0.0 +/- 0.0	0.0 +/- 0.0	0.0 +/- 0.0

Number of Infractions	Training Conditions	New Town	New Weather	New Town & Weather
Opposite Lane	3.33 +/- 0.47	16.67 +/- 1.89	2.33 +/- 1.89	22.67 +/- 0.47
Sidewalk	5.33 +/- 1.25	38.33 +/- 3.4	8.33 +/- 1.89	56.33 +/- 6.85
Collision-static	15.33 +/- 3.09	67.33 +/- 10.4	17.0 +/- 4.97	94.0 +/- 13.49
Collision-car	56.0 +/- 6.38	90.33 +/- 11.67	24.0 +/- 5.72	139.67 +/- 8.73
Collision-pedestrian	5.33 +/- 1.7	15.0 +/- 1.41	4.67 +/- 0.47	20.67 +/- 1.7