/carlaILTrainer

Carla Imitation Learning Trainer

Primary LanguageJupyter NotebookMIT LicenseMIT

carlaILTrainer

Carla Imitation Learning Trainer

The main project belongs to Carla Team, and I just wrote a training system for the main project. Here you can find more information about the original project.

Please check out issue and issue to find out why this code was published.

TODO:

  • Run all branches , but just back propagate on one ( use a mask for that). @merryHunter solved this in the python file.
  • Use the Tensorflow queues for data loading
  • Compatibility fix for carla checkpoint loading
  • Fix the loss function (Use only one loss)
  • Balance the dataset according to labels in training mode
  • Test the whole system with the recent benchmark

Contribution:

Please feel free to pull a request. Any features and any changes are welcome.

All Contributors:

Paper Results:

This is an auto table generator for the paper writing purpose. Tables are based on the CoRL 2017 Carla Team Paper. This can help me to have a gorgeous table easier and with a lot of fewer efforts. Currently, Text, LaTex, and HTML are supported. Before you are going to run this code, you have to prepare the results for three different runs for each town in Carla. You can see an example in the following passage.

Output Support: Text, Latex, HTML

Example: python calc_CORL2017_Tables.py --path "./CarlaPaper_ReExperiment/" -v -n "CoRL-2017 Carla Paper"

"./CarlaPaper_ReExperiment/" contains the following folders:

CarlaPaper_ReExperiment/
├── CarlaPaperModel_Test01_CoRL2017_Town01
│   ├── log_201807010542
│   ├── log_201807011830
│   ├── log_201807020224
│   ├── measurements.csv
│   ├── metrics.json
│   ├── res
│   └── summary.csv
├── CarlaPaperModel_Test01_CoRL2017_Town02
│   ├── log_201807031155
│   ├── log_201807032256
│   ├── log_201807050552
│   ├── measurements.csv
│   └── summary.csv
├── CarlaPaperModel_Test02_CoRL2017_Town01
│   ├── log_201807020250
│   ├── log_201807020739
│   ├── log_201807021521
│   ├── measurements.csv
│   └── summary.csv
├── CarlaPaperModel_Test02_CoRL2017_Town02
│   ├── log_201807050555
│   ├── measurements.csv
│   └── summary.csv
├── CarlaPaperModel_Test03_CoRL2017_Town01
│   ├── log_201807022050
│   ├── log_201807030243
│   ├── log_201807030738
│   ├── measurements.csv
│   └── summary.csv
├── CarlaPaperModel_Test03_CoRL2017_Town02
│   ├── log_201807052015
│   ├── log_201807060744
│   ├── measurements.csv
│   └── summary.csv
├── results.html
└── results.laTex

P.S: results.html and results.laTex are produced by this code after execution. These are a little bit different because of different configuration and different Carla version. In these results, bikes were in the test only. Please share your code and enhancements by a pull request with the world if you use this in your papers and add a feature.

Example Output:

MODEL: CoRL-2017 Carla Paper_ReExperiment

Tasks Training Conditions New Town New Weather New Town & Weather
Straight 0.98 +/- 0.0 0.97 +/- 0.0 0.97 +/- 0.010.95 +/- 0.0
One Turn 0.94 +/- 0.01 0.68 +/- 0.010.94 +/- 0.020.7 +/- 0.0
Navigation 0.89 +/- 0.0 0.42 +/- 0.0 0.83 +/- 0.010.46 +/- 0.02
Nav. Dynamic0.88 +/- 0.0 0.42 +/- 0.020.8 +/- 0.03 0.46 +/- 0.03

Infractions Training Conditions New Town New Weather New Town & Weather
Opposite Lane 16.78 +/- 0.04 2.89 +/- 0.0 8.13 +/- 0.133.89 +/- 0.5
Sidewalk 7.46 +/- 1.32 2.65 +/- 0.348.13 +/- 0.132.26 +/- 0.2
Collision-static 16.78 +/- 0.04 8.67 +/- 0.0 8.13 +/- 0.1312.73 +/- 0.01
Collision-car 16.78 +/- 0.04 8.67 +/- 0.0 8.13 +/- 0.1312.73 +/- 0.01
Collision-pedestrian16.78 +/- 0.04 8.67 +/- 0.0 8.13 +/- 0.1312.73 +/- 0.01

Infractions Training Conditions New Town New Weather New Town & Weather
Opposite Lane 13.09 +/- 2.68 1.03 +/- 0.11 16.86 +/- 0.091.35 +/- 0.1
Sidewalk 11.22 +/- 0.1 0.9 +/- 0.05 16.86 +/- 0.090.95 +/- 0.04
Collision-static 33.66 +/- 0.3 17.68 +/- 0.2616.86 +/- 0.0926.88 +/- 0.7
Collision-car 33.66 +/- 0.3 17.68 +/- 0.2616.86 +/- 0.0926.88 +/- 0.7
Collision-pedestrian33.66 +/- 0.3 17.68 +/- 0.2616.86 +/- 0.0926.88 +/- 0.7

Infractions Training Conditions New Town New Weather New Town & Weather
Opposite Lane 17.4 +/- 3.7 2.26 +/- 0.0523.47 +/- 8.473.12 +/- 0.17
Sidewalk 16.27 +/- 4.14 0.62 +/- 0.0312.71 +/- 3.750.68 +/- 0.03
Collision-static 66.63 +/- 0.2 24.88 +/- 0.535.12 +/- 0.2840.43 +/- 0.45
Collision-car 66.63 +/- 0.2 24.88 +/- 0.535.12 +/- 0.2840.43 +/- 0.45
Collision-pedestrian66.63 +/- 0.2 24.88 +/- 0.535.12 +/- 0.2840.43 +/- 0.45

Infractions Training Conditions New Town New Weather New Town & Weather
Opposite Lane 20.17 +/- 2.74 1.5 +/- 0.18 24.95 +/- 12.951.66 +/- 0.04
Sidewalk 13.05 +/- 2.98 0.65 +/- 0.064.26 +/- 0.87 0.68 +/- 0.07
Collision-static 4.51 +/- 1.07 0.38 +/- 0.052.15 +/- 0.55 0.41 +/- 0.05
Collision-car 1.19 +/- 0.14 0.28 +/- 0.031.48 +/- 0.28 0.27 +/- 0.02
Collision-pedestrian14.14 +/- 5.64 1.66 +/- 0.157.35 +/- 0.97 1.83 +/- 0.14

Number of Infractions Training Conditions New Town New Weather New Town & Weather
Opposite Lane 0.0 +/- 0.0 3.0 +/- 0.0 0.0 +/- 0.0 3.33 +/- 0.47
Sidewalk 2.33 +/- 0.47 3.33 +/- 0.470.0 +/- 0.0 5.67 +/- 0.47
Collision-static 0.0 +/- 0.0 0.0 +/- 0.0 0.0 +/- 0.0 0.0 +/- 0.0
Collision-car 0.0 +/- 0.0 0.0 +/- 0.0 0.0 +/- 0.0 0.0 +/- 0.0
Collision-pedestrian 0.0 +/- 0.0 0.0 +/- 0.0 0.0 +/- 0.0 0.0 +/- 0.0

Number of Infractions Training Conditions New Town New Weather New Town & Weather
Opposite Lane 2.67 +/- 0.47 17.33 +/- 1.7 0.33 +/- 0.4720.0 +/- 1.41
Sidewalk 3.0 +/- 0.0 19.67 +/- 1.250.33 +/- 0.4728.33 +/- 0.94
Collision-static 0.0 +/- 0.0 0.0 +/- 0.0 0.0 +/- 0.0 0.0 +/- 0.0
Collision-car 0.0 +/- 0.0 0.0 +/- 0.0 0.0 +/- 0.0 0.0 +/- 0.0
Collision-pedestrian 0.0 +/- 0.0 0.0 +/- 0.0 0.0 +/- 0.0 0.0 +/- 0.0

Number of Infractions Training Conditions New Town New Weather New Town & Weather
Opposite Lane 4.0 +/- 0.82 11.0 +/- 0.0 1.67 +/- 0.4713.0 +/- 0.82
Sidewalk 4.33 +/- 0.94 40.33 +/- 1.253.0 +/- 0.82 59.33 +/- 1.7
Collision-static 0.0 +/- 0.0 0.0 +/- 0.0 0.0 +/- 0.0 0.0 +/- 0.0
Collision-car 0.0 +/- 0.0 0.0 +/- 0.0 0.0 +/- 0.0 0.0 +/- 0.0
Collision-pedestrian 0.0 +/- 0.0 0.0 +/- 0.0 0.0 +/- 0.0 0.0 +/- 0.0

Number of Infractions Training Conditions New Town New Weather New Town & Weather
Opposite Lane 3.33 +/- 0.47 16.67 +/- 1.89 2.33 +/- 1.8922.67 +/- 0.47
Sidewalk 5.33 +/- 1.25 38.33 +/- 3.4 8.33 +/- 1.8956.33 +/- 6.85
Collision-static 15.33 +/- 3.09 67.33 +/- 10.4 17.0 +/- 4.9794.0 +/- 13.49
Collision-car 56.0 +/- 6.38 90.33 +/- 11.6724.0 +/- 5.72139.67 +/- 8.73
Collision-pedestrian 5.33 +/- 1.7 15.0 +/- 1.41 4.67 +/- 0.4720.67 +/- 1.7