Scoring software used in the IceVision competition.
For ground truth we use exactly the same format as in the annotations repository.
For solutions we use a slightly different TSV (tab-separated values) file format. It contains all detected traffic signs of given classes on provided frame sequences. A solution file must contain header and the following fields:
frame: sequence + frame number, e.g.2018-02-13_1418_left/000032.xtl,ytl,xbr,ybr: bounding box coordinates, integer or float. Note:xtlmust be bigger thanxbrandytlmust be bigger thanybrclass: traffic sign code. Valid values are:2.1,2.4,3.1,3.24,3.27,4.1,4.2,5.19,5.20,8.22.
Some examples can be found in examples/ folder.
During online stage participants have to detect the following traffic signs:
Click to expand table
| Code | Image | Description |
|---|---|---|
| 2.1 |  |
Main road |
| 2.4 |  |
Yield road |
| 3.1 |  |
No entry |
| 3.24 |  |
Maximum speed limit |
| 3.27 |  |
No stopping |
| 4.1 |       |
Proceed in the given direction |
| 4.2 |    |
Pass on the given side |
| 5.19 |   |
Pedestrian crossing |
| 5.20 |  |
Road bump |
| 8.22 |    |
Obstacle |
Traffic sign images: © Wikimedia Commons Contributors / CC-BY-SA-3.0
Bounding boxes with an area smaller than 100 pixels are ignored during evaluation. Detection is considered successful if IoU is bigger or equal to 0.5 and bounding box has a correct class code. If sign is detected twice, then detection with a smallest IoU will be counted as a false positive. Each false positive or incorrect detection results in penalty equal to 2 points.
If IoU of a true positive detection is bigger than 0.85, it results in adding
1 point to final result. Otherwise points are calculated as
((IoU - 0.5)/0.35)^0.25.
The final score is computed as sum of all true positive points minus all penalties.
For exact algorithm, please, refer to the code.
Scoring software is written in Rust, so you'll need to grab a Rust installation in order to compile it. In general, we use the latest stable release of the Rust compiler, but older versions may work as well.
$ git clone https://github.com/icevision/score
$ cd ripgrep
$ cargo build --release
$ ./target/release/icevision-score --help
icevision-score 0.1.0
Artyom Pavlov <newpavlov@gmail.com>
IceVision competition scoring software
USAGE:
icevision-score <ground_truth> <solution>
FLAGS:
-h, --help Prints help information
-V, --version Prints version information
ARGS:
<ground_truth> Path to a directory with ground truth TSV files.
<solution> Path to a solution TSV file.
For ground truth we use some files from annotations repository.
$ ./icevision-score examples/ground_truth/ examples/good.tsv
Total score: 1.783
Total penalty: 0.000
Score 2.1: 0.000
Score 2.4: 0.783
Score 3.1: 0.000
Score 3.24: 0.000
Score 3.27: 0.000
Score 4.1: 1.000
Score 4.2: 0.000
Score 5.19: 0.000
Score 5.20: 0.000
Score 8.22: 0.000
Penalty 2.1: 0.000
Penalty 2.4: 0.000
Penalty 3.1: 0.000
Penalty 3.24: 0.000
Penalty 3.27: 0.000
Penalty 4.1: 0.000
Penalty 4.2: 0.000
Penalty 5.19: 0.000
Penalty 5.20: 0.000
Penalty 8.22: 0.000
Licensed under either of
at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.