Benchmarking script
Closed this issue · 5 comments
I will happily create a benchmarking script, to evaluate speed of deepstack on different platforms. It would do a batch job of say 30 detections and present the runtime. I don’t want to reinvent the wheel, perhaps you have a script already?
We don't have a standard script yet for that. Please create it and make a PR to include it in benchmark directory.
OK this will be a script which just posts every photo in a directory for processing. Wont include the photos, although we could have a standard set of photos in a seperate repo perhaps
Created a notebook here (will breakout into script) and the results are in, with tflite being lightning fast, processing 15 images (obj detection):
| Platform | Speed (sec) | Predictions |
|---|---|---|
| Mac Pro with deepstack (add version tag) | 51.9 | 91 |
| RPi4 with tflite-server | 9.33 | 159 |
ps which is the 'benchmark directory'?
Interesting. Thanks for these @robmarkcole , i will push a nightly tag pending the setup of the cicd pipeline so you can benchmark against the new detection pipeline.
BTW, as i have not setup the benchmark directory, can you make a PR for these to include the benchmark scripts and we can review and make neccessary updates.
I am also curios, what are the specs for the MacBook Pro?
Hardware Overview:
Model Name: MacBook Pro
Model Identifier: MacBookPro14,1
Processor Name: Dual-Core Intel Core i5
Processor Speed: 2.3 GHz
Number of Processors: 1
Total Number of Cores: 2
L2 Cache (per Core): 256 KB
L3 Cache: 4 MB
Hyper-Threading Technology: Enabled
Memory: 8 GB