Scripted benchmarking output?
Closed this issue · 5 comments
Hi, interesting project. I've played around with GL_vs_VK a bit and have only briefly looked at the code. It didn't appear implemented, so figured I'd ask, if there were any plans to make this more script-able friendly for benchmarking? Presently it doesn't appear that the test cases don't end on their own after any length of time. If there was a benchmark switch to have the test end gracefully on its own after a given amount of time/frames would be convenient as well as then dumping any frame-rate statistics to the CLI or a log file. Do you have plans to implement such or would accept patches for it? Apologies if there is, so far I've only seen the FPS displayed to the window title bar and didn't see any other options.
Thanks,
Michael
Hey @michaellarabel, thanks. ;)
Yes, you are right. Currently it's not self-testing and requires user to measure results (e.g. by looking at window's title bar) and the tests are in "endless" mode. I though about this feature, but it was only an idea in my head and possible future feature. I guess if there is a need for it, I might implement something like that - shouldn't take much time for me anyway and would be useful.
What kind of statistics would you like to get?
After quick brainstorming I though about adding CLI switches:
-benchmark- turn into benchmark mode, collect stats and end afterTtime-time $T- specify benchmark time (with either per-test default time or one constant like 10s-15s)
Easiest way would be to drop stats toSTDOUTand you could do with it whatever you want. Writing them to files might be good, but running process multiple times would overwrite files (or create too many), so I'm not sure if that's the best idea.
As for collected stats, simplest way would be to get:
- average FPS (and frame-time)
- mean FPS (counter periodically, for example in 1s distances)
- minimum/maximum FPS (measured each frame)
- minimum/maximum frame-time (as above)
I'm not sure if anything related to frames is worth getting, as these rendering at the beginning slightly plummets in terms of frame-time/FPS. I could - for example - discard data from ~1st second. I think that giving time instead of frames should be better considering that some tests depends on current time and rendering 100 frames in 0.1s is not the same as rendering 100 frames in 2s (which can make a difference on my HW in test #3 on Vulkan - frame rate is not stable but slightly jumps up and down depending on camera position).
It would be nice to have some kind of CPU/GPU load workload too, but I'm afraid currently I don't have that much time (at least until I'm finished with my master's thesis) since this would require some OS-specific knowledge and I'm not sure how reliable it would be anyway.
If you have any requests or ideas, don't hesitate to write them. :)
I'll take a look tomorrow on that and try to implement it in general way so that all tests get this functionality automatically.
Yes that would be very useful, thanks. The -benchmark/-b would be great and if -time is optional otherwise default to something. I would probably think ~30 seconds should be good. That's what I was playing around with when basically hacking in the gl and vlk Window files to do some basic measurements, but really was just a quick hack while waiting to hear what your ideas or plans were.
If it's just to stdout would be fine rather than dealing with file handling.
Probably the min/avg/max FPS would be most useful for these tests. I was playing around with dumping the frame-time for comparison, but at least on the Intel Linux box I was playing with, after the first few seconds it seemed relatively stable and so not too interesting if I were to line graph it.
I'm looking to then automate it via the phoronix-test-suite benchmarking framework I maintain. From there I can then poll the CPU/GPU utilization and other metrics concurrently.
Thanks for your help in seeing it benchmark friendly.
Hey @michaellarabel,
Sorry for late response, but I was swamped with other matters. I've managed to implement this quite nicely, however minimum and maximum values are kinda tricky to get right, as simple timing on CPU side is not enough since commands can be send pretty much instantaneously and simply be queued in the driver for execution. However average values seems to be be computed correctly always and seems reliable. I think this can be fixed by not counting single frames but taking measurements from batch (like 'what has been done from last (at least) 1 second' or even 'collect data averaged from last 6 measured frames). I don't know how useful it would be and maybe you have a better idea here, so I'll wait with any changes for some feedback.
I've also added some device/api/drivers/versions/test-name debug informations to be printed so that each output can have some meaning on it's own.
Please test this and if close this issue if you're satisfied. If you have any more ideas please post them. theoretically this can be easily extended to support much better statistics gathering, but I didn't want to impact application performance in any way and I think for now that' enough. :)
Example run commands:
./GL_vs_VK -t 1 -api gl -m -benchmark
GL_vs_VK.exe -t 3 -api vk -benchmark -time 5
Also, it's implemented in a way so that if you interrupt execution by closing window earlier, it will still have correct measurements from time it was running as it's updating data after each frame and as long as there are no errors, it will present gathered statistics.
BTW. Is you test suite publicly available? Testing with CPU/GPU usage monitoring might come in handy if it's some publicly available easy-to-use tool ;)
The new options are working out great, thanks! I posted some benchmarks on different drivers/GPUs at http://www.phoronix.com/scan.php?page=article&item=gl-vs-vk&num=1 Hopefully that will also get some more attention to your project.
Yes, the test suite is open-source and can be used rather easy i.e. on Ubuntu:
sudo apt install php-cli php-xml git-core
git clone https://github.com/phoronix-test-suite/phoronix-test-suite.git
cd phoronix-test-suite
MONITOR=all ./phoronix-test-suite benchmark gl-vs-vk
(Here's basically its integration bits for how gl_vs_vk is then automated via the test suite: https://openbenchmarking.org/innhold/d71fa7d6cf530bdf596f67ca01d28baec228afb5 )
That would run with all detected monitoring sensors available (GPU load reporting isn't supported/offered on some drivers/platforms). Here's an example run I just quickly did and then opted to upload the results: https://openbenchmarking.org/result/1706063-TR-MONITORIN10
Thanks again for adding the benchmark/time switches.
Glad to hear that. I've read this, thanks! :) Also thanks for providing some information on your test-suite, it might come in handy some time. ;)
I'm closing this one then. If you'd have some more requests or ideas for scenarios to tests feel free to raise new issues.
BTW. I may fix max/min frame time to use Xsec time window because currently results are pretty unusable to me. Though I'm currently busy with other stuff so probably in some near future.