SimGetImages() is extremely slow at sending even very small images

[yannbouteiller]

I know this problem is already well known, but obviously this is very painful for Reinforcement Learning approaches because we need to sample a huge amount of observations to make our algorithms learn anything.

On the laptop I use for testing, using SimGetImages() on one single drone with 240*320 images I get images at ~17 FPS. If I use it on both drones I drop to ~ 6 FPS (times 2 images = ~ 12 FPS).

I was hoping that using much lower resolutions would greatly improve this, but it does not. With 12*16 images for instance I get ~ 20 FPS instead of 17 with one drone, and ~ 7 FPS instead of 6 with two drones.

According to pyinstrument which I use for profiling, all this time is spent in the IOLoop of tornado:

118.958 simGetImages airsimneurips/client.py:217
│ │ [100 frames hidden] airsimneurips, msgpackrpc, tornado, l...
│ │ 118.844 start tornado/ioloop.py:753
│ │ └─ 118.760 [self]

But I suppose this just means that AirSim is super slow at computing even very small camera images.

[saihv]

It is indeed true that AirSim/Unreal are slow at image capture, but I am surprised by the very minimal gain in performance between 320x240 and 16x12. What is the configuration of your laptop - and what is your GPU usage like as you're running this? I have a feeling you're being massively bottlenecked somewhere. If you get a chance, could you perhaps try a couple of things:

1. As you're grabbing images, can you run in the Unreal console (which you can access by pressing the tilde key ~ / ``` ) stat unitgraph and post a screenshot of the graph that appears?

2. Can you repeat your benchmarking by locking the game framerate? You can do this by going to the console (~) and using this command : t.maxFPS 30 (I would also recommend trying 15 or 20 to evaluate)

3. If stat unitgraph says most of the time is being spent on the GPU thread, a more detailed breakdown can be obtained by running the Unreal GPU profiler. The command is ProfileGPU in the console. A screenshot of that could also be very helpful to diagnose the problem.

https://docs.unrealengine.com/en-US/Engine/Performance/GPU/index.html

[yannbouteiller]

Hello @saihv , to open the unreal console, do you mean press ~ when the mouse is captured in the airsim window? Because nothing happens when I do this. (I have a azerty keyboard but even when I switch to qwerty and press ` , or shift + ` , which is supposed to give ~, nothing happens)

[saihv]

Interesting - yes, all you would have to do is focus on the window and then press `. (My bad, it's just the back-tic and not the tilde itself, although Unreal refers to it as Tilde, so you shouldn't need shift). Could you make sure the keypress is actually sending that character through, say, notepad?

Just FYI, the console is super tiny, and usually shows up as a little textbox on the bottom of the screen.

[yannbouteiller]

Sorry my bad, I was messing up with the french keyboard. The screenshots for unitgraph are:

EDIT: the screenshots are not representative, I'll make them again.

[yannbouteiller]

Ok so the screenshots are:

240*320 images no framerate locking:

24*32 images no framerate locking:

stopped capturing images no framerate locking:

Setting t.maxFPS doesn't seem to change anything.

[saihv]

Okay, upon first looks, I think there's something wrong here. Your game frame rate is still decently high, and what I find curious is that your draw thread is taking almost as long as the GPU (rendering) itself (IIRC, Draw is the time it takes to prepare the draw calls on the CPU, GPU is how long it actually takes to render them) which makes me think the GPU is being left waiting for something. For instance, this is a screenshot from my PC while capturing images from one drone.

Could you please try the 24x32 image size with our native benchmarking script - with --level_name Soccer_Field_Easy as the argument.

@ironclownfish Any thoughts?

[yannbouteiller]

Sorry again, t.maxFPS does change something.

320*240 images with 30 game FPS:

320*240 images with 20 game FPS:

320*240 images with 15 game FPS:

320*240 images with 5 game FPS:

I don't really know how to interpret these graphs

[saihv]

@yannbouteiller t.maxFPS is supposed to lock the main game framerate to a certain number, which it is doing as you can see by the 'Frame' time on the right. I was mainly curious about whether the image capture FPS is increasing for you when you lock the FPS of the full game.

[yannbouteiller]

It possibly improves the general frame rate a tiny bit but setting it to 15 I am still around 7 FPS for both 240 * 320 and 24 * 32 images (retrieved on both drones), all the time being spent waiting in tornado's IOLoop.

I will try your benchmarking script.

[yannbouteiller]

Using the native benchmarking script I get similar results: for 1 drone, about ~20 FPS for 240 * 320 images and ~ 23 FPS for 24 * 32 images.

[madratman]

@yannbouteiller, can you try the following:

• In your settings.json, add "ViewMode": "NoDisplay" - or just use airsim_neurips_god_settings_nodisplay.json

• In the console, first type r.Vsync 0 to assert frame rate is not limited with monitor refresh speed, then t.maxFPS 10000 (some big number to see what's the max rate you can get from your machine), and then run the benchmarking script and tell us what you get. You can ofcourse run Stat FPS and Stat UnitGraph to see if unreal frame rate increases.

[yannbouteiller]

Then the image FPS becomes much better, however there is still not a huge gain in performance from 240 * 320 to 24 * 32 :

240 * 320:

24 * 32:

Is there a way to enforce these unreal options without using the unreal console through the visualization window ? This sounds pretty much exactly like what I want to do for tier 3 in my computing cluster in the other issue (#112)

[madratman]

Even though I have been able to find ways to do this, I am not really able to do this :\ . See below

This can be done via ini files. In AirSimExe/Saved/Config/LinuxNoEditor, make a ConsoleVariables.ini and dump

[Startup]
r.VSync=0
t.MaxFPS=10000

in it.
Source: https://forums.dovetailgames.com/threads/a-guide-to-engine-ini-settings.4672/

Instead of ConsoleVariables.ini, I have tried adding it to Engine.ini and GameUserSettings.ini to no avail.

Next in line are the unreal binary command line options. "Rendering" section is relevant. https://docs.unrealengine.com/en-US/Programming/Basics/CommandLineArguments/index.html
I have been able to disable vsync via the NoVSync option (./AirSimExe.sh -windowed -opengl4 -NoVSync but for the life of me, I can't set MaxFPS.
I did try the FPS command listed therein to no avail.

@saihv @ironclownfish if you have any insight, please comment

[madratman]

@yannbouteiller I just gave the benchmark option a shot, seems to be it:

./AirSimExe.sh -windowed -NoVSync -BENCHMARK

I am not quite sure how it will affect everything else
BENCHMARK: Run game at fixed-step in order to process each frame without skipping any frames. This is useful in conjunction with DUMPMOVIE options.

After some testing, everything seems to be working well. This is what you need, indeed.