Airsim crashes during training

Question

Airsim crashes during training

Ujwal2910 opened this issue 5 years ago · 16 comments

The simulator has been crashing an awful lot of times. After i start the training, usually after 300 or 200 steps the simulator crashes. This was the last log after the crash -

[2019.10.17-14.13.10:666][798]LogLevel: WORLD TRANSLATION END {-1810, -860, 172} took 0.3551 ms [2019.10.17-14.13.43:011][801]LogLevel: WORLD TRANSLATION BEGIN {-1810, -860, 172} -> {0, 0, 0} [2019.10.17-14.13.43:012][801]LogLevel: WORLD TRANSLATION END {0, 0, 0} took 0.3122 ms [2019.10.17-14.13.43:121][801]LogPackageName: SearchForPackageOnDisk took 0.109s to resolve Soccer_Field_Easy. [2019.10.17-14.13.43:122][801]LogStreaming: Display: ULevelStreaming::RequestLevel(/MSBuild2018/Maps/Soccer_Field_Easy_LevelInstance_12) is flushing async loading [2019.10.17-14.13.43:123][801]LogStreaming: Display: Flushing async loaders. [2019.10.17-14.13.43:161][802]LogUObjectHash: Compacting FUObjectHashTables data took 3.60ms [2019.10.17-14.13.44:131][862]LogTemp: spawning player [2019.10.17-14.13.44:143][862]LogLevel: WORLD TRANSLATION BEGIN {0, 0, 0} -> {-1810, -860, 172} [2019.10.17-14.13.44:144][862]LogLevel: WORLD TRANSLATION END {-1810, -860, 172} took 0.3570 ms [2019.10.17-14.14.25:888][445]LogLevel: WORLD TRANSLATION BEGIN {-1810, -860, 172} -> {0, 0, 0} [2019.10.17-14.14.25:888][445]LogLevel: WORLD TRANSLATION END {0, 0, 0} took 0.2592 ms [2019.10.17-14.14.25:993][445]LogPackageName: SearchForPackageOnDisk took 0.104s to resolve Soccer_Field_Easy. [2019.10.17-14.14.25:997][445]LogStreaming: Display: ULevelStreaming::RequestLevel(/MSBuild2018/Maps/Soccer_Field_Easy_LevelInstance_13) is flushing async loading [2019.10.17-14.14.25:997][445]LogStreaming: Display: Flushing async loaders. [2019.10.17-14.14.26:034][446]LogUObjectHash: Compacting FUObjectHashTables data took 3.03ms [2019.10.17-14.14.27:009][506]LogTemp: spawning player [2019.10.17-14.14.27:023][506]LogLevel: WORLD TRANSLATION BEGIN {0, 0, 0} -> {-1810, -860, 172} [2019.10.17-14.14.27:023][506]LogLevel: WORLD TRANSLATION END {-1810, -860, 172} took 0.5131 ms Signal 11 caught. Signal 11 caught. Signal 11 caught. Malloc Size=65538 LargeMemoryPoolOffset=65554 CommonUnixCrashHandler: Signal=11 Malloc Size=65535 LargeMemoryPoolOffset=131119 Malloc Size=119280 LargeMemoryPoolOffset=250416 Failed to find symbol file, expected location: "/home/ujwal/Nips Drone/AirSim/AirSimExe/Binaries/Linux/AirSimExe.sym" terminating with uncaught exception of type std::__1::system_error: close: Bad file descriptor Signal 6 caught. Malloc Size=77368 LargeMemoryPoolOffset=327800 Engine crash handling finished; re-raising signal 11 for the default handler. Good bye. Segmentation fault (core dumped)
Could you please provide an update on this asap.

Answer 1 · 2019-10-17T20:28:08.000Z

this is happening at simloadlevel, it seems?
just checking if you're simloadlevel-ing after each episode?
or is the same as reported in #90 and #77?

Answer 2 · 2019-10-18T08:19:34.000Z

Yes, I am loading the level after each reset. Even #90 and #77 are almost the same but still, there has not been any solution provided.

Answer 3 · 2019-10-19T12:20:38.000Z

Fatal error: [File:/home/madratman/projects/UnrealEngine/4.22/UnrealEngine/Engine/Source/Runtime/Core/Private/Unix/UnixPlatformMemory.cpp] [Line: 776]
Ran out of memory allocating 4295032832 bytes with alignment 65536
Signal 11 caught.
Malloc Size=65538 LargeMemoryPoolOffset=131122
CommonUnixCrashHandler: Signal=11
Malloc Size=65535 LargeMemoryPoolOffset=196687
Malloc Size=111328 LargeMemoryPoolOffset=308032
Malloc Size=80654 LargeMemoryPoolOffset=388702
Engine crash handling finished; re-raising signal 11 for the default handler. Good bye.
Segmentation fault (core dumped)

same problem

Answer 4 · 2019-10-20T07:40:25.000Z

So this time have been not loading the level but just reseting it. Could you please look into this error and report back with a solution asap-

``
[2019.10.19-09.29.36:061][ 0]LogLoad: Took 2.729065 seconds to LoadMap(/Game/MainMenu)
[2019.10.19-09.29.36:068][ 0]LogSlate: Took 0.007354 seconds to synchronously load lazily loaded font '../../../Engine/Content/EngineFonts/Faces/RobotoBold.ufont' (160K)
[2019.10.19-09.29.36:221][ 0]LogStreaming: Display: ULevelStreaming::RequestLevel(/Game/FlyingCPP/Maps/Blocks_LevelInstance_1) is flushing async loading
Checking Pak Config[2019.10.19-09.29.36:332][ 0]LogLoad: (Engine Initialization) Total time: 7.92 seconds
[2019.10.19-09.29.36:418][ 0]LogRenderer: Reallocating scene render targets to support 1872x1028 Format 10 NumSamples 1 (Frame:1).
[2019.10.19-09.29.36:644][ 0]LogInit: Adding external input plugin.
[2019.10.19-09.29.36:651][ 0]LogContentStreaming: Texture pool size now 1000 MB
[2019.10.19-09.29.36:694][ 0]LogRenderer: Reallocating scene render targets to support 128x128 Format 10 NumSamples 1 (Frame:1).
[2019.10.19-09.29.36:730][ 0]LogRenderer: Reallocating scene render targets to support 1872x1028 Format 10 NumSamples 1 (Frame:2).
[2019.10.19-09.29.36:795][ 0]LogSlate: Slate font cache was flushed
[2019.10.19-09.29.51:763][914]LogSlate: Took 0.008749 seconds to synchronously load lazily loaded font '../../../Engine/Content/EngineFonts/Faces/RobotoBold.ufont' (160K)
[2019.10.19-09.29.51:938][920]LogPackageName: SearchForPackageOnDisk took 0.096s to resolve Soccer_Field_Easy.
[2019.10.19-09.29.51:941][920]LogStreaming: Display: ULevelStreaming::RequestLevel(/MSBuild2018/Maps/Soccer_Field_Easy_LevelInstance_2) is flushing async loading
[2019.10.19-09.29.51:941][920]LogStreaming: Display: Flushing async loaders.
[2019.10.19-09.29.52:277][922]LogUObjectHash: Compacting FUObjectHashTables data took 5.67ms
[2019.10.19-09.29.52:943][963]LogTemp: spawning player
[2019.10.19-09.29.52:956][963]LogLevel: WORLD TRANSLATION BEGIN {0, 0, 0} -> {-1810, -860, 172}
[2019.10.19-09.29.52:957][963]LogLevel: WORLD TRANSLATION END {-1810, -860, 172} took 0.6297 ms
[2019.10.19-09.29.53:943][ 24]LogUMG: Display: Widget Class RaceUI_C - Loaded Fast Template.
[2019.10.19-09.29.56:300][170]LogUMG: Display: Widget Class RacerStatusInfo_C - Loaded Fast Template.
[2019.10.19-09.50.15:574][660]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.19-09.50.16:575][722]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.19-09.50.17:578][784]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.19-10.42.22:518][277]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.19-10.42.23:523][339]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.19-10.42.24:525][401]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.19-10.42.25:526][463]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.19-11.08.38:980][886]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
terminating with uncaught exception of type std::__1::bad_weak_ptr: bad_weak_ptrterminating with uncaught exception of type std::__1::bad_weak_ptr: bad_weak_ptr

Signal 6 caught.
Signal 6 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554
CommonUnixCrashHandler: Signal=6
Malloc Size=65535 LargeMemoryPoolOffset=131119
Malloc Size=119280 LargeMemoryPoolOffset=250416
Failed to find symbol file, expected location:
"/home/ujwal/Nips Drone/AirSim/AirSimExe/Binaries/Linux/AirSimExe.sym"
Malloc Size=67917 LargeMemoryPoolOffset=318349
Engine crash handling finished; re-raising signal 6 for the default handler. Good bye.
Aborted (core dumped)
``

Answer 5 · 2019-10-21T22:49:12.000Z

@Ujwal2910 we just updated the binaries with a fix to these reset bugs. can you please re-download airsim.zip and let us know if it works now?

Answer 6 · 2019-10-22T09:06:50.000Z

Yeah i tested that. The simulation still crashes giving the following log-
`` Signal 11 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554
CommonUnixCrashHandler: Signal=11
Malloc Size=65535 LargeMemoryPoolOffset=131119
Malloc Size=111328 LargeMemoryPoolOffset=242464
Failed to find symbol file, expected location:
"/home/ujwal/Nips Drone/AirSim/AirSimExe/Binaries/Linux/AirSimExe.sym"
[2019.10.22-08.53.54:677][315]LogCore: === Critical error: ===
Unhandled Exception: SIGSEGV: unaligned memory access (SIMD vectors?)

[2019.10.22-08.53.54:677][315]LogCore: Fatal error!

0x0000000002747307 AirSimExe!UnknownFunction(0x2547307)
0x0000000002747100 AirSimExe!UnknownFunction(0x25470ff)
0x000000000274e20d AirSimExe!UnknownFunction(0x254e20c)
0x0000000002fe9057 AirSimExe!UnknownFunction(0x2de9056)
0x00000000031146fe AirSimExe!UnknownFunction(0x2f146fd)
0x00000000027112fb AirSimExe!UnknownFunction(0x25112fa)
0x00000000027111b1 AirSimExe!UnknownFunction(0x25111b0)
0x0000000002747c77 AirSimExe!UnknownFunction(0x2547c76)
0x000000000274e626 AirSimExe!UnknownFunction(0x254e625)
0x0000000002fe9057 AirSimExe!UnknownFunction(0x2de9056)
0x00000000031114b3 AirSimExe!UnknownFunction(0x2f114b2)
0x000000000311830f AirSimExe!UnknownFunction(0x2f1830e)
0x0000000003111f7d AirSimExe!UnknownFunction(0x2f11f7c)
0x0000000003120079 AirSimExe!UnknownFunction(0x2f20078)
0x0000000003111f7d AirSimExe!UnknownFunction(0x2f11f7c)
0x0000000003120079 AirSimExe!UnknownFunction(0x2f20078)
0x0000000003111f7d AirSimExe!UnknownFunction(0x2f11f7c)
0x0000000002fe9057 AirSimExe!UnknownFunction(0x2de9056)
0x00000000031146fe AirSimExe!UnknownFunction(0x2f146fd)
0x0000000004a289b5 AirSimExe!UnknownFunction(0x48289b4)
0x00000000027112fb AirSimExe!UnknownFunction(0x25112fa)
0x00000000027111b1 AirSimExe!UnknownFunction(0x25111b0)
0x0000000004cd4c68 AirSimExe!UnknownFunction(0x4ad4c67)
0x0000000004cd811f AirSimExe!UnknownFunction(0x4ad811e)
0x0000000004cd7158 AirSimExe!UnknownFunction(0x4ad7157)
0x0000000004cd1aa9 AirSimExe!UnknownFunction(0x4ad1aa8)
0x0000000004a3eb34 AirSimExe!UnknownFunction(0x483eb33)
0x000000000267d69e AirSimExe!UnknownFunction(0x247d69d)
0x0000000002671abc AirSimExe!UnknownFunction(0x2471abb)
0x000000000267d53c AirSimExe!UnknownFunction(0x247d53b)
0x000000000269bf4f AirSimExe!UnknownFunction(0x249bf4e)
0x000000000268fc50 AirSimExe!UnknownFunction(0x248fc4f)
0x0000000004a29b9a AirSimExe!UnknownFunction(0x4829b99)
0x0000000004a262b7 AirSimExe!UnknownFunction(0x48262b6)
0x00000000056b6f4a AirSimExe!UnknownFunction(0x54b6f49)
0x00000000056b6a1a AirSimExe!UnknownFunction(0x54b6a19)
0x0000000002ba145d AirSimExe!UnknownFunction(0x29a145c)
0x0000000002ba00a3 AirSimExe!UnknownFunction(0x29a00a2)
0x0000000002b9e4e5 AirSimExe!UnknownFunction(0x299e4e4)
0x00000000056b17a1 AirSimExe!UnknownFunction(0x54b17a0)
0x00000000056ac3db AirSimExe!UnknownFunction(0x54ac3da)
0x000000000500c050 AirSimExe!UnknownFunction(0x4e0c04f)
0x0000000004e966df AirSimExe!UnknownFunction(0x4c966de)
0x000000000258364b AirSimExe!UnknownFunction(0x238364a)
0x0000000002589f5a AirSimExe!UnknownFunction(0x2389f59)
0x0000000005da8b7f AirSimExe!UnknownFunction(0x5ba8b7e)
0x00007fcd1763ab97 libc.so.6!__libc_start_main(+0xe6)
0x0000000002579029 AirSimExe!UnknownFunction(0x2379028)

[2019.10.22-08.53.54:718][315]LogExit: Executing StaticShutdownAfterError
[2019.10.22-08.53.54:793][315]LogCore: Warning: Unable to statfs('/home/ujwal/Nips Drone/AirSim/AirSimExe/Saved/Crashes/crashinfo-AirSimExe-pid-29448-027E455308D756FB78021C8CC6EE2988/AirSimExe.log'): errno=2 (No such file or directory)
Malloc Size=87509 LargeMemoryPoolOffset=329989
Engine crash handling finished; re-raising signal 11 for the default handler. Good bye.
Segmentation fault (core dumped)
``

Also for reset i am using the following snippet-
` def reset(self):

    self.baseline_racer.stop_odometry_callback_thread()
    self.baseline_racer.reset_race()
    # self.baseline_racer = BaselineRacer(drone_name="drone_1", viz_traj=self.viz_traj,
    #                                viz_traj_color_rgba=[1.0, 1.0, 0.0, 1.0], viz_image_cv2=self.viz_image_cv2)
    # # self.baseline_racer.load_level(self.level_name)
    self.baseline_racer.start_race(self.race_tier)
    self.baseline_racer.initialize_drone()
    self.baseline_racer.takeoff_with_moveOnSpline()`

Is this fine and if not can you suggest me a different snippet to restart the episode?

Answer 7 · 2019-10-22T17:56:58.000Z

@Ujwal2910 my bad.
We only updated the v1.0 / qualification binaries.
I just updated the v0.3 / training linux binaries https://github.com/microsoft/AirSim-NeurIPS2019-Drone-Racing/releases/tag/v0.3.0-linux

Please re-download AirSim.zip

Yeah, you don't need to update the pythonclient. just ensure it's airsimneurips v1.0

Answer 8 · 2019-10-23T11:45:07.000Z

hasn't improved. Still crashes after almost the same time-

`
[2019.10.23-09.14.05:470][ 0]LogLoad: Took 2.599030 seconds to LoadMap(/Game/MainMenu)
[2019.10.23-09.14.05:477][ 0]LogSlate: Took 0.007271 seconds to synchronously load lazily loaded font '../../../Engine/Content/EngineFonts/Faces/RobotoBold.ufont' (160K)
[2019.10.23-09.14.05:572][ 0]LogStreaming: Display: ULevelStreaming::RequestLevel(/Game/FlyingCPP/Maps/Blocks_LevelInstance_1) is flushing async loading
Checking Pak Config[2019.10.23-09.14.05:692][ 0]LogLoad: (Engine Initialization) Total time: 7.14 seconds
[2019.10.23-09.14.05:812][ 0]LogRenderer: Reallocating scene render targets to support 1600x900 Format 10 NumSamples 1 (Frame:1).
[2019.10.23-09.14.05:816][ 0]LogRHI: Using OpenGL program LRU cache: 0
[2019.10.23-09.14.06:041][ 0]LogInit: Adding external input plugin.
[2019.10.23-09.14.06:045][ 0]LogContentStreaming: Texture pool size now 1000 MB
[2019.10.23-09.14.06:059][ 0]LogRenderer: Reallocating scene render targets to support 128x128 Format 10 NumSamples 1 (Frame:1).
[2019.10.23-09.14.06:132][ 0]LogRenderer: Reallocating scene render targets to support 1600x900 Format 10 NumSamples 1 (Frame:2).
[2019.10.23-09.14.06:175][ 0]LogLinux: Setting swap interval to 'Immediate'
[2019.10.23-09.14.06:194][ 0]LogSlate: Slate font cache was flushed
[2019.10.23-09.14.11:030][297]LogSlate: Took 0.008636 seconds to synchronously load lazily loaded font '../../../Engine/Content/EngineFonts/Faces/RobotoBold.ufont' (160K)
[2019.10.23-09.14.11:256][302]LogPackageName: SearchForPackageOnDisk took 0.146s to resolve Soccer_Field_Easy.
[2019.10.23-09.14.11:258][302]LogStreaming: Display: ULevelStreaming::RequestLevel(/MSBuild2018/Maps/Soccer_Field_Easy_LevelInstance_2) is flushing async loading
[2019.10.23-09.14.11:258][302]LogStreaming: Display: Flushing async loaders.
[2019.10.23-09.14.11:606][304]LogUObjectHash: Compacting FUObjectHashTables data took 6.40ms
[2019.10.23-09.14.12:258][334]LogTemp: spawning player
[2019.10.23-09.14.12:269][334]LogLevel: WORLD TRANSLATION BEGIN {0, 0, 0} -> {-1810, -860, 172}
[2019.10.23-09.14.12:270][334]LogLevel: WORLD TRANSLATION END {-1810, -860, 172} took 0.5233 ms
[2019.10.23-09.14.13:263][386]LogUMG: Display: Widget Class RaceUI_C - Loaded Fast Template.
[2019.10.23-09.14.15:605][528]LogUMG: Display: Widget Class RacerStatusInfo_C - Loaded Fast Template.
[2019.10.23-09.14.17:172][614]LogRenderer: Reallocating scene render targets to support 1872x1028 Format 10 NumSamples 1 (Frame:617).
[2019.10.23-09.34.35:690][ 30]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.23-09.34.45:089][612]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.23-09.34.46:090][674]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.23-09.39.48:839][417]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.23-09.39.49:842][479]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.23-09.39.50:842][541]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.23-09.39.51:842][603]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.23-09.39.52:842][665]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.23-09.50.57:035][781]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.23-09.50.58:037][843]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.23-09.50.59:039][905]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.23-09.51.00:039][967]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.23-09.51.01:039][ 29]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.23-09.51.02:039][ 91]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.23-09.51.03:040][153]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.23-09.51.04:047][215]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.23-09.51.05:048][277]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.23-09.51.06:049][339]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.23-10.29.21:401][366]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.23-10.29.22:401][428]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.23-11.16.04:123][871]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.23-11.16.05:124][933]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
[2019.10.23-11.16.06:125][995]LogBlueprintUserMessages: [drone_1] "Penalty 2 incurred +1 seconds."
Signal 11 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554
CommonUnixCrashHandler: Signal=11
Malloc Size=65535 LargeMemoryPoolOffset=131119
Malloc Size=111328 LargeMemoryPoolOffset=242464
Failed to find symbol file, expected location:
"/home/ujwal/Desktop/Drone Training/AirSim/AirSimExe/Binaries/Linux/AirSimExe.sym"
Malloc Size=84172 LargeMemoryPoolOffset=326652
Engine crash handling finished; re-raising signal 11 for the default handler. Good bye.
Segmentation fault (core dumped)

`
We havent been able to carry out a full training yet and the time is flying as the deadline approaches. Please resolve this issue so we can atleast train some models on the environment

Answer 9 · 2019-10-23T20:26:36.000Z

We have fixed a couple more crash cases in the binaries. Please re-download and give them a try.

Answer 10 · 2019-10-24T11:26:07.000Z

No still crashes and now even early.
terminating with uncaught exception of type std::__1::bad_weak_ptr: bad_weak_ptr
Signal 6 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554
CommonUnixCrashHandler: Signal=6
Malloc Size=65535 LargeMemoryPoolOffset=131119
Malloc Size=111328 LargeMemoryPoolOffset=242464
Failed to find symbol file, expected location:
"/home/ujwal/Desktop/Drone Training/AirSim/AirSimExe/Binaries/Linux/AirSimExe.sym"
Malloc Size=92400 LargeMemoryPoolOffset=334880
Engine crash handling finished; re-raising signal 6 for the default handler. Good bye.
Aborted (core dumped)

Answer 11 · 2019-10-24T17:56:22.000Z

@Ujwal2910 can you give us a script to reproduce this?
I recommend using this function to reset the drones: https://github.com/madratman/AirSim-NeurIPS2019-Drone-Racing/blob/tests/one_drone_end_race_and_reset_called/baselines/repro_crash_one_drone_ends_race_and_reset_called.py#L7

I testing the current binaries with the script above and am not seeing any crashes.

def dummy_reset(baseline_racer, drone_names, tier):
    baseline_racer.airsim_client.simPause(True)
    baseline_racer.airsim_client.reset()
    for drone_name in drone_names:
        baseline_racer.airsim_client.enableApiControl(vehicle_name=drone_name)
        baseline_racer.airsim_client.arm(vehicle_name=drone_name)
    baseline_racer.airsim_client.simPause(False) # unpause sim to simresetrace works as it's supposed to
    baseline_racer.airsim_client.simResetRace()
    baseline_racer.airsim_client.simStartRace(tier=tier)

Answer 12 · 2019-10-25T17:13:33.000Z

Hey @Ujwal2910. Any updates on this? Please see discussions in #77 and #104

Answer 13 · 2019-10-25T18:51:24.000Z

Hey @madratman, Thanks for the dummy_reset function. The simulator hasn't crashed at all the everything seems to be working completely fine. I can now finally start the traning.
Thanks.

Answer 14 · 2019-10-25T22:04:14.000Z

Awesome, thanks for your patience Ujwal. Glad it's working now!

Answer 15 · 2019-11-14T05:17:59.000Z

The crash has started again.

Crash log-

Signal 11 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554
CommonUnixCrashHandler: Signal=11
Malloc Size=65535 LargeMemoryPoolOffset=131119
Malloc Size=111328 LargeMemoryPoolOffset=242464
Failed to find symbol file, expected location:
"/home/ujwal/Desktop/Drone Training/AirSim/AirSimExe/Binaries/Linux/AirSimExe.sym"
Malloc Size=83251 LargeMemoryPoolOffset=325731
Engine crash handling finished; re-raising signal 11 for the default handler. Good bye.
Segmentation fault (core dumped)

could you update after looking into this?

Answer 16 · 2019-11-14T06:02:40.000Z

Did you change anything when you say "started again"? See discussion here #123 (comment)