This project builds on top of AirSim to provide efficient parallel training for Reinforcement Learning-based quadrotor controllers. Check out our 3 minute video bellow which explains our approach. We have made modifications to AirLib and the Unreal plugin code, along with python files inside /PRL4AirSim/PRL4AirSim.
Read our paper (https://arxiv.org/pdf/2209.11094.pdf) and cite us.
@article{saunders2022parallel,
title={Parallel Reinforcement Learning Simulation for Visual Quadrotor Navigation},
author={Saunders, Jack and Saeedi, Sajad and Li, Wenbin},
journal={arXiv preprint arXiv:2209.11094},
year={2022}
}
![](https://raw.githubusercontent.com/SaundersJE97/PRL4AirSim/master/PRL4AirSim/images/ArxivLogo.png)
PRL4AirSim provides a simulation framework, built on AirSim, which provides efficient parallel training. We add the following functionality:
- Batch rendering, to remove multiple game and render thread synchronisation using
simGetBatchImages
to encompas all requests into a single request. - Episodic Training, using
resetVehicle
rather thanreset
which causes the entire simulation to reset. - Non-interactive simulator environment, removing ego-perspective rendering and agent-agent collision interaction.
- Ape-X implementation using RPC servers
Build AirSim normally. Ensure you clone this repository rather than AirSim, then follow their steps as we have modified the AirLib and AirSim plugins.
Find the requirements.txt
file which includes modules used in our python environment and versions.
The required modules are:
- AirSim
- PyTorch
- Numpy
- OpenCV
- wandb (for logging)
- json
- msgpackrpc (https://github.com/tbelhalfaoui/msgpack-rpc-python/tree/fix-msgpack-dep) see bellow (use version 4.5.3)
msgpack changed its name from msgpack-python to msgpack, which leads to an outdated version installed. When running the outdated version, the python instance is used rather than the quicker c++ version. Hence, we need to correct this using https://github.com/tbelhalfaoui/msgpack-rpc-python/tree/fix-msgpack-dep
. If Jupyter package is installed, tornado will also be outdated. The torndao version needs to be <5.
Alternatively, run this to install the cython binaries of msgpack:
pip install Cython
git clone -b 0.5.6 https://github.com/msgpack/msgpack-python.git
cd msgpack-python
make
We include a binary which we have used to test this framework.
Download this here: https://drive.google.com/file/d/1PZxAJjld3xS5QLkC2-lSMHRRsR1VZmP8/view
and place it inside the PRL4AirSim/UEBinary
. This should lead to the following file structure:
-- PRL4AirSim
---- UEBinary
------ Engine
------ MyProject2
---- Manifest_NonUFSFiles_Linux.txt
---- MyProject2.sh
This will be compatible with start.sh
, now if you want to create your own custom environments, you can compile them and place it in here. We recommend compiling the project as the UE editor has additional overheads.
One major disavantage of AirSim is the configuration file for the quadrotors within the simulator. Every quadrotor requires a specific configuration which leads to thousands of repetitive lines within the configuration file located in /Documents/AirSim/settings.json
. In the future we hope to automate the num_drones
setting in our config to initialise the number of quadrotors on the AirSim side. However, for now we need this setting (num_drones
) to match the number of vehicles in the Documents/AirSim/settings.json
file. Where each quadrotor id and camera id increments:
{
"SeeDocsAt": "https://github.com/Microsoft/AirSim/blob/master/docs/settings.md",
"SimMode": "Multirotor",
"ViewMode": "NoDisplay",
"SettingsVersion": 1.2,
"ClockSpeed": 4.0,
"LocalHostIp": "127.0.0.1",
"ApiServerPort": 29001,
"SubWindows": [],
"Vehicles": {
"Drone0": {
"X": 0,
"Y": 0,
"Z": 0,
"VehicleType": "SimpleFlight",
"AutoCreate": true,
"DefaultVehicleState": "Armed",
"Cameras": {
"depth_cam_0": {
"CaptureSettings": [
{
"FOV_Degrees": 90,
"ImageType": 1,
"Width": 32,
"Height": 32
}
],
"Pitch": 0.0,
"Roll": 0.0,
"Yaw": 0.0,
"X": 0.0,
"Y": 0.0,
"Z": -0.6
}
}
},
"Drone1": {
"X": 0,
"Y": 0,
"Z": 0,
"VehicleType": "SimpleFlight",
"AutoCreate": true,
"DefaultVehicleState": "Armed",
"Cameras": {
"depth_cam_1": {
"CaptureSettings": [
{
"FOV_Degrees": 90,
"ImageType": 1,
"Width": 32,
"Height": 32
}
],
"Pitch": 0.0,
"Roll": 0.0,
"Yaw": 0.0,
"X": 0.0,
"Y": 0.0,
"Z": -0.6
}
}
}
...
}
}
Furthermore, we modify the ApiServerPort
and LocalHostIp
setting when running the parallel architecture. We have a set of templates that can be swapped out to test any number of quadrotors up to 60 (although we have personally tested with 200) in PRL4AirSim/TemplateConfigurationFiles
.
We provide a config.json
file which contains all parameters of the experiment.
{
"max_episodes" : 100000,
"state_space" : [2, 32, 32],
"action_space" : 2,
"buffer_Size": 15000,
"batch_size" : 32,
"learning_rate" : 0.001,
"discount_factor" : 0.95,
"replace_target_count_episode" : 150,
"num_drones" : 50,
"from_artifact" : "",
"projectName" : "MyProject2",
"envProcesses" : 1,
"storage_port" : 29000,
"headless" : false
}
We used this framework to test visual navigation, as a result the state space and action space is specific for this task. Changing this involves modifying simulator.py
, DQNTrainer.py
, and DQNetwork.py
.
A quick way to test our framework is to use the start.py
python script, which will run a local version of the PRL4AirSim. However, we recommend to run all subprocesses on networked computers.
There are four components:
File | Description |
---|---|
PRL4AirSim/UEBinary/{ProjectName}.sh |
AirSim Unreal Engine Binary |
PRL4AirSim/PyClient.py |
PyClient AirSim Client hosting a local DQN network |
PRL4AirSim/Storage.py |
Replay Buffer |
PRL4AirSim/Trainer.py |
Global network parameter trainer |
Alternatively, we recommend running the airsim and local DQN network on multiple compute units as shown in the figure bellow, thus spreading the GPU and CPU load. This can be done by running the PyClient and UEBinary remotely, ensuring the IP of the buffer is defined on these machines. Check out start.py
on how to do this.
Ape-X enables us to run multiple DQN local instances in parallel (hence parallel RL). This results in an increased data sample rate from the environmnet. The bottleneck of AirSim is the high CPU utilisation from the quadrotor dynamics which are calculated every step for each vehicle. Vectorising the environment can improve memory utilisation
We have modified the AirLib library to accept new commands that enable vectorised based environment interaction. Hence, intialise the MultiRotor client normally
Note: in our code we use client.call(...) without differentiating between the UnrealEngine and Replay buffer. However, for this documentation simClient means the UnrealEngine client and replayClient is the Replay bufer
simClient = airsim.MultirotorClient(ue_ip_address : str, ue_port : int)
simClient.confirmConnection()
bufferClient = msgpackrpc.Client(msgpackrpc.Address(buffer_ip_address : str, buffer_port : int))
print(bufferClient.call("confirmConnection"))
simClient.call('simGetBatchImages', requests : [airsim.ImageRequest], vehicle_names : [str])
simClient.call_async('moveByVelocityZBatch', vx_vec : [float], vy_vec : [float], z_vec : [float], time : float, driveTrainType : airsim.DrivetrainType, yawMode : airsim.YawMode, vehicle_names : [str])
simClient.call('simSetVehiclePoseBatch', poses : [airsim.Pose], vehicle_names : [str])
simClient.call_async("resetVehicle", vehicle_name : str, pose : airsim.Pose, orientation : airsim.Quaternionr)
todo
- more move commands (moveByVelocityBatch(...)
Similarly to AirSim, we use a RPC server to host the replay buffer. This buffer is a centralised storage used by the local agent instances to fill experiences to, and an external trainer client which samples experiences.
def pushMemory(self, state, action, next_state, reward, not_done)
# For Epislon-Greedy action selection, epsilon can decrement with the size of the experience replay.
def getMemoryPushCounter(self)
# There is a delay between when the replay buffer starts and when agents start interacting with the environment.
# We use this to properly define when the experiment starts.
def startSimulation(self)
# Used to track episodic training
def finishEpisode(self)
# The trainer calls this to sample experiences from the buffer. We store experiences as a dictionary of numpy arrays
# which need to be converted to python lists for the RPC server to encode and sent to the trainer client. See 'Utils.convertStateDicToListDic()'
def sampleFromStorage(self)
We provide the environment presented within the paper to allow others to validate our approach. However, to create a custom environment, we recomend you follow the following steps to prevent agent interaction.
To make the quadrotor invisible in the scene, change the 'Hidden in Scene Capture' to True. This will make it invisible to other drones but the spectator actor can still see it. Go to details, then rendering, this will show the setting 'ACtor Hidden In Game'.
We need to specifically remove agent-agent interaction while also enabling environment interaction. Hence we need to define all components of the quadrotor blueprint 'BP_FlyingPawn' as 'Pawn' and ignore any overlaps that occour between this group. To do this, we modify the collision response within the agent blueprint.
There are five components to change within the 'BP_FlyingPawn' blueprint: BodyMesh, Prop3, Prop2, Prop1, Prop0. For all of these, go to collisions, then change the collision presents to custom. Thange the Object Type to 'Pawn' and then in 'Object Responses' change the Pawn to Ignore as shown bellow.
Now to remove collisions between 'Pawns', we need to ignore the event 'ActorBeginOverlap' which we can do using a Blueprint Event Graph. Add the following event graph to 'BP_FlyingPawn'.
Agents will interact with the environment without interacting with each other.
The code on lines 21-26 in Simulation.py is the stored location of where the drones will start and where the end goal is. Please replace these with the test corridor in the map (or make your own) with the coordinates in the unreal engine. Then the function ‘convert_pos_UE_to_AS’ will convert these coordinates to the airsim coordinate frame. E.g
self.areans_train_long = np.array([
[Utils.convert_pos_UE_to_AS(self.origin_UE, np.array([start_x, start_y, start_z])), Utils.convert_pos_UE_to_AS(self.origin_UE, np.array([end_x, end_y, end_z]))]
])
Where start and end are the coordinates in the unreal engine editor.
Then, in lines 50-51 in PyClient.py, you will find the code to load the model. Replace the arguments of ‘load(…)’ with the path of the pth file created after training.
print("loaded best model")
agent.load('{}/BestModelSaves/dqn.pth'.format(pathlib.Path().resolve()))
You will also need to set epsilon to 0 in line 47 to ensure greedy actions are taken.
agent = DQNTrainer.DQNTrainer(image_input_dims=Utils.getConfig()['state_space'],
n_actions=Utils.getConfig()['action_space'],
replayMemory_size=Utils.getConfig()['buffer_Size'],
batch_size=Utils.getConfig()['batch_size'],
learningRate=Utils.getConfig()['learning_rate'],
discount_factor=Utils.getConfig()['discount_factor'],
epsilon=0.0, # <----------- this has been changed to 0.0
replace_target_count_episode=Utils.getConfig()['replace_target_count_episode'])
-
Correct installation of RPC library.
-
DQNetwork and DQNTrainer used for both the PyClient and Trainer
In the future we will break the functionality of both the DQNetwork and DQNTrainer to make it clearer what functions are used for each.
- AirSim and UnrealEngine Coordinate Reference Frame
The reference frame in AirSim is centered around the player start actor. To spawn agents at certain locations, we capture the position within the unreal engine and then transpose it to the AirSim frame. Use the function 'convert_pos_UE_to_AS' to convert between UE and AS in Utils.py
def convert_pos_UE_to_AS(origin_UE : np.array, pos_UE : np.array):
Jack Sanders: JS3442@bath.ac.uk
Sajad Saeedi: s.saeedi@ryerson.ca
Wenbin Li: W.Li@bath.ac.uk