Camera sensor position drifts while driving

[hildebrandt-carl]

Hi,

I was trying to attach a camera to the vehicle and then drive it around while recording the camera data. I attached the camera inside the car using:

cam = Camera(pos=(0, 0, 1), direction=(0, 1, 0), fov=60, resolution=(512, 512))
ego_vehicle.attach_sensor('cam', cam)

However I notice as soon as I start driving the camera's position starts drifting as shown below [note this is 5x playback]:

You can replicate this behavior using the python script below. Note you will need to change the home parameter in the BeamNGpy.

import numpy as np
import matplotlib.pyplot as plt

from time import sleep
from beamngpy.sensors import Camera
from beamngpy import BeamNGpy, Scenario, Vehicle

def plot_cam(plt, view):
    overhead_ax = plt.gca()
    view = view.convert('RGB') 
    overhead_ax.imshow(np.asarray(view))
    overhead_ax.set_aspect('equal', 'datalim')

def main():
    beamng = BeamNGpy('localhost', 64256, home='BeamNG.tech.v0.21.3.0')
    bng = beamng.open(launch=True)

    # Create a scenario in west_coast_usa
    scenario = Scenario("smallgrid", 'tech_test', description='Random driving for research')

    # https://documentation.beamng.com/vehicle_system/vehicles/index.html
    ego_vehicle = Vehicle('ego_vehicle', model='etk800', licence='Carlos', color='White')

    # Create sensors
    cam = Camera(pos=(0, 0, 1), direction=(0, 1, 0), fov=60, resolution=(512, 512))
    ego_vehicle.attach_sensor('cam', cam)

    # Add the vehicle
    scenario.add_vehicle(ego_vehicle)

    # Add traffic vehicles
    target_vehicle = Vehicle('target_vehicle', model='etk800', licence='Traffic', color='Red')
    scenario.add_vehicle(target_vehicle, pos=(0, -10, 0))

    # Create the cam view
    plt.figure(1, figsize=(3, 3))

    # Try and start beamng
    try:
        # Compile the scenario and place it in BeamNG's map folder
        scenario.make(bng)

        # bng.hide_hud()
        bng.set_deterministic()  # Set simulator to be deterministic
        bng.set_steps_per_second(60)  # With 60hz temporal resolution

        # Load and start the scenario
        bng.load_scenario(scenario)
        bng.start_scenario()

        # Make sure we are following the ego vehicle
        bng.switch_vehicle(ego_vehicle)

        sensor_rate = 4 #Hz
        sensor_timing = 1.0 / sensor_rate
        time_counter = 0

        while time_counter <= 100:
            # Set this loop to a set frequency
            sleep(sensor_timing)
            time_counter += sensor_timing

            # Retrieve sensor data and show the camera data.
            ego_vehicle.poll_sensors()
            sensors = ego_vehicle.sensors
            view = sensors['cam'].data["colour"]

            # Plot the cam view
            plot_cam(plt, view)

            # Drive the car automatically (throttle between 0;1 / steering between -1;1)
            ego_vehicle.control(throttle=1, steering=0)
            target_vehicle.control(throttle=1, steering=0)

            # Draw the plot
            plt.draw()
            plt.pause(0.001)

    finally:
        bng.close()

if __name__ == '__main__':
    main()

Carl

[Palculator]

Hello, I remember we had a similar bug in earlier versions, but it seems like you're using at least 0.21 of BeamNG.tech. It would be helpful to know which BeamNGpy version you are using, because my gut feeling is the bug that caused Issue 115 could also cause drift.

[hildebrandt-carl]

Hi @Palculator

Sorry for not listing that information in the original post. You are 100% right I am using the same version as issue 115. I am using beamngpy 1.19.1. I will upgrade to a newer version and reopen this if that doesn't solve it. However I suspect you are right about the issue.

Thanks for the help,
Carl