Property-Based Testing: Formalized Robotic Testing for Standard Compliance

Copyright and licence

Author

Salman Omar Sohail

Co-authors

Nico Hochgeschwender, Paul G. Pl ̈oger, Sven Schneider

Publications

Important

Please cite the following:

Automated Testing of Standard Conformance for Robots

@INPROCEEDINGS{10260447,
  author={Sohail, Salman Omar and Schneider, Sven and Hochgeschwender, Nico},
  booktitle={2023 IEEE 19th International Conference on Automation Science and Engineering (CASE)}, 
  title={Automated Testing of Standard Conformance for Robots}, 
  year={2023},
  volume={},
  number={},
  pages={1-8},
  abstract={Verifying that robots conform to standards is an essential part of any responsible development process. A common technique to verify whether robots faithfully meet the requirements of standards is testing. Unfortunately, conformance testing is, to a large extent, a manual and therefore costly exercise which needs to be repeated for every robot under test. In this paper, we propose an automated approach to robot standard conformance testing (RSCT). Based on an analysis of conformance clauses in robot standards (e.g., ISO 23482–1 and ISO 10218) we identify, harmonise and formalise high-level properties representing the assertions to be tested. To verify whether properties hold, we put forward a Robot Test Definition Language (RTDL) which enables developers to specify reusable test scenarios which can be executed in simulation. We evaluate the approach with four heterogeneous robot platforms, identifying previously undiscovered defects in a robot driver. Our approach enables the automated testing of 40% of the conformance clauses defined in ISO 23482–1.},
  keywords={Computer aided software engineering;Automation;ISO Standards;Manuals;Conformance testing;Robots;Standards},
  doi={10.1109/CASE56687.2023.10260447},
  ISSN={2161-8089},
  month={Aug},}

Property-Based Testing in Simulation for Verifying Robot Action Execution in Tabletop Manipulation

@INPROCEEDINGS{9568837,
  author={Sohail, Salman Omar and Mitrevski, Alex and Hochgeschwender, Nico and Plöger, Paul G.},
  booktitle={2021 European Conference on Mobile Robots (ECMR)}, 
  title={Property-Based Testing in Simulation for Verifying Robot Action Execution in Tabletop Manipulation}, 
  year={2021},
  volume={},
  number={},
  pages={1-7},
  abstract={An important prerequisite for the reliability and robustness of a service robot is ensuring the robot’s correct behavior when it performs various tasks of interest. Extensive testing is one established approach for ensuring behavioural correctness; this becomes even more important with the integration of learning-based methods into robot software architectures, as there are often no theoretical guarantees about the performance of such methods in varying scenarios. In this paper, we aim towards evaluating the correctness of robot behaviors in tabletop manipulation through automatic generation of simulated test scenarios in which a robot assesses its performance using property-based testing. In particular, key properties of interest for various robot actions are encoded in an action ontology and are then verified and validated within a simulated environment. We evaluate our framework with a Toyota Human Support Robot (HSR) which is tested in a Gazebo simulation. We show that our framework can correctly and consistently identify various failed actions in a variety of randomised tabletop manipulation scenarios, in addition to providing deeper insights into the type and location of failures for each designed property.},
  keywords={Software architecture;Service robots;Reliability theory;Ontologies;Robustness;Software reliability;Mobile robots},
  doi={10.1109/ECMR50962.2021.9568837},
  ISSN={},
  month={Aug},}

Framework Overview

Code Documentation

Docs: Property-Based Testing: Formalized Robotic Testing for Standard Compliance

Running Framework (Hypothesis|ROS|Gazebo|Allure|TextX)

Basic setup

./atg.sh

Run from inside the src/property_based_tester folder, this applies the tests

python3 -m pytest --alluredir=results tests/pbt_mobile_robot.py -v -W ignore::DeprecationWarning 
python3 -m pytest --alluredir=results tests/pbt_robotic_arm.py -v -W ignore::DeprecationWarning 
python3 -m pytest --alluredir=results tests/pbt_random.py -v -W ignore::DeprecationWarning

The first test is for any ground vehicle
The second test is for robotic manipulators
The third test is for randomized testing

NOTE: Examples for applying these tests are available in src/property_based_tester/ robot_test_definition_language/*_test_definitions

Run for the navigation tests

roslaunch jackal_navigation odom_navigation_demo.launch 
roslaunch husky_navigation move_base_mapless_demo.launch

Robot configuration inside the src/property_based_tester/configuration/property_based_tester_params.yaml

Current available robots:

 Robot: 
   robot_urdf_name: husky # URDF name
   robot_spawner_name: spawn_husky_controller.launch
   robot_size: [0.580, 0.430, 0.250] # x,y,z in meters

 Robot: 
   robot_urdf_name: jackal_robot_issac # URDF name
   robot_spawner_name: spawn_jackal_controller.launch
   robot_size: [0.990, 0.670, 0.390] # x,y,z in meters

 Robot: 
   robot_urdf_name: xarm6_gripper
   robot_spawner_name: spawn_xarm6_controller.launch

 Robot: 
   robot_urdf_name: rovo # URDF name
   robot_spawner_name: spawn_rovo_wo_controller.launch
   robot_controller: rovo_standalone_controller.launch
   robot_velocity: /rovo_velocity_controller/cmd_vel
   robot_size: [1.200, 1.230, 0.530] # x,y,z in meters

 Robot: 
   robot_urdf_name: b1 # URDF name
   robot_spawner_name: spawn_b1_wo_controller.launch
   robot_controller: b1_standalone_controller.launch
   robot_velocity: /b1/cmd_vel/smooth
   robot_size: [1.10, 0.45, 0.50] # x,y,z in meters

Launching their navigation drivers

roslaunch husky_navigation move_base_mapless_demo.launch

roslaunch jackal_navigation odom_navigation_demo.launch

roslaunch rovo_navi odom_navigation.launch

roslaunch b1_gazebo qre_gazebo_nav.launch

Result generation (Allure)

Run from inside the src/property_based_tester folder, this generates the results viewable from chrome

./result_generation.sh

Collision sensor addition in Gazebo

Use the default template to add the senor plugin to your robot.

Link and joint template:

<!-- Links and joints templplate if you do not already have -->
 <link name="sim_collider">
    <collision>
      <origin xyz="0 0 0" rpy="0 0 0"/>
      <geometry>
        <box size="2.0 0.8 0.1"/>
      </geometry>
    </collision>
    <visual>
      <origin xyz="0 0 0" rpy="0 0 0"/>
      <geometry>
         <box size="2.0 0.8 0.1"/>
      </geometry>
    </visual>
    <inertial>
      <mass value="1e-5" />
      <origin xyz="0 0 0" rpy="0 0 0"/>
      <inertia ixx="1e-6" ixy="0" ixz="0" iyy="1e-6" iyz="0" izz="1e-6" />
    </inertial>
  </link>

  <joint name="sim_collider_joint" type="fixed">
    <origin xyz="0 0 0" rpy="0 0 0"/>
    <parent link="base_link"/> <!-- Change param parent link -->
    <child link="sim_collider"/>   <!-- Change param same as link -->
  </joint>

Sensor template:

<!-- Contact Sensor -->
  <gazebo reference="sim_collider"> <!-- Change param same as link -->
    <sensor name="sim_collider_link_contact_sensor" type="contact">
      <always_on>true</always_on>
      <update_rate>100</update_rate>
      <contact>
        <collision>base_link_fixed_joint_lump__sim_collider_collision_2</collision> <!-- Change param same as link -->
      </contact>
      <plugin filename="libgazebo_ros_bumper.so" name="gazebo_ros_sim_collider_controller">
        <always_on>true</always_on>
        <update_rate>100</update_rate>
        <frameName>world</frameName> <!-- Change param -->
        <bumperTopicName>bumper_contact_state</bumperTopicName>
      </plugin>
    </sensor>
    <material>Gazebo/Orange</material>
  </gazebo>

Convert your URDF to SDF as there are parsing issues in detecting collision name.

gz sdf -p my.urdf > my.sdf

Incase you have to convert your xacro urdf to URDF

rosrun xacro xacro -o model.urdf model.urdf.xacro

Search in the .sdf file the collision tag corresponding to your URDF and place it in the contact sensors collision tag (i.e. sim_collider).

Robot Test Definition Language RTDL (TextX)

Run from inside the src/property_based_tester/robot_test_definition_language folder, generates a DSL

./rtdl.sh

Test definitions are written in src/property_based_tester/robot_test_definition_language/standard_test_definitions.pblg
Used for feeding into hypothesis for generating tests

Running Framework (Hypothesis|ROS|Omniverse Nvidia IssacSim|Allure|TextX)

(In the works!) Generate Universal Scene Description physics based simulation in ISSAC Sim.

./omni.sh

Acknowledgements

Supervised by:

Prof. Dr. Nico Hochgeschwender
Prof. Dr. Paul G. Pl ̈oger
M.Sc. Sven Schneider

Resource Provision:

MYBOTSHOP

Software Requirements

Ubuntu 20.04 LTS
Python 3.6.12 64-bit
Gazebo 7.16.1
Catkin-pkg 0.4.22-100
ROS-Noetic
numpy 1.11.0
numpy-stl
cuda 11.0
cuddnn 8
tensorflow 1.4.0
keras 2.0.8
pandas 0.17.1
termcolor 1.1.0
pytest==4.6.11
maven
jdk 8
allure-pytest==2.6.0
allure==2.6.0
torch==1.4.0
torchvision==0.5.0
allure-python-commons==2.6.0
hypothesis--4.57.1
Nvidia-Issac-Sim
TextX

Required ROS Packages

Husky
Jackal
xARM6
Ouster
Gazebo-pkgs
MBS ROVO2 (Closed source)
Quadruped B1 (Closed source)
Yocs Velocity Smoother
RoboticsGroup Gazebo Plugins

Hardware Requirements

These constitute the bare minimum requirements to run this package.

16 Gb RAM
AMD Ryzen 5 5600H CPU @ 3.30GHz
Nvidia GeForce RTX 3060
250 Gb SSD

Known Issues

The simulator bugging out can be caused by fast physics rendering. Turn the real time factor of the physics simulatior down to get better results.

Movebase not working

The movebase is not running. Make sure the topics are correct for the odom, map, and cmd_vel. Also, make sure that sim_time is enabled for move_basei.e.:

<param name="use_sim_time" value="true" />

Ouster or other Gazebo plugins not working

Add this and place the plugins binaries and shared objects into this folder

export GAZEBO_PLUGIN_PATH=/home/sorox23/robotic_ws/master_thesis_ws/src/property_based_tester/plugins/gazebo_plugins_manual_amd64

SOHAIL1996/property_based_tester