osrf/lrauv

M3 Test plan

braanan opened this issue · 16 comments

braanan commented

Integration tests

Science sensors:

  • Spawn the vehicle at different positions in the simulation and run the science payload run RegressionTests/IgnitionTests/testScienceSensors.xml

CTD: Confirm that LRAUV application sensor reading values for temperature (celsius), salinity (psu), and pressure (decibar), match the expected values at the vehicle's location in the Ignition simulation.

Note: vehicle configuration must include CTD_Seabird.loadAtStartup 1 bool, CTD_Seabird.simulateHardware 1 bool

In the LRAUV application:

run RegressionTests/IgnitionTests/testScienceSensors.xml
get CTD_Seabird.sea_water_temperature
get CTD_Seabird.sea_water_salinity
get CTD_Seabird.sea_water_pressure

Example of expected output:

>run RegressionTests/IgnitionTests/testScienceSensors.xml
2021-08-04T05:15:01.588Z,1628054101.588 [CommandLine](IMPORTANT): got command run ./Missions/testScienceSensors.xml
2021-08-04T05:15:01.588Z,1628054101.588 [MissionManager](INFO): Loading Mission: ./Missions/testScienceSensors.xml
...
>get CTD_Seabird.sea_water_temperature
2021-07-28T17:33:03.613Z,1627493583.613 [CommandLine](IMPORTANT): got command get CTD_Seabird.sea_water_temperature
2021-07-28T17:33:03.613Z,1627493583.613 [CommandLine](IMPORTANT): CTD_Seabird.sea_water_temperature 16.717402 degC
...
>get CTD_Seabird.sea_water_pressure
2021-07-28T17:33:37.608Z,1627493617.608 [CommandLine](IMPORTANT): got command get CTD_Seabird.sea_water_pressure
2021-07-28T17:33:37.608Z,1627493617.608 [CommandLine](IMPORTANT): CTD_Seabird.sea_water_pressure -0.000010 db
...
>get CTD_Seabird.sea_water_salinity
2021-07-28T17:35:16.847Z,1627493716.847 [CommandLine](IMPORTANT): got command get CTD_Seabird.sea_water_salinity
2021-07-28T17:35:16.847Z,1627493716.847 [CommandLine](IMPORTANT): CTD_Seabird.sea_water_salinity 33.433807 psu

Fluorometer: Confirm that LRAUV application sensor reading values for Chlorophyll (microgram_per_liter) match the expected values at the vehicle's location in the Ignition simulation.

Note: vehicle configuration must include WetLabsBB2FL.loadAtStartup 1 bool, WetLabsBB2FL.simulateHardware 1 bool

In the LRAUV application:

run RegressionTests/IgnitionTests/testScienceSensors.xml
get WetLabsBB2FL.mass_concentration_of_chlorophyll_in_sea_water

Example of expected output:

>get WetLabsBB2FL.mass_concentration_of_chlorophyll_in_sea_water
2021-07-28T17:43:27.353Z,1627494207.353 [CommandLine](IMPORTANT): got command get WetLabsBB2FL.mass_concentration_of_chlorophyll_in_sea_water
2021-07-28T17:43:27.353Z,1627494207.353 [CommandLine](IMPORTANT): WetLabsBB2FL.mass_concentration_of_chlorophyll_in_sea_water 5.039003 ug/l

Note: the sensor data values will be logged to the Slate, so you can also unserialize and review the data.

./bin/unserialize -c Logs/latest/slate CTD_Seabird.sea_water_temperature CTD_Seabird.sea_water_salinity CTD_Seabird.sea_water_pressure WetLabsBB2FL.mass_concentration_of_chlorophyll_in_sea_water

Acoustic sensors:

  • Spawn 2 vehicles at different positions in the simulation and run a RegressionTests/IgnitionTests/testAcTracking.xml to get the range (m), azimuth, and elevation angles between one vehicle to the other. Compare the reported values to expected values.
  • Confirm the dropout rante matches Ignition's configurable range depentent dropout rate using the procidure above at 100 m, 1000 m, 1500 m, 2500 m.

Note: vehicle configuration must include DAT.loadAtStartup 1 bool, DAT.simulateHardware 1 bool, DAT.ignoreElevationAngle 0 bool

Variables to check:

DAT.azimuth_vehicleFrame (radian)
DAT.elevation_vehicleFrame (radian)
DAT.acoustic_contact_range (meter)

Example:

get DAT.acoustic_contact_range meter
2021-08-04T04:42:29.187Z,1628052149.187 CommandLine: got command get DAT.acoustic_contact_range meter
2021-08-04T04:42:29.187Z,1628052149.187 CommandLine: DAT.acoustic_contact_range 454.192871 m

Unserialize:

./bin/unserialize -c Logs/latest/slate DAT.acoustic_contact_range DAT.azimuth_vehicleFrame DAT.elevation_vehicleFrame

Acceptance tests

run Science/circle_acoustic_contact.xml

run Science/trackPatchChl_yoyo.xml

mabelzhang commented

Took a first look regarding what infrastructure is needed. @caguero

  • To test these in the way they're worded now, we would need to add an interface to spawn the vehicle(s) at different locations.
    Currently, the vehicle startup location is hardcoded in the SDF.
    That interface could be done

    • via the ign-launch Ruby way (ugly syntax), or
    • via a plugin.

    Alternatively, instead of adding an interface, it might be simpler to

    • move the vehicle via control commands on a known trajectory along which we keep checking the sensor values, or
    • teleport the vehicle, if moving large distances is too slow.

  • Corollary to the bullet above: For the acoustic sensor, we'd need to spawn 2 vehicles.

  • Sensor values

    LRAUV application sensor reading values . . . match the expected values . . . in the Ignition simulation

    This needs some scoping. How do we want to define the expected values? One complication I see is that the LRAUV sim values are loaded and interpolated using algorithms that assume the data is sorted, i.e. netCDF format. Unless we use the same algorithms in the Ignition plugin, the values probably will not match.

    Is it good enough if it's a reasonable interpolation, or do we really want the numbers to match within some delta?

    Update: I'm rereading what I quoted and I'm interpreting it differently. I think it's just saying we need to check the protobuf message communication is working – that the MBARI side is receiving the value we computed on the Ignition side. Is that interpretation correct?

  • Double-checking config files @braanan

    Note: vehicle configuration must include CTD_Seabird.loadAtStartup 1 bool, CTD_Seabird.simulateHardware 1 bool
    Note: vehicle configuration must include WetLabsBB2FL.loadAtStartup 1 bool, WetLabsBB2FL.simulateHardware 1 bool

    Which file should we be looking at? I see them in Config/Science.cfg and one in Config/sim/Science.cfg

    Note: vehicle configuration must include DAT.loadAtStartup 1 bool, DAT.simulateHardware 1 bool, DAT.ignoreElevationAngle 0 bool

    Does it suffice to look at Config/Sensor.cfg?

chapulina commented

interface to spawn the vehicle(s) at different locations.

For manual testing, another option is to spawn the vehicle with an ign service call. For automated testing, we can call the create service from the test itself.

Another alternative I can think of is leaving the vehicle at the origin and changing the coordinates of the origin itself (i.e. #81)

Finally, maybe it wouldn't be too crazy to have SDF files specific to each mission? 🤔

mabelzhang commented

SDF files specific to each mission?

😅 I would put that as last resort... we did that for the multi-LRAUV, and there are many many magic numbers that are copied over half a dozen times, with the only difference being the vehicle name. Arjo wouldn't like to update all of them whenever we change hydrodynamics.

Although... it IS the simplest... but the most untidy.

chapulina commented

many many magic numbers that are copied over half a dozen times

Are they mostly within plugins? I think we may be able to leverage server config files for that. They have a similar syntax to ign-launch.

mabelzhang commented

Yes they are, all in buoyant_tethys.sdf.
Essentially, each vehicle needs these 200 lines:
https://github.com/osrf/lrauv/blob/2bf29303ba568cbd5ab8198855bb6370b46efda4/lrauv_ignition_plugins/worlds/buoyant_tethys.sdf#L173-L369

Server config looks promising. We can just add an export to the colcon hooks for that env var.
Will we be able to, say, have a new acoustic_tracking.sdf that instantiates 2 LRAUVs, leave all the plugin parameters above blank?
Right now, that's 10 plugins and 4 custom sensors.

chapulina commented

these 200 lines:
10 plugins and 4 custom sensors.

If we don't expect those to change for different missions, we could just hardcode them into lrauv_description.

If they may change according to the mission, we could have a few different server config files that we install and choose at runtime the appropriate one for that mission.

Will we be able to, say, have a new acoustic_tracking.sdf that instantiates 2 LRAUVs, leave all the plugin parameters above blank?

Yeah I think we can accomplish this with either of the options above

mabelzhang commented

hardcode them into lrauv_description.

We actually moved all the plugins out of lrauv_description because we wanted to keep that package just the model description, and the _plugins package holding everything else, to follow conventions. We also upload the model.sdf in lrauv_description to Fuel. In the ign-gazebo examples is where we supply the plugins.

  • 👍1
chapulina commented

spawn the vehicle(s) at different locations.

@mabelzhang , on #90 I added a callback for the SimInitStruct, which I believe is called once for every vehicle. If that assumption is right, I think we can use that to spawn or position the vehicles at startup.

On that PR, I'm currently assuming a single vehicle and setting its lat / lon to be the origin of the world. But once we have multiple vehicles, I think we could set the world origin to coincide with the first vehicle and spawn the 2nd with respect to that one.

Let me know what you think.

mabelzhang commented

Checklist for values tested in the original post:

  • temperature #97
  • salinity (value must be in range (28, 38), bound on MBARI side, see comment below)
  • pressure #91, in values[1]. (not written to Slate - need to figure out on MBARI side)
  • chlorophyll #97, in values[0]
  • acoustic comms integration test
mabelzhang commented

@braanan The integration test asks for get CTD_Seabird.sea_water_pressure, but ocean pressure doesn't seem to be directly written to the slate.
ExternalSimIgnition.cpp calculates the pressure, which is used to calculate the command, but doesn't write it to the slate:

    float pressure( AuvMath::OceanPressure( depth, latitude ) );

Could you help us look into where we could put ocean pressure into CTD_Seabird.sea_water_pressure?
This is in addition to the salinity value mismatch in CTD_Seabird.sea_water_salinity. We are writing a value of 2 in the protobuf message results_.salinity_ to SimSlate::SALINITY_PART_PER_THOUSAND, but the sensor returns 28.

mabelzhang commented

@chapulina Vehicle namespacing from config params is now implemented on the MBARI side.

Test default vehicle name

Use commit hash newer than e0cfd9b2ddff8246bb4a28756e949994cfb22dab on branch feature/2021-02-12-ignition-sim.
Remember to update submodules recursively to get the new Config:

git submodule update --recursive

This should get you a Config/vehicle.cfg with Vehicle.name = "tethys"; (note lowercase t).

Test it out:

$ make ignition
$ bin/LRAUV
$ ign topic -l
/tethys/command_topic

Test a different vehicle name

Make a copy of the MBARI repo locally (this is the way MBARI recommended, directory is ~1.9 GB):

cp -r lrauv-application lrauv-application-2

In the new directory, edit lrauv-application-2/Config/vehicle.cfg, modify Vehicle.name, for example

   Vehicle.name            = "daphne";

Don't commit from this new directory.

Test it out, you should get daphne in the Ignition topic name:

$ cd lrauv-application-2
$ bin/LRAUV
$ ign topic -l
/daphne/command_topic
  • 👍1
braanan commented

@mabelzhang re salinity, no conversions from ppt to psu made at the driver level or anywhere else, so salinity values read in by ExternalSimIgnition should be unchanged.

Also, don't worry about integrating the pressure measurement from Ign in the lrauv-app for now. The CTD driver can take care of those conversions and we can take care of integrating pressure later on. Thanks

  • 👍1
mabelzhang commented

@braanan I'm seeing this comment in the code:

// Looks like we're reading the simulated salinity in PPT, but using it as PSU

It's in multiple files:

$ grep -rI SALINITY_PART_PER_THOUSAND .
./Source/scienceModule/deprecated/CTD_SeabirdLCM.cpp:            && SimSlate::Read( SimSlate::SALINITY_PART_PER_THOUSAND, salinity_ ) // Looks like we're reading the simulated salinity in PPT, but using it as PSU
./Source/scienceModule/CTD_NeilBrown.cpp:            && SimSlate::Read( SimSlate::SALINITY_PART_PER_THOUSAND, salinity_ ) // Looks like we're reading the simulated salinity in PPT, but using it as PSU
./Source/scienceModule/CTD_Seabird.cpp:            && SimSlate::Read( SimSlate::SALINITY_PART_PER_THOUSAND, salinity_ ) // Looks like we're reading the simulated salinity in PPT, but using it as PSU
./Source/simulatorModule/ExternalSimIgnition.cpp:    SimSlate::Write( SimSlate::SALINITY_PART_PER_THOUSAND, results_.salinity_ );
./Source/simulatorModule/ExternalSim.cpp:    SimSlate::Write( SimSlate::SALINITY_PART_PER_THOUSAND, results_.salinity_ );
./Source/data/SimSlate.cpp:    case SALINITY_PART_PER_THOUSAND:
./Source/data/SimSlate.cpp:        return "SALINITY_PART_PER_THOUSAND";
./Source/data/SimSlate.h:        SALINITY_PART_PER_THOUSAND, // Used in CTDs

In ExternalSimIgnition, the printouts show that we are getting the correct value (2) in lrauv-app, but the request for salinity immediately after is incorrect (28). It probably has to do with the comment above?

2021-09-29T00:59:09.325Z,1632877149.325 [ExternalSim](INFO): Received state from Ignition (printed only once in a while):
2021-09-29T00:59:09.325Z,1632877149.325 [ExternalSim](INFO):   header.stamp.sec: 7
2021-09-29T00:59:09.325Z,1632877149.325 [ExternalSim](INFO):   header.stamp.nsec: 0
2021-09-29T00:59:09.325Z,1632877149.325 [ExternalSim](INFO):   temperature: 1.000027
2021-09-29T00:59:09.325Z,1632877149.325 [ExternalSim](INFO):   salinity: 2.000007
2021-09-29T00:59:09.325Z,1632877149.325 [ExternalSim](INFO):   density: 0.000000
2021-09-29T00:59:09.325Z,1632877149.325 [ExternalSim](INFO):   values[0]: 2.999989
2021-09-29T00:59:09.325Z,1632877149.325 [ExternalSim](INFO):   (some fields omitted in printout)
ead9b2bbb561>get CTD_Seabird.sea_water_temperature
2021-09-29T00:59:09.460Z,1632877149.460 [CommandLine](IMPORTANT): got command get CTD_Seabird.sea_water_temperature
2021-09-29T00:59:09.460Z,1632877149.460 [CommandLine](IMPORTANT): CTD_Seabird.sea_water_temperature 1.000024 degC
ead9b2bbb561>get CTD_Seabird.sea_water_salinity
2021-09-29T00:59:09.463Z,1632877149.463 [CommandLine](IMPORTANT): got command get CTD_Seabird.sea_water_salinity
2021-09-29T00:59:09.463Z,1632877149.463 [CommandLine](IMPORTANT): CTD_Seabird.sea_water_salinity 28.000000 psu
ead9b2bbb561>get WetLabsBB2FL.mass_concentration_of_chlorophyll_in_sea_water
2021-09-29T00:59:09.467Z,1632877149.467 [CommandLine](IMPORTANT): got command get WetLabsBB2FL.mass_concentration_of_chlorophyll_in_sea_water
2021-09-29T00:59:09.467Z,1632877149.467 [CommandLine](IMPORTANT): WetLabsBB2FL.mass_concentration_of_chlorophyll_in_sea_water 2.999989 ug/l
braanan commented

Hmm... Maybe, though 1 ppt = 1 psu so that conversion should be meaningless — I'll take a quick look and let you know what I find. Thanks

braanan commented

OK @mabelzhang, so looks like you're running into an issue with some hardcoded value bounds in the CTD_Seabird driver.

You can see that in CTD_Seabird::getSimulatedData() line 602-603 the value is bounded and overwritten:

        // Truncate out-of-bounds salinity for sim
        if( salinity_ < minSalinityBound_ ) salinity_ = minSalinityBound_;
        if( salinity_ > maxSalinityBound_ ) salinity_ = maxSalinityBound_;

Sure enough, these values are defined in the class constructor to be:

      maxSalinityBound_( 38 ),
      minSalinityBound_( 28 ),

Looks like the salinity data in the csv file should all satisfy those bounds, so maybe choose a different value for the test?
image

  • 👍1
mabelzhang commented

@braanan Thank you for looking into that!!

I picked a number (33) within the (28, 38) range to test, and the query on CTD_Seabird.sea_water_salinity is now correct:

2021-09-29T04:27:18.195Z,1632889638.195 [ExternalSim](INFO): Received state from Ignition (printed only once in a while):
2021-09-29T04:27:18.195Z,1632889638.195 [ExternalSim](INFO):   header.stamp.sec: 20
2021-09-29T04:27:18.195Z,1632889638.195 [ExternalSim](INFO):   header.stamp.nsec: 0
2021-09-29T04:27:18.195Z,1632889638.195 [ExternalSim](INFO):   temperature: 1.000018
2021-09-29T04:27:18.195Z,1632889638.195 [ExternalSim](INFO):   salinity: 33.000019
2021-09-29T04:27:18.195Z,1632889638.195 [ExternalSim](INFO):   density: 0.000000
2021-09-29T04:27:18.195Z,1632889638.195 [ExternalSim](INFO):   values[0]: 3.000007
2021-09-29T04:27:18.195Z,1632889638.195 [ExternalSim](INFO):   (some fields omitted in printout)
ead9b2bbb561>get CTD_Seabird.sea_water_temperature
2021-09-29T04:27:18.621Z,1632889638.621 [CommandLine](IMPORTANT): got command get CTD_Seabird.sea_water_temperature
2021-09-29T04:27:18.621Z,1632889638.621 [CommandLine](IMPORTANT): CTD_Seabird.sea_water_temperature 0.999994 degC
ead9b2bbb561>get CTD_Seabird.sea_water_salinity
2021-09-29T04:27:18.624Z,1632889638.624 [CommandLine](IMPORTANT): got command get CTD_Seabird.sea_water_salinity
e2021-09-29T04:27:18.624Z,1632889638.624 [CommandLine](IMPORTANT): CTD_Seabird.sea_water_salinity 33.000004 psu
ead9b2bbb561>get WetLabsBB2FL.mass_concentration_of_chlorophyll_in_sea_water
2021-09-29T04:27:18.629Z,1632889638.629 [CommandLine](IMPORTANT): got command get WetLabsBB2FL.mass_concentration_of_chlorophyll_in_sea_water
2021-09-29T04:27:18.629Z,1632889638.629 [CommandLine](IMPORTANT): WetLabsBB2FL.mass_concentration_of_chlorophyll_in_sea_water 2.999984 ug/l
  • 🚀1