Rosbag `makerule.py` creates backwards rules instead of the required rule

[maiertopo]

Hi everyone,

I am generating migration rules for a specific message in my project that has as a sub-type another message of mine.

I have updated the main message a couple times (with migrations) and then also updated the sub-message by adding a new field to it without changing anything else in the main message. When generating the rules for this step, makerule.py only created the rule for updating the sub-message but no specific rule for the main message.

Let's say my main message is

bool main_a
MySubMsg[] main_msgs

And MySubMsg is

bool sub_a

In this initial state let's further assume (for simplicity's sake) that the main message has the checksum a0 and the sub-message has checksum b0.

My initial step was to update the main message like so

bool main_a
bool main_b
MySubMsg[] main_msgs

and create the migration rule with

$ rosrun rosbag makerule.py MainMessage.saved main_rule_0.bmr

Now the latest main message has checksum a1.

Later I updated the sub-message to

bool sub_a
float32 sub_b

and created another rule file (sub_rule_0.bmr), checksum of sub-message is now b1. It's worth mentioning that no matter if I create a rule for the saved type of the main message or the sub-message, the same rule is created that migrates the sub-type to b1.

Up to this point everything seems to work as intended, migrating old bag files works and they get the respective fields updated according to the migration rules files.

I tried creating a dedicated rule for the main message, since the sub-type changed while the rest hasn't.
When calling makerule.py it would tell me that no rule file is necessary so it won't be generated. In the following call MainMessage2.saved is the saved message that does not contain the updated sub-type yet.

$ rosrun rosbag makerule.py MainMessage2.saved main_rule_1.bmr
The following migrations need to occur:
 * From: my_msgs/MainMessage [a1]
   To:   my_msgs/MainMessage [a2]

No additional rule files needed to be generated.  main_rule_1.bmr not created.

Now I decided to add a new field to the main message

bool main_a
bool main_b
bool main_c
MySubMsg[] main_msgs

so I need the migration rule from a2 to the new checksum a3.

But since the migration from a1 to a2 was not generated, the behavior of makerule.py is not as I would have expected and a lot of unnecessary and potentially dangerous migration rules are created.

$ rosrun rosbag makerule.py MainMessage3.saved main_rule_1.bmr  
WARNING: Within rule [GENERATED.update_my_msgs_MainMessage_a2] cannot migrate from subtype [MySubMsg] to [MySubMsg]..
WARNING: Within rule [GENERATED.update_my_msgs_MainMessage_a2] cannot migrate from subtype [MySubMsg] to [MySubMsg]..
The following migrations need to occur:
 * From: my_msgs/MainMessage [a2]
   To:   my_msgs/MainMessage [a3]
    3 rules missing:
     * From: my_msgs/MainMessage [a2]
       To:   my_msgs/MainMessage [a0]
     * From: my_msgs/MySubMsg [b1]
       To:   my_msgs/MySubMsg [b0]
     * From: my_msgs/MainMessage [a1]
       To:   my_msgs/MainMessage [a3]

The necessary rule files have been written to: main_rule_1.bmr

So as expected, it tries to generate the migration rule from a2 to a3. The migration from a1 to a2 was never generated so the solution of makerule.py is to revert most of the changes, going back to the initial checksums in both message types (a0 and b0), and finally generating the rule to go from a1 to a3 directly which includes the update of the sub-message.

This is dangerous in my opinion, because a bag file that is for example only missing the newly added main_c field will potentially lose information about e.g. main_b and sub_b when it is migrated since they are not present in the old state.

The solution I found is to create the file main_rule_1.bmr for the missing rule from a1 to a2 by hand which leads to the correct behavior, as makerule.py then is able to only generate the required rule from a2 to a3 when updating MainMessage.

$ rosrun rosbag makerule.py MainMessage3.saved main_rule_2.bmr
The following migrations need to occur:
 * From: my_msgs/MainMessage [a2]
   To:   my_msgs/MainMessage [a3]
    1 rules missing:
     * From: my_msgs/MainMessage [a2]
       To:   my_msgs/MainMessage [a3]

The necessary rule files have been written to: main_rule_2.bmr

So the workflow in this case is:

1. Change sub-type (including its migration rule) without changing the MainMessage .
2. Try to generate the rule file for MainMessage with makerule.py.
3. If makerule.py does not generate the file, create the rule file manually with the checksum that makerule.py output suggests.
4. Further migrations should work as intended.

As mentioned I also tried directly generating the rule file for MainMessage after the change in the MySubMsg. This effectively generates the identical rule file as if you would just migrate the sub-messages directly, so the required migration rule (a1 -> a2) is still missing.

I might be using this in an incorrect way, but I think I exhausted the options I have with makerule.py (I have been following the official tutorial at http://wiki.ros.org/rosbag/migration).