Missing labels for some viewpoints

[chenguanqi]

Hi!
Thank you for open-sourcing the code and data, this work is very interesting and significant.

When using the provided annotations, I found that some landmark and action words do not have their corresponding viewpoints. For example, the following picture is the visualization of the case whose instr_id is 2804_2. This picture only shows the information of the annotated action words. If the viewpoint does not contain its corresponding action words, I just let the first token "[CLS]" be the action word of this viewpoint. Thus, in this case, only the first viewpoint contains its action words. The verbs like "turn right" and "walk up" do not have their corresponding viewpoints.

Is this normal? If it is abnormal, could you please check the data?

Thanks again!

[cwhao98]

@chenguanqi

Hi Guanqi,

Thanks for your interest!

Firstly, indeed, some few viewpoints don't have their corresponding annotations. This mainly can be attributed to

1. repeatability. The sub-instructions in some viewpoints are similar to the previous ones, e.g, go up the stairs all the way. For the case what you mentioned above, the agent is required to walk up the stairs at the begining. However, the later instruction is still 'walk up the stairway and down the hallway', the main thing to do is to go up the stairs. Also, in some scenes, several stairs are joined together, so we did not mark them.

2. ambiguity. Some scenes are just too complicated, and the instructions are not so easy to discern. In this case, we think it may be better not to limit it to focus on specific parts, but to combine the overall information.

Secondly, in our implementation, when calculating loss, a binary mask is used to represent whether there is annotation at each viewpoint. Hope it helps you.

Thanks again.

[chenguanqi]

Thank you for your reply. It is very helpful!