What are the differences between unseen validation data and training data
Closed this issue · 3 comments
For behavior-CL, I think the unseen validation data and the training data of the same task (e.g. pick2&place) are generated in different environments. For example, if I assume the unseen validation data of pick2&place are all generated in [bedroom, living room], then the training ones of pick2&place are all generated in [bathroom, kitchen]. Am I correct ?
For environmental-CL, do the unseen validation data and the training data of the same environment contain non-overlapped behaviors ? For example, if I assume the unseen validation data in kitchen contains pick2&place, then the training data in kitchen does not contain pick2&place. Am I correct ?
Thank you very much.
Hi @wqshmzh,
The differences between training and unseen validation splits are environment instances (i.e., FloorPlan1, FloorPlan2, ...) used in completing tasks, following the evaluation protocol of the ALFRED benchmark. Thus, training/unseen is not necessarily related to our proposed incremental behavior and environment setups.
For behavior-CL, I think the unseen validation data and the training data of the same task (e.g. pick2&place) are generated in different environments. For example, if I assume the unseen validation data of pick2&place are all generated in [bedroom, living room], then the training ones of pick2&place are all generated in [bathroom, kitchen]. Am I correct ?
They CAN be generated in the same environment types (i.e., kitchen, livingroom, bedroom, and bathroom). For that example, the training ones of pick2&place can be generated in [bedroom, livingroom] as well, but not in the same FloorPlan's used for the unseen validation.
For environmental-CL, do the unseen validation data and the training data of the same environment contain non-overlapped behaviors ? For example, if I assume the unseen validation data in kitchen contains pick2&place, then the training data in kitchen does not contain pick2&place. Am I correct ?
They CAN contain overlapped behaviors. For that example, "kitchen" also contains pick2&place.
Hope it helps.
Closing this issue due to inactivity.
OK I see. This really helps. Thank you.