figure out pixel-wise training problem

[ahundt]

pixel-wise training isn't making any progress, we need to figure out why.

Options to try fixing / possible problem sources:

• try pretraining on single prediction `delta_depth
  • DONE and got super crazy/surprising results! delta depth outperformed the levine model
  • convert the single prediction model to pixel-wise and visualize performance
  • train the pixel wise and visualize performance
• build a "pipeline validation model" that takes the answer as an additional input and make sure it is always right! #403
• Try pixel wise input on single grasp prediction models
• accessing the wrong coordinate when applying a loss (flipped axes for example)
• need to apply the loss over a larger area (gaussian loss could help)
• different input than [delta_depth, sin(theta), cos(theta)] might produce better results
• are pixel-wise models set up correctly? Try pascal_voc as a baseline to check
• should we be doing a fixed crop + resize in eval, perhaps a scale difference is the problem?
• xyz images and thus delta depth values not correct? See warning below:

WARNING: expected 10 time steps but found 0 in feature: move_to_grasp/time_ordered/xyz_image/preprocessed in
dataset 097  File "grasp_train.py", line 641, in <module>
    main()
 check get_training_dictionaries().  File "grasp_train.py", line 636, in main
    model_name=FLAGS.grasp_model)
 check get_training_dictionaries().  File "grasp_train.py", line 459, in eval
    grasp_sequence_max_time_step=grasp_sequence_max_time_step)
 check get_training_dictionaries().  File "/home/ahundt/src/costar_ws/src/costar_plan/costar_google_brainrobo
tdata/grasp_dataset.py", line 1962, in get_training_tensors
    random_crop_dimensions=random_crop_dimensions, random_crop_offset=random_crop_offset)
 check get_training_dictionaries().  File "/home/ahundt/src/costar_ws/src/costar_plan/costar_google_brainrobo
tdata/grasp_dataset.py", line 1842, in get_training_dictionaries
    ' check get_training_dictionaries().'.join(traceback.format_stack()))

[ahundt]

progress has been made in #410