Reproducing the results

[mfawzy]

Hi,

I am trying to reproduce the same results as the paper but could not get anyway?

My settings are: 27 frames, 0.00001 lr and 2 GPUs.

Can you check if there is something missing in the settings?

[alecda573]

@mfawzy were you ever able to reproduce their results on the 27 frame model for cpn detections?

[goldbricklemon]

@mfawzy @alecda573 I ran the 27 frame experiment on cpn 2D detections multiple times, with the exact same setting from the code (2 GPUs, with a total of 2x256=512 batch size). The best I could ever achieve was some 49.xx mm MPJPE, but it was the only run with a sub-50mm result.
Apparently, no one was really able to reproduce those results so far: #4

[alecda573]

@moetherat what was the actionwise MPJPE for the model that went sub 50mm on valid error? What is the lowest you have gotten for the 81frame model if you have trained it? What were the hyperparameters that you used? Also, how many epochs did you let the 27frame model run for?

Could you also comment on the lr you used lrdecay for the 27 frame model?

[goldbricklemon]

@alecda573 I did not perform action-wise evaluation so far, unfortunately. If I do, I will let you know. For hyperparameters, I used exactly the ones stated in the readme (or the default values in the args collection):
LR: 0.00004
LR decay: 0.99
BatchSize 512

I trained for 120 epochs. For the best run, the best results were around epoch 60-80, with slight overfitting afterwards. There were, however, other runs, where 100-120 epochs were needed for convergence.
So far, I did not train the 81 frame variant.

[alecda573]

@moetherat thanks for sharing this info. You should perform action wise evaluation because there will most likely be a drop. If you are hitting 49mm evaluation error for the 27frame model it may very well drop to 47.7 for the reported 47.7 Action-wise MPJPE reported for cpn detections.
Out of curiosity have you tried using larger batch sizes or are you restricted at 512 because of your compute?

[goldbricklemon]

@alecda573 Sorry, I thought that "action-wise" MPJPE meant the per-action results (i.e. walking, sitting, etc.). What exactly do you mean with action-wise results? The per-action MPJPE values simply averaged?

I tried going larger (i.e. 1024), but found no advantage in doing so, at least not with limited experimentation regarding the LR.

[alecda573]

@moetherat If you are reporting the error as the result from the training log as 49mm error that is the frame-wise error. What is reported in the paper is the action-wise average so compute the error for each action then average.

This can be lower because there can be less frames for easier actions (Walking, Smoking, etc) and more frames for harder actions (Sitting, posing, etc).

If you run the following code this will give action wise MPJPE for protocol 1 and 2:

`
def evaluate(test_generator, action=None, return_predictions=False, use_trajectory_model=False):
epoch_loss_3d_pos = 0
epoch_loss_3d_pos_procrustes = 0
epoch_loss_3d_pos_scale = 0
epoch_loss_3d_vel = 0
with torch.no_grad():
if not use_trajectory_model:
model_pos.eval()
# else:
# model_traj.eval()
N = 0
for _, batch, batch_2d in test_generator.next_epoch():
inputs_2d = torch.from_numpy(batch_2d.astype('float32'))
inputs_3d = torch.from_numpy(batch.astype('float32'))

        ##### apply test-time-augmentation (following Videopose3d)
        inputs_2d_flip = inputs_2d.clone()
        inputs_2d_flip [:, :, :, 0] *= -1
        inputs_2d_flip[:, :, kps_left + kps_right,:] = inputs_2d_flip[:, :, kps_right + kps_left,:]

        ##### convert size
        inputs_2d, inputs_3d = eval_data_prepare(receptive_field, inputs_2d, inputs_3d)
        inputs_2d_flip, _ = eval_data_prepare(receptive_field, inputs_2d_flip, inputs_3d)

        if torch.cuda.is_available():
            inputs_2d = inputs_2d.cuda()
            inputs_2d_flip = inputs_2d_flip.cuda()
            inputs_3d = inputs_3d.cuda()
        inputs_3d[:, :, 0] = 0

        predicted_3d_pos = model_pos(inputs_2d)
        predicted_3d_pos_flip = model_pos(inputs_2d_flip)
        predicted_3d_pos_flip[:, :, :, 0] *= -1
        predicted_3d_pos_flip[:, :, joints_left + joints_right] = predicted_3d_pos_flip[:, :,
                                                                  joints_right + joints_left]

        predicted_3d_pos = torch.mean(torch.cat((predicted_3d_pos, predicted_3d_pos_flip), dim=1), dim=1,
                                      keepdim=True)

        del inputs_2d, inputs_2d_flip
        torch.cuda.empty_cache()

        if return_predictions:
            return predicted_3d_pos.squeeze(0).cpu().numpy()


        error = mpjpe(predicted_3d_pos, inputs_3d)
        epoch_loss_3d_pos_scale += inputs_3d.shape[0]*inputs_3d.shape[1] * n_mpjpe(predicted_3d_pos, inputs_3d).item()

        epoch_loss_3d_pos += inputs_3d.shape[0]*inputs_3d.shape[1] * error.item()
        N += inputs_3d.shape[0] * inputs_3d.shape[1]

        inputs = inputs_3d.cpu().numpy().reshape(-1, inputs_3d.shape[-2], inputs_3d.shape[-1])
        predicted_3d_pos = predicted_3d_pos.cpu().numpy().reshape(-1, inputs_3d.shape[-2], inputs_3d.shape[-1])

        epoch_loss_3d_pos_procrustes += inputs_3d.shape[0]*inputs_3d.shape[1] * p_mpjpe(predicted_3d_pos, inputs)

        # Compute velocity error
        epoch_loss_3d_vel += inputs_3d.shape[0]*inputs_3d.shape[1] * mean_velocity_error(predicted_3d_pos, inputs)

if action is None:
    print('----------')
else:
    print('----'+action+'----')
e1 = (epoch_loss_3d_pos / N)*1000
e2 = (epoch_loss_3d_pos_procrustes / N)*1000
e3 = (epoch_loss_3d_pos_scale / N)*1000
ev = (epoch_loss_3d_vel / N)*1000
print('Protocol #1 Error (MPJPE):', e1, 'mm')
print('Protocol #2 Error (P-MPJPE):', e2, 'mm')
print('Protocol #3 Error (N-MPJPE):', e3, 'mm')
print('Velocity Error (MPJVE):', ev, 'mm')
print('----------')

return e1, e2, e3, ev`

[goldbricklemon]

@alecda573 Yeah, I just checked the code again, thanks for the pointer. I will let you know, as soon as I find time for running this evaluation.

[alecda573]

@moetherat cheers mate, looking forward to you reporting what you get! Did you set any specific seed with torch? I am going to try and reproduce what you said, will let run for120 epochs on 27 f rame model

[sijiamo]

@alecda573 Yeah, I just checked the code again, thanks for the pointer. I will let you know, as soon as I find time for running this evaluation.

Hello,have you ever get the result as paper mentioned on 27 frame? I use the same settings like lr -0.00004, and I get the best result on 49th epoch with 48.2mm "action-wise" MPJPE. However, it is different from 47.0 mm as paper mentioned. Can you tell me about your work in details? Thank you!