VISION-SJTU/RECCE

Unexpected performance drop when not using provided test code.

deepfakeLearnerHi opened this issue · 5 comments

deepfakeLearnerHi commented

Reproducing with a bit more data (20 frames per video), I got a fairly good checkpoint. However, when I evaluated the checkpoint without the provided test code, the performance dropped unexpected. I wonder if I did something wrong.
Here's what I did:

  1. Load my test data
  2. Perform augmentation: images normalized with mean an std both [0.5, 0.5, 0.5]; images resized to (299, 299)
  3. Labels: Real=0, Fake=1
  4. Put all data to dataloader
  5. Feed all data from dataloader to the model and calculate the ACC and AUC.

Please let me know if I am missing any important step.
And it will be very helpful if a well-trained checkpoint could be provided.

Blosslzy commented

Hi, can your good checkpoint get the same results as shown in the paper?

XJay18 commented

Hi, actually I cannot find any important missing steps based on the description. I suggest modifying the provided test code line by line to find the reason.

deepfakeLearnerHi commented

Found the problem although I don't know why:
I did augmentations using the torchvision.transforms, which led to the bad results.
However, when I used the augmentation which the source code did, it worked just fine.

I think your source code is good though, something must be wrong on my side.
Thanks for your time!

XJay18 commented

Hi, thanks for reporting the reason. I guess it may result from the different implementations of them.
Best wishes

xarryon commented

Reproducing with a bit more data (20 frames per video), I got a fairly good checkpoint. However, when I evaluated the checkpoint without the provided test code, the performance dropped unexpected. I wonder if I did something wrong. Here's what I did:

  1. Load my test data
  2. Perform augmentation: images normalized with mean an std both [0.5, 0.5, 0.5]; images resized to (299, 299)
  3. Labels: Real=0, Fake=1
  4. Put all data to dataloader
  5. Feed all data from dataloader to the model and calculate the ACC and AUC.

Please let me know if I am missing any important step. And it will be very helpful if a well-trained checkpoint could be provided.

Hi, could u share the detection performance you get with more frames? Thanks!