Parameters to reproduce results from the paper

[mpvartak]

Hi folks,

Thanks for sharing this code! It looks like the gen_diff.py script takes a large number of arguments (https://github.com/peikexin9/deepxplore/blob/master/MNIST/gen_diff.py#L21) many of which are not optional. Can you share the values you used to generate the results in the paper? And also, can you comment on what are reasonable values for each of these arguments?

Manasi

[peikexin9]

Manasi, in ImageNet/MNIST a reasonable command is "python gen_diff.py occl 1 0.1 10 1 50 0"
To describe the parameters sequentially,
"occl" is for occlusion transformation,
1 is for balancing weight between different NNs such that one NN we aim to make it predict differently can have adjustable weights. (highly tunable and doesn't have the best value, we usually use 1)
0.1 is for weight parameter that balance objectives of achieving discrepancies and neuron coverage (highly tunable and doesn't have the best value, we usually use 0.1),
10 is the step size during gradient descent of generating inputs (we usually use 10, but I remember I tried 1 or other values and both can generate inputs)
1 is the number of seed input based on which you generate difference-inducing input (you can pick arbitrary values),
50 is number of iterations in gradient descent (you can try a different number of iterations, usually the larger number of iterations the more likely to generate difference-inducing inputs)
0 is the threshold for neuron coverage (you can pick in [0,1]).

Thanks!

[mpvartak]

Cool, thanks! Could you provide the values for the remaining datasets too, eg driving, PDF

[peikexin9]

Example for your reference:
MINIST: python gen_diff.py blackout 1 0.1 10 20 50 0
ImageNet: python gen_diff.py occl 1 0.1 10 20 50 0
Driving: python gen_diff.py light 1 0.1 10 20 50 0
PDF: python gen_diff.py 2 0.1 10 20 50 0
Drebin: python gen_diff.py 1 0.5 20 50 0

[mpvartak]

Thanks!