gradient value is very small in energy constraints situation

[OswaldoBornemann]

I tried to implement the knowledge network in the energy constraints situation on my own task. However, I found that the grad from the get_mean_shift function is very small, which will not have any effect on the original mean. For example, the grad values are below:

tensor([[[[ 2.7554e-08]],

         [[-8.8887e-09]],

         [[ 8.4224e-09]],

         ...,

         [[ 5.1067e-12]],

         [[ 3.8011e-11]],

         [[-2.2646e-12]]],


        [[[-5.8177e-09]],

         [[ 1.4201e-09]],

         [[-2.9926e-10]],

         ...,

         [[ 7.6124e-12]],
...
         [[-4.7238e-11]],

         [[-5.8617e-11]],

         [[ 4.8769e-11]]]], device='cuda:0')

So what is your case?

[gaozhihan]

Thank you for your interest in our work and your question. In our experiments, we have observed the norm of the grad is typically around $10^{-4}$. This value is not extremely small compared with the original mean, which is around $10^{-1}$. To ensure the effectiveness, we also use a rather large scale factor.
We plan to release the code for experiments on N-body MNIST in the near future. Please stay tuned for updates and more details.

[OswaldoBornemann]

So your case is also in the energy constraint task? @gaozhihan

[gaozhihan]

Yes, it is the case of the experiments on N-body MNIST, with alignment of energy conservation.

[OswaldoBornemann]

I see. Thanks for your kind reply. We look forward to your released code regarding the energy constraint to revise our code. Thank you very much.