clessig/atmorep

solve the performance degradation when computing the latitude weights

Opened this issue · 0 comments

When using the weighted_mse loss we see a performance degradation due to the computation of the latitude weights in multifield_data_sampler.py.

Tasks: speed up weight computation so to avoid the hiccups in the performance.

epoch: 0 [1/91 (1%)]    Loss: 1.17013 : 0.91218 :: 0.56689 (1.63 s/sec)
epoch: 0 [2/91 (2%)]    Loss: 1.25394 : 1.05042 :: 0.56396 (87.65 s/sec)
epoch: 0 [3/91 (3%)]    Loss: 1.29323 : 1.13104 :: 0.56055 (35.11 s/sec)
epoch: 0 [4/91 (4%)]    Loss: 1.10516 : 0.81147 :: 0.55713 (16.48 s/sec)
epoch: 0 [5/91 (5%)]    Loss: 1.16656 : 0.92310 :: 0.55420 (87.29 s/sec)
epoch: 0 [6/91 (7%)]    Loss: 1.31812 : 1.18307 :: 0.55176 (3.58 s/sec)
epoch: 0 [7/91 (8%)]    Loss: 1.21557 : 1.01287 :: 0.54932 (89.66 s/sec)
epoch: 0 [8/91 (9%)]    Loss: 1.11991 : 0.83303 :: 0.54639 (84.81 s/sec)
epoch: 0 [9/91 (10%)]   Loss: 1.19671 : 0.98798 :: 0.54443 (36.95 s/sec)
epoch: 0 [10/91 (11%)]  Loss: 1.17417 : 0.93769 :: 0.54199 (15.76 s/sec)
epoch: 0 [11/91 (12%)]  Loss: 1.12489 : 0.87167 :: 0.53955 (84.16 s/sec)
epoch: 0 [12/91 (13%)]  Loss: 1.15331 : 0.90734 :: 0.53857 (3.78 s/sec)