jorenham/Lmo

integer overflow in `linalg.sh_legendre` on windows

jorenham opened this issue · 0 comments

See: https://github.com/jorenham/Lmo/actions/runs/6687308385/job/18167801378

The np.int_ is a C long, which is 32bits in win64, because, well.. err... logic...?

... anyway, this is could be solved by replacing all np.int_'s by np.int64.

Another option could be to switch _sh_jacobi_f in the cases where _sh_jacobi_i will overflow given k and dtype in
sh_legendre, and similarly in sh_jacobi(k, a, b, dtype).

Note that in practise, this is only a problem in _lm._l_weights_pwm. The _lm._l_weights_ostat method is slower, slightly less precise, but a lot less likely to overflow.

For any users facing this issue; the workaround is to use e.g. trim=1e-15 instead of trim=0. This forces it to use the ostat weights, since the pwm weights can only handle integer trimming.