Silent mis-handling of NaNs in input data

[buhrmann]

Hi, you're probably not even maintaining this anymore, but in case you do, I've found that NaNs in the input data are not handled correctly, I think. In case you want to add something about this in the docs...

I'm not sure exactly what happens, but if I have a matrix with 3 columns, and I make some values in the third column NaNs, that will affect calculations of MI between the first two:

import infotheory
import numpy as np

it = infotheory.InfoTools(dims=3, nreps=0)
it.set_equal_interval_binning(nbins=[10, 10, 10], mins=[0, 0, 0], maxs=[1, 1, 1])

mat = np.random.default_rng(42).random((100, 3))
# mat[40:50, 2] = np.nan

it.add_data(mat)
it.mutual_info([0, 1, -1])

Without NaNs in the third variable the resulting MI between the first two variables is 0.6699268749439261. With NaNs, the result is 0.7039212101565637. So the strange thing is that data in an unrelated variable affects the MI between the first two.

After a quick Iook only, one thing that may be happening is that PyFloat_AsDouble may silently convert NaNs to -1 (https://docs.python.org/3/c-api/float.html#c.PyFloat_AsDouble) perhaps? Which would then mean that the minima passed in are not correct anymore. Though I don't know how this would then affect unrelated variables.

Let me know if you want me to help out addressing this, if you're still aiming to support this package.

Btw, I'm an old friend of Eduardo Izquierdo's ;)

[buhrmann]

Ah, interestingly, if I replace occurrences of np.nan with np.inf, at least they don't affect calculations with unrelated variables, so this may be a work-around. Also, the MI between two random variables where one contains np.infs is lower the more np.infs there are, not sure if that makes sense or not...

Another potentially helpful observation is that the calculations seem to finish much quicker in the presence of NaNs, as if some code paths or parts of the data are being skipped (noticeable with larger datasets).

[buhrmann]

FWIW, here is the output of display_snapshot() for the above code without NaNs, with NaNs and with Infs respectively:

# No missing data:
************************ SNAPSHOT ************************
Number of total datapoints added = 100
Number of populated bins = 97
Has data been collapsed across multiple shifted binnings? Yes
Size of average binned data = 97
**********************************************************

# 10 NaNs in variable with id 2:
************************ SNAPSHOT ************************
Number of total datapoints added = 100
Number of populated bins = 89
Has data been collapsed across multiple shifted binnings? Yes
Size of average binned data = 89
**********************************************************

# 10 Infs in variable with id 2:
************************ SNAPSHOT ************************
Number of total datapoints added = 100
Number of populated bins = 99
Has data been collapsed across multiple shifted binnings? Yes
Size of average binned data = 99
**********************************************************