Bears-R-Us/arkouda

groupby aggregation with `skipnan=False` differs from pandas

stress-tess opened this issue · 1 comments

while working #3762, I found that the way we handle nan in columns when doing a groupby aggregation doesn't match pandas

with skipna=True (the default right now) we ignore NaN values and leave whatever the identity for that aggregation (i.e. MAX_INT when doing min, 0 when doing sum, etc). when we set this to False, we take NaN values into account but they override every other value (they end up being min and max in their segment.

This is not the best example, but it's the one that was popping up in my test

min aggregation with skipna:

>>> ak_df.groupby(group_on).min()
               nums2  nums3
nums1                      
1.0     1.000000e+00     10
3.0    1.797693e+308      9
4.0     7.000000e+00      7

>>> pd_df.groupby(group_on).min()
       nums2  nums3
nums1              
1.0      1.0     10
3.0      NaN      9
4.0      7.0      7

min aggregation without skipna:

>>> ak_df.groupby(group_on).min()
     key1    key2  nums1  nums2  nums3
0  valuew  valueA    1.0    1.0      9
1  valuex  valueB    NaN    NaN      7

>>> pd_df.groupby(group_on).min()
     key1    key2  nums1  nums2  nums3
0  valuew  valueA    1.0    1.0      9
1  valuex  valueB    4.0    7.0      7

I'm unable to reproduce this, so it appears to have been resolved at some point