groupby aggregation with `skipnan=False` differs from pandas
stress-tess opened this issue · 1 comments
stress-tess commented
while working #3762, I found that the way we handle nan
in columns when doing a groupby aggregation doesn't match pandas
with skipna=True
(the default right now) we ignore NaN
values and leave whatever the identity for that aggregation (i.e. MAX_INT when doing min, 0 when doing sum, etc). when we set this to False
, we take NaN
values into account but they override every other value (they end up being min and max in their segment.
This is not the best example, but it's the one that was popping up in my test
min aggregation with skipna
:
>>> ak_df.groupby(group_on).min()
nums2 nums3
nums1
1.0 1.000000e+00 10
3.0 1.797693e+308 9
4.0 7.000000e+00 7
>>> pd_df.groupby(group_on).min()
nums2 nums3
nums1
1.0 1.0 10
3.0 NaN 9
4.0 7.0 7
min aggregation without skipna
:
>>> ak_df.groupby(group_on).min()
key1 key2 nums1 nums2 nums3
0 valuew valueA 1.0 1.0 9
1 valuex valueB NaN NaN 7
>>> pd_df.groupby(group_on).min()
key1 key2 nums1 nums2 nums3
0 valuew valueA 1.0 1.0 9
1 valuex valueB 4.0 7.0 7
stress-tess commented
I'm unable to reproduce this, so it appears to have been resolved at some point