KxSystems/pykx

Addition of isnull, isna, notnull, notna, idxmax, idxmin, kurt and sem functions

nipsn opened this issue · 5 comments

nipsn commented

name: Addition of isnull, isna, notnull, notna, idxmax, idxmin, kurt and sem functions.
about: Missing pandas API functionality
title: 'Addition of isnull, isna, notnull, notna, idxmax, idxmin, kurt and sem functions.'
labels: ''
assignees: '@nipsn @tortolavivo23 @neutropolis @MiguelGomezC'

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like
An implementation of said functions.

Describe alternatives you've considered

Additional resource
Links to pandas documentation of said functions:

rianoc-kx commented

https://pypi.org/project/pykx/2.5.1/

https://code.kx.com/pykx/2.5/release-notes/changelog.html#pykx-251

Additions
Pandas API additions: isnull, isna, notnull, notna, idxmax, idxmin, kurt, sem.

Thanks for the contributions!

rianoc-kx commented

Some small changes I made you will see in the final 2.5.1 MR https://github.com/KxSystems/pykx/pull/31/files

src/pykx/pandas_api/pandas_meta.py

Addition of these lines so behaviour with Keyed tables is consistent across the APIs.
.i.e the operation only runs on the value columns, not the key columns

        if 'Keyed' in str(type(tab)):
            tab = q('value', tab)

tests/test_pandas_api.py
I removed the symbol column when testing with the numeric_only=False default value.
This is because we support both Pandas 1.* and 2.* but 1.* tests were failing because sorting string columns were not allowed.

Pandas 2.1.4

>>> import pykx as kx
>>> tab = kx.q('([] sym: 100?`foo`bar`baz`qux; price: 250.0f - 100?500.0f; ints: 100 - 100?200)')
>>> tab.pd().idxmin()
sym       0
price    22
ints     44
dtype: int64

Pandas 1.5.3

>>> import pykx as kx
>>> tab = kx.q('([] sym: 100?`foo`bar`baz`qux; price: 250.0f - 100?500.0f; ints: 100 - 100?200)')
>>> tab.pd().idxmin()
...
TypeError: reduction operation 'argmin' not allowed for this dtype
nipsn commented

Hi Rian!
Thanks for keeping us up to date with these changes.

I have one question then: I noticed that in the README you mention how pandas>=1.2, < 2.2.0 versions are supported. However, when setting up the dev environment, pip did of course only pull the latest version that matched the specification.

Do you know of a way to be able to test both pandas major versions at once? The best solution that comes to mind would be to have two Python virtual environments with these two major versions and switching back and forth, but it might turn cumbersome in the long run...

rianoc-kx commented

We have gitlab testing pipelines which run in both versions as soon as we push.
We do also test manually by uninstalling one version and installing the other locally as well.

To make this less cumbersome a function could be added to your ~/.bashrc:

p1p2 () {
 python -m pip install pandas==1.5.3 && \
 eval "$@" && \
 python -m pip install pandas==2.1.4 && \
 eval "$@"
}

Then the test will run on both versions:

p1p2 python -m pytest -vvv -n 0 --no-cov --junitxml=report.xml tests/test_pandas_api.py::test_pandas_idxmin

We'll research/think about making this easier going forward

  • 🚀1
nipsn commented

That checks out!
I'll try it next time around