Slight difference in handling of datetime/timedelta precisions in pandas 2.0.0
Closed this issue · 6 comments
Description
Ok @d-chambers, this feels like it should be fairly trivial, but I also know that those are the things that turn into nightmares, so I want to get your take. When I fixed the website I noticed the CI was now failing on one of the tests. I've traced it to an update to pandas 2.0.0, which conveniently rolled out today and introduces some breaking changes to how it handles precisions of datetime/timedelta objects.
The particular test that failed is tests/test_utils/test_time_utils.py::TestTimeDelta::test_identity_function_on_delta_series, which now fails because the input Series has timedelta precisions of "us" while the output of to_datetime64 has a precision of "ns". This is because we are explicitly converting the output to "ns" (see obsplus.utils.time.py, line 303). This didn't fail before because earlier versions of pandas actually ignored the fact that the array used to generate the input Series had a precision of "us" and just converted it to "ns".
To Reproduce
Install a new conda environment using the yml for obsplus's test environment.
Run the tests.
Expected behavior
I'm not actually sure. My instinct is to just change the precision of the inputs in the test and move on, but I feel like there's more subtlety that should be investigated.
The 'correct' behavior would be to figure out some way to pass the expected precision down through to_timedelta64.
Versions (please complete the following information):
This is probably independent of OS/Obsplus/python versions but is caused by having pandas >= 2.0.0.
Times suck...
Let me think on it a bit. Also curious what other changes Pandas 2.0.0 made, that's a big milestone.
Let me think on it a bit.
In our conversions functions, should we just add a dispatch function which converts any timedelta64 or datetime64 to the same object with the correct precision?
I think that makes sense. Do you want to work on this, or do you want me to have a go at it?
Either way, but I probably won't have time until the semester is over in early May.
All right. I might take it when I'm bored one day.
I think I addressed this, at least partially, in #266. We can revisit if more issues crop up.