sky-uk/anticipy

interpolate_df() changes datetime unit of date column

capelastegui opened this issue · 0 comments

We have seen forecasts crashing with the following error:

  File "/home/jupyter_user/workspace/nsa.notebooks-voice/virtual/lib/python3.9/site-packages/anticipy/forecast.py", line 928, in run_forecast
    return run_forecast_single(df_y,
  File "/home/jupyter_user/workspace/nsa.notebooks-voice/virtual/lib/python3.9/site-packages/anticipy/forecast.py", line 1386, in run_forecast_single
    df_data = pd.concat(l_df_data, sort=False, ignore_index=True)
  File "/home/jupyter_user/workspace/nsa.notebooks-voice/virtual/lib/python3.9/site-packages/pandas/core/reshape/concat.py", line 393, in concat
    return op.get_result()
  File "/home/jupyter_user/workspace/nsa.notebooks-voice/virtual/lib/python3.9/site-packages/pandas/core/reshape/concat.py", line 680, in get_result
    new_data = concatenate_managers(
  File "/home/jupyter_user/workspace/nsa.notebooks-voice/virtual/lib/python3.9/site-packages/pandas/core/internals/concat.py", line 189, in concatenate_managers
    values = _concatenate_join_units(join_units, copy=copy)
  File "/home/jupyter_user/workspace/nsa.notebooks-voice/virtual/lib/python3.9/site-packages/pandas/core/internals/concat.py", line 486, in _concatenate_join_units
    concat_values = concat_compat(to_concat, axis=1)
  File "/home/jupyter_user/workspace/nsa.notebooks-voice/virtual/lib/python3.9/site-packages/pandas/core/dtypes/concat.py", line 132, in concat_compat
    return cls._concat_same_type(to_concat_eas)
  File "/home/jupyter_user/workspace/nsa.notebooks-voice/virtual/lib/python3.9/site-packages/pandas/core/arrays/datetimelike.py", line 2251, in _concat_same_type
    new_obj = super()._concat_same_type(to_concat, axis)
  File "/home/jupyter_user/workspace/nsa.notebooks-voice/virtual/lib/python3.9/site-packages/pandas/core/arrays/_mixins.py", line 230, in _concat_same_type
    return super()._concat_same_type(to_concat, axis=axis)
  File "arrays.pyx", line 190, in pandas._libs.arrays.NDArrayBacked._concat_same_type
ValueError: all the input array dimensions except for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 141 and the array at index 1 has size 616```

We have traced this back to a bug in interpolate_df() that changes the datetime unit of the date column. In this case, the input data had a date column with dtype datetime64[us], and interpolate_df() returned a date with dtype datetime64[ns]. The crash happened when concatenating both series together.