Pandas v2.1.0 broke Market Calendars
davidlatte opened this issue · 5 comments
Greetings devs!
There is a critical bug/dependency with pandas_market_calendars and pandas itself that is causing this module to be completely broken. It appears that the new version of pandas has changed something under the hood that is no longer compatible with some of your data structures. I have verified that this is failing on both Windows and Linux machines. Please help!
Test Case
Upgrade Pandas to v2.1.0
pip install pandas --upgrade
Then simply run your own Quick Start code
import pandas_market_calendars as mcal
# Create a calendar
nyse = mcal.get_calendar('NYSE')
# Show available calendars
print(mcal.get_calendar_names())
early = nyse.schedule(start_date='2012-07-01', end_date='2012-07-10')
earlyError Log
File "C:\Users\User\venv\Lib\site-packages\pandas_market_calendars\holidays_nyse.py", line 306, in july_5th_holiday_observance
return datetime_index[datetime_index.year < 2013]
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'Timestamp' object is not subscriptable
I am not involved in pandas_market_calendars day to day so would not call myself one of the devs, but I am unable to reproduce in a fresh venv on Windows using python 3.11.3:
(venv311) MINGW64 ~/code/open-source/pandas_market_calendars_env
$ python --version
Python 3.11.3
(venv311) MINGW64 ~/code/open-source/pandas_market_calendars_env
$ pip list
Package Version
----------------------- -------
exchange-calendars 4.2.8
korean-lunar-calendar 0.3.1
numpy 1.25.2
pandas 2.1.0
pandas-market-calendars 4.2.1
pip 22.3.1
pyluach 2.2.0
python-dateutil 2.8.2
pytz 2023.3
setuptools 65.5.0
six 1.16.0
toolz 0.12.0
tzdata 2023.3
(venv311) MINGW64 ~/code/open-source/pandas_market_calendars_env
$ cat example.py
import pandas_market_calendars as mcal
# Create a calendar
nyse = mcal.get_calendar('NYSE')
# Show available calendars
print(mcal.get_calendar_names())
(venv311) MINGW64 ~/code/open-source/pandas_market_calendars_env
$ python example.py
['ASX', 'BMF', 'B3', 'CFE', 'CBOE_Futures', 'CBOE_Equity_Options', 'CBOE_Index_Options', 'CME_Equity', 'CBOT_Equity', 'CME_Agriculture', 'CBOT_Agriculture', 'COMEX_Agriculture', 'NYMEX_Agriculture', 'CME_Rate', 'CBOT_Rate', 'CME_InterestRate', 'CBOT_InterestRate', 'CME_Bond', 'CBOT_Bond', 'CMEGlobex_Livestock', 'CMEGlobex_Live_Cattle', 'CMEGlobex_Feeder_Cattle', 'CMEGlobex_Lean_Hog', 'CMEGlobex_Port_Cutout', 'CMEGlobex_FX', 'CME_FX', 'CME_Currency', 'CMEGlobex_EnergyAndMetals', 'CMEGlobex_Energy', 'CMEGlobex_CrudeAndRefined', 'CMEGlobex_NYHarbor', 'CMEGlobex_HO', 'HO', 'CMEGlobex_Crude', 'CMEGlobex_CL', 'CL', 'CMEGlobex_Gas', 'CMEGlobex_RB', 'RB', 'CMEGlobex_MicroCrude', 'CMEGlobex_MCL', 'MCL', 'CMEGlobex_NatGas', 'CMEGlobex_NG', 'NG', 'CMEGlobex_Dutch_NatGas', 'CMEGlobex_TTF', 'TTF', 'CMEGlobex_LastDay_NatGas', 'CMEGlobex_NN', 'NN', 'CMEGlobex_CarbonOffset', 'CMEGlobex_CGO', 'CGO', 'C-GEO', 'CMEGlobex_NGO', 'NGO', 'CMEGlobex_GEO', 'GEO', 'CMEGlobex_Metals', 'CMEGlobex_PreciousMetals', 'CMEGlobex_Gold', 'CMEGlobex_GC', 'GC', 'CMEGlobex_SilverCMEGlobex_SI', 'SI', 'CMEGlobex_Platinum', 'CMEGlobex_PL', 'PL', 'CMEGlobex_BaseMetals', 'CMEGlobex_Copper', 'CMEGlobex_HG', 'HG', 'CMEGlobex_Aluminum', 'CMEGlobex_ALI', 'ALI', 'CMEGlobex_QC', 'QC', 'CMEGlobex_FerrousMetals', 'CMEGlobex_HRC', 'HRC', 'CMEGlobex_BUS', 'BUS', 'CMEGlobex_TIO', 'TIO', 'CME Globex Equity', 'CME Globex Fixed Income', 'CME Globex Interest Rate Products', 'EUREX', 'HKEX', 'ICE', 'ICEUS', 'NYFE', 'NYSE', 'stock', 'NASDAQ', 'BATS', 'DJIA', 'DOW', 'IEX', 'Investors_Exchange', 'JPX', 'LSE', 'OSE', 'SIFMAUS', 'SIFMA_US', 'Capital_Markets_US', 'Financial_Markets_US', 'Bond_Markets_US', 'SIFMAUK', 'SIFMA_UK', 'Capital_Markets_UK', 'Financial_Markets_UK', 'Bond_Markets_UK', 'SIFMAJP', 'SIFMA_JP', 'Capital_Markets_JP', 'Financial_Markets_JP', 'Bond_Markets_JP', 'SIX', 'SSE', 'TSX', 'TSXV', 'BSE', 'NSE', 'TASE', 'AIXK', 'ASEX', 'BVMF', 'CMES', 'IEPA', 'XAMS', 'XASX', 'XBKK', 'XBOG', 'XBOM', 'XBRU', 'XBSE', 'XBUD', 'XBUE', 'XCBF', 'XCSE', 'XDUB', 'XFRA', 'XETR', 'XHEL', 'XHKG', 'XICE', 'XIDX', 'XIST', 'XJSE', 'XKAR', 'XKLS', 'XKRX', 'XLIM', 'XLIS', 'XLON', 'XMAD', 'XMEX', 'XMIL', 'XMOS', 'XNYS', 'XNZE', 'XOSL', 'XPAR', 'XPHS', 'XPRA', 'XSAU', 'XSES', 'XSGO', 'XSHG', 'XSTO', 'XSWX', 'XTAE', 'XTAI', 'XTKS', 'XTSE', 'XWAR', 'XWBO', 'us_futures', '24/7', '24/5']
Is there anything else in your environment?
Can you go one step farther in your test?
early = nyse.schedule(start_date='2012-07-01', end_date='2012-07-10')
earlyThis is the step that is throwing the error. Sorry if that wasn't clear.
Here is my own Windows environment for reference.
(venv) PS C:\Users\User\code\> python --version
Python 3.11.4
(venv) PS C:\Users\User\code\> pip list
Package Version
----------------------- -------
exchange-calendars 4.2.8
korean-lunar-calendar 0.3.1
numpy 1.25.2
pandas 2.1.0
pandas-market-calendars 4.2.1
pip 23.1.2
pyluach 2.2.0
python-dateutil 2.8.2
pytz 2023.3
setuptools 65.5.0
six 1.16.0
toolz 0.12.0
tzdata 2023.3
This looks the same as this issue with exchange_calendars.
My apologies, I get the error when I go a step further. There appears to be a change in how pandas handles Index.map() it appears to no longer pass through the index and then fallback to value-by-value calls. The below code run in pandas 2.0.3 but not in 2.1.0 (where the assertion fails):
import pandas as pd
from pandas.tseries.holiday import Holiday
def _observance(d):
assert isinstance(d, pd.DatetimeIndex)
return d[d.year < 2013]
h = Holiday(
# When July 4th is a Thursday, the next day is a half day prior to 2013.
# Since 2013 the early close is on Wednesday and Friday is a full day
"Fridays after Independence Day prior to 2013",
month=7,
day=5,
days_of_week=(4,), # FRIDAY
observance=_observance,
start_date=pd.Timestamp("1996-01-01"),
)
result = h.dates(pd.Timestamp("1991-01-01"), pd.Timestamp("2023-01-01"))
print(result)
I believe there are only two places where the observance expects an index (both july_5th_holiday_observance defs). However, there are a number of other unit tests failing with the latest pandas due to other small changes. I have most of them passing, but am trusting the tests to guide me.