Data Differences Between `trading_calendars` and `pandas_market_calendars`

Question

Data Differences Between `trading_calendars` and `pandas_market_calendars`

gerrymanoim opened this issue 4 years ago · 8 comments

gerrymanoim commented 4 years ago

re rsheftel/pandas_market_calendars#41

Summary status

Extras not covered by diff below:

Pre 1990 NYSE holidays
- #162

Details

code generating the table below

from collections import namedtuple 

import pandas_market_calendars as pmc
import trading_calendars as tc

# todo port this back to trading calendars
def get_tc_calendar_names():
  return list(tc.calendar_utils._default_calendar_factories.keys())+list(tc.calendar_utils._default_calendar_aliases.keys())

tc.get_calendar_names = get_tc_calendar_names
# end todo

pmc_calendars = set(pmc.get_calendar_names())
tc_calendars = set(tc.get_calendar_names())
tc_to_pmc_mapping = {
    'XASX': 'ASX',
    'XHKG': 'HKEX',
    'XTKS': 'JPX',
    'XOSL': 'OSE',
    'XSWX': 'SIX',
    'XSHG': 'SSE',
}

in_both = (tc_calendars & pmc_calendars) | set(tc_to_pmc_mapping.keys())

CompareResult = namedtuple("Result", ["only_tc", "only_pmc"])

def compare_calendars(cal: str):
  tc_cal = tc.get_calendar(cal).all_sessions
  pmc_cal_name = tc_to_pmc_mapping.get(cal, cal)
  pmc_cal = pmc.get_calendar(pmc_cal_name).valid_days(min(tc_cal), max(tc_cal))
  if not (tc_cal ^ pmc_cal).empty:
    return CompareResult

out = {}
for cal in in_both:
  diff = compare_calendars(cal)
  if diff:
    out[cal] = diff

# everything below this is markdown formatting
out_str = []
for cal, result in out.items():
  out_str.append(f"## {cal} \n")
  out_str.append("Trading Days only in `trading_calendars` \n")
  out_str.extend([f"- {item} \n" for item in result.only_tc])
  out_str.append("\n")
  out_str.append("Trading Days only in `pandas_market_calendars` \n")
  out_str.extend([f"- {item} \n" for item in result.only_pmc])
  out_str.append("\n")
md = "".join(out_str); print(md)

XHKG

Trading Days only in trading_calendars

1990-06-18 00:00:00+00:00
1990-08-27 00:00:00+00:00
1991-02-14 00:00:00+00:00
1991-06-18 00:00:00+00:00
1991-08-26 00:00:00+00:00
1992-06-15 00:00:00+00:00
1992-07-22 00:00:00+00:00
1992-08-31 00:00:00+00:00
1993-06-14 00:00:00+00:00
1993-08-30 00:00:00+00:00
1993-09-17 00:00:00+00:00
1994-06-14 00:00:00+00:00
1994-08-29 00:00:00+00:00
1995-06-19 00:00:00+00:00
1995-08-28 00:00:00+00:00
1996-06-17 00:00:00+00:00
1996-08-26 00:00:00+00:00
1997-02-06 00:00:00+00:00
1997-06-30 00:00:00+00:00
1997-07-02 00:00:00+00:00
1997-08-18 00:00:00+00:00
1997-10-02 00:00:00+00:00
1998-08-17 00:00:00+00:00
1998-10-02 00:00:00+00:00
1999-09-16 00:00:00+00:00
1999-12-31 00:00:00+00:00
2001-07-06 00:00:00+00:00
2001-07-25 00:00:00+00:00

Trading Days only in pandas_market_calendars

1990-05-01 00:00:00+00:00
1990-05-02 00:00:00+00:00
1990-07-02 00:00:00+00:00
1990-10-01 00:00:00+00:00
1991-05-01 00:00:00+00:00
1991-05-21 00:00:00+00:00
1991-07-01 00:00:00+00:00
1991-10-01 00:00:00+00:00
1992-05-01 00:00:00+00:00
1992-05-11 00:00:00+00:00
1992-07-01 00:00:00+00:00
1992-10-01 00:00:00+00:00
1993-05-28 00:00:00+00:00
1993-07-01 00:00:00+00:00
1994-05-02 00:00:00+00:00
1994-05-18 00:00:00+00:00
1994-07-01 00:00:00+00:00
1995-05-01 00:00:00+00:00
1995-05-08 00:00:00+00:00
1995-10-02 00:00:00+00:00
1996-05-01 00:00:00+00:00
1996-05-24 00:00:00+00:00
1996-07-01 00:00:00+00:00
1996-10-01 00:00:00+00:00
1997-05-01 00:00:00+00:00
1997-05-14 00:00:00+00:00
1998-05-01 00:00:00+00:00
1998-05-04 00:00:00+00:00
2021-04-06 00:00:00+00:00

CFE

Trading Days only in trading_calendars

Trading Days only in pandas_market_calendars

1994-04-27 00:00:00+00:00
2004-06-11 00:00:00+00:00
2007-01-02 00:00:00+00:00
2012-10-29 00:00:00+00:00
2012-10-30 00:00:00+00:00
2018-12-05 00:00:00+00:00

XSWX

Trading Days only in trading_calendars

1994-12-30 00:00:00+00:00
1995-12-29 00:00:00+00:00
2000-12-29 00:00:00+00:00
2005-12-30 00:00:00+00:00
2006-12-29 00:00:00+00:00
2011-12-30 00:00:00+00:00
2016-12-30 00:00:00+00:00
2017-12-29 00:00:00+00:00

Trading Days only in pandas_market_calendars

LSE

Trading Days only in trading_calendars

1995-05-08 00:00:00+00:00
1999-12-31 00:00:00+00:00

Trading Days only in pandas_market_calendars

1995-05-01 00:00:00+00:00

XSHG

Trading Days only in trading_calendars

2020-01-31 00:00:00+00:00
2022-01-31 00:00:00+00:00
2022-04-04 00:00:00+00:00
2022-05-02 00:00:00+00:00
2022-09-12 00:00:00+00:00
2022-10-06 00:00:00+00:00
2022-10-07 00:00:00+00:00
2023-06-23 00:00:00+00:00
2023-10-06 00:00:00+00:00
2024-02-09 00:00:00+00:00
2024-04-05 00:00:00+00:00
2024-09-16 00:00:00+00:00
2024-10-07 00:00:00+00:00
2025-01-28 00:00:00+00:00
2025-05-02 00:00:00+00:00
2025-06-02 00:00:00+00:00
2025-10-06 00:00:00+00:00
2025-10-07 00:00:00+00:00

Trading Days only in pandas_market_calendars

2020-10-08 00:00:00+00:00

XASX

Trading Days only in trading_calendars

1993-01-27 00:00:00+00:00
1994-01-27 00:00:00+00:00
1995-01-27 00:00:00+00:00
1996-01-29 00:00:00+00:00
1998-01-27 00:00:00+00:00
1999-01-27 00:00:00+00:00
2000-01-27 00:00:00+00:00
2001-01-29 00:00:00+00:00
2004-01-27 00:00:00+00:00
2005-01-27 00:00:00+00:00
2006-01-27 00:00:00+00:00
2007-01-29 00:00:00+00:00
2009-01-27 00:00:00+00:00
2010-01-27 00:00:00+00:00
2011-01-27 00:00:00+00:00
2012-01-27 00:00:00+00:00
2015-01-27 00:00:00+00:00
2016-01-27 00:00:00+00:00
2017-01-27 00:00:00+00:00
2018-01-29 00:00:00+00:00
2021-01-27 00:00:00+00:00

Trading Days only in pandas_market_calendars

1990-12-31 00:00:00+00:00
1994-01-26 00:00:00+00:00
1994-12-27 00:00:00+00:00
1995-01-26 00:00:00+00:00
1996-01-26 00:00:00+00:00
1998-01-26 00:00:00+00:00
1999-01-26 00:00:00+00:00
1999-12-31 00:00:00+00:00
2000-01-26 00:00:00+00:00
2001-01-26 00:00:00+00:00
2004-01-26 00:00:00+00:00
2005-01-26 00:00:00+00:00
2005-12-27 00:00:00+00:00
2006-01-26 00:00:00+00:00
2007-01-26 00:00:00+00:00
2009-01-26 00:00:00+00:00
2010-01-26 00:00:00+00:00
2010-04-26 00:00:00+00:00
2011-01-26 00:00:00+00:00
2011-04-26 00:00:00+00:00
2011-12-27 00:00:00+00:00
2012-01-26 00:00:00+00:00
2015-01-26 00:00:00+00:00
2016-01-26 00:00:00+00:00
2016-12-27 00:00:00+00:00
2017-01-26 00:00:00+00:00
2018-01-26 00:00:00+00:00
2021-01-26 00:00:00+00:00

TSX

Trading Days only in trading_calendars

1993-12-24 00:00:00+00:00
1999-12-24 00:00:00+00:00
2004-12-24 00:00:00+00:00
2010-12-24 00:00:00+00:00

Trading Days only in pandas_market_calendars

2001-09-11 00:00:00+00:00
2001-09-12 00:00:00+00:00

BMF

Trading Days only in trading_calendars

1997-07-09 00:00:00+00:00
2020-07-09 00:00:00+00:00

Trading Days only in pandas_market_calendars

1995-12-29 00:00:00+00:00
2000-12-29 00:00:00+00:00
2006-12-29 00:00:00+00:00
2014-06-12 00:00:00+00:00
2017-12-29 00:00:00+00:00

Answer 1 · 2020-09-04T22:12:02.000Z

cc @jmccorriston (FYI - @rsheftel )

Answer 2 · 2020-09-08T21:06:53.000Z

For TSX:

I can't find any sources saying that Christmas Eve was a holiday, but I found this source suggesting that the market was open on 2010-12-24 (but closed by 2:41pm ET, suggesting it closed early): https://toronto.ctvnews.ca/modest-tsx-gain-as-trading-takes-a-christmas-break-1.589362

There's also this wiki page on the 9/11 closes around the world (includes TSX): https://en.wikipedia.org/wiki/Closings_and_cancellations_following_the_September_11_attacks

So I think the Toronto Stock exchange implementation in trading_calendars is correct.

Answer 3 · 2020-09-08T21:18:38.000Z

FWIW https://finance.yahoo.com/quote/RY.TO/history?period1=1292112000&period2=1293753600&interval=1d&filter=history&frequency=1d shows volume on TSX on 2010-12-24.

Answer 4 · 2020-09-09T20:20:14.000Z

For HKG:

It looks like Hong Kong only started observing Labour Day in 1999 (https://en.wikipedia.org/wiki/International_Workers%27_Day, "In Hong Kong, 1 May is known as Labour Day and has been considered a public holiday since 1999."). pmc is correct here, tc needs to be updated.

Establishment day started in 1997: https://www.scmp.com/news/hong-kong/education-community/article/1983718/everything-you-need-know-about-hong-kongs-return. pmc is correct here, tc needs to be updated.

National Day also first observed in 1997: http://www.cnn.com/WORLD/9710/01/china.bday/. pmc is correct, tc needs to be updated.

Buddha's birthday also only started as a public holiday in 1998: https://www.cantoneseclass101.com/blog/2020/04/24/celebrating-buddhas-birthday-in-hong-kong/, https://www.officeholidays.com/holidays/hong-kong/the-buddhas-birthday. We should trim the calendar in tc to reflect this. However, it looks like there's still a discrepancy in 1998. pmc suggests that 1998-05-04 should be a trading day while tc suggests it should be a holiday (1998-05-03 actual holiday, observed on the Monday).

There are several British holidays that were observed during British rule pre-1997 as well:

Liberation Day: https://en.wikipedia.org/wiki/Liberation_Day_(Hong_Kong) (Last Monday of August, stopped observing in 1997.
Queen's Bday: https://en.wikipedia.org/wiki/Public_holidays_in_Hong_Kong#:~:text=Under%20the%20administration%20of%20the,the%20People's%20Republic%20of%20China.
Potentially August Bank holiday?

It looks like there are a lot of details in the pmc Hong Kong calendar implementation that we are missing in tc. I pinged the person who added them to pmc to see if we can get more info, but my inclination is to pull in the historical calendars that they added to pmc. From what I can tell so far, they are correct.

One exception: 2021-04-06 was a public holiday per https://www.hkex.com.hk/News/HKEX-Calendar?sc_lang=en (Ching Ming festival). This is correct in tc.

Answer 5 · 2020-09-09T20:46:32.000Z

For CFE:

Was closed on 2004-06-11 for National Day of Mourning (https://www.isda.org/2004/06/11/u-s-exchange-and-market-closings-on-june-11-2004-for-mourning-president-ronald-reagan/). tc is correct here.
Was closed on 2007-01-02 for National Day of Mourning (https://www.isda.org/2007/01/02/u-s-exchange-closings-on-january-2-2007-for-mourning-president-gerald-ford/#:~:text=(ISDA)%20announces%20the%20following%20guidance,former%20U.S.%20President%20Gerald%20Ford.). tc is correct here.
Was closed on 2018-12-05 for National Day of Mourning (https://www.prnewswire.com/news-releases/cboe-global-markets-exchanges-to-close-wednesday-for-day-of-mourning-to-honor-president-george-hw-bush-300758638.html#:~:text=1%2C%202018%20%2FPRNewswire%2F%20%2D%2D,who%20passed%20away%20November%2030.). tc is correct here.
Was closed on 2012-10-29 and 2012-10-30 for Hurricane Sandy (http://ir.cboe.com/press-releases/2012/29-oct-2012a). tc is correct here.

I can't seem to find any references for Nixon's Day of Mourning on 1994-04-27... Not sure what to do about this one.

Answer 6 · 2020-09-15T19:59:59.000Z

Discussion of pre-1990 NYSE (XNYS) calendar here: #162

Answer 7 · 2020-09-15T23:54:42.000Z

Discussion of LSE differences here: #163

Answer 8 · 2020-09-16T00:22:25.000Z

Discussion of XKTS (JPX) differences here: #164