`trading_calendars` is now a standalone module
ssanderson opened this issue · 14 comments
Hi @rsheftel,
I just wanted to post an FYI that we've just landed a change in Zipline to factor out zipline.utils.calendars into a standalone trading_calendars module. We're working on adding support for international markets to Quantopian, so I expect there to be a fair amount of activity over there in the coming months.
(Feel free to close this issue by the way, I just figured users of this module would likely be interested in this info as well.)
Thank you very much for the heads up. Is your goal to make the new trading_calendars a stand alone tool? Would it make sense to merge this project and yours? I could always just copy over the new calendars you create since this code is based on yours. What are your thoughts?
Our immediate reasons for factoring out the calendars were:
- We had a few internal projects that depended on Zipline only for the calendars. Zipline is a pretty heavyweight dependency since it uses scipy and a bunch of file formats for on-disk data. It's also currently stuck on a relatively old pandas for backwards-compatibility reasons. Moving out the calendars makes it easier for projects that depend on them to not have to take on the rest of zipline's baggage.
- The test suite for the calendars is relatively slow. Since we expect to add a bunch of new calendars in the coming months, we didn't want to add even more build time to Zipline's test suite. (This is mitigated somewhat by the fact that we've merged a few performance improvements to the calendars themselves during this refactoring, e.g., quantopian/trading_calendars#4).
- Having a standalone calendars module has been a longstanding feature request from other Zipline users.
We do intend for trading_calendars to be a standalone tool, in the sense that it should be installable and usable independently from Zipline. That said, I expect the project to evolve somewhat in lockstep with Zipline, at least for the next few months as we iron out our model for supporting global equity markets (for example, I expect there to be some interesting work to figure out how to support lunch hour on asian markets, and that work will likely require changes in both trading_calendars and zipline).
As far as merging the repositories goes, I think that's an interesting question. It'd be nice for there to be a single standard place for trading calendars in the financial pydata ecosystem; there are lots of markets with lots of calendars, and having lots of eyeballs on the calendar implementations increases the chances that we catch historical inaccuracies or learn about upcoming changes to various markets. My only concern with merging is that we haven't always done a great job at Quantopian of accepting community contributions to Zipline, because, given the nature of our domain, we try to take correctness and performance very seriously, which means it's often a lot of work (for both a contributor and a reviewer) to get a change to the point where we're comfortable accepting it. We also tend to have relatively stringent backwards compatibility constraints because of the amount of Q community code that depends on our libraries.
If both of those restrictions sound okay to you, then I think some form of merger probably makes sense. Are there particular changes that have been made here that you would expect to be important to forward-port to trading_calendars?
Thank you very much for the detailed response. Many of my reasons for creating this package were similar to yours, primarily to break it free from the larger Zipline project to lower the dependency requirements. It sounds like the major point of difference is that I am very willing to include any new contribution from people and let the marketplace decide if the adds are correct or have value (as I don't have the expertise to validate every market). At this point over half the calendars have come from people other than myself (or the original Zipline that this is forked from). It makes sense that you will have higher standards for accepting new material.
All that said I agree that the pydata world should have one source for this data for all the positive benefits you mentioned.
Here is a possible solution:
Givens:
- This package is just a fork with some modifications from your Zipline it should be certainly possible to keep in sync or merge.
- All the real value is in the calendar definitions more that the utilities around them
- The core "MarketCalendar()" class and the specific implementations for each market is where those definitions are. I have not changed the structure of the MarketCalendar() class in any material way from your original version.
Plan:
- I take a look at your new trading_calendars package and see if the changes you made can be brought into this package.
- Once that is done I can let you know the diffs of functionality here that is not in yours.
- Then you (we?) can decide if a merge makes sense given your operating model.
- If a merge does make sense, great we do that.
- If not then perhaps we could refactor out just the modules that define the calendars for each market (like "exchange_calendar_nyse.py") into a new package that we both inherit from. That way each can evolve the wrapper code as needed, but there is one golden source for the actual calendars. Additionally that new "calendars only package" could have a "beta" and "prod" version of the calendars so that the community could add/change calendars in the "beta" stream, my package could import from that, and only when you approve the changes do they get promoted to the prod stream which your package uses.
Let me know what you think.
Hey @rsheftel! Sorry for the slow reply. I was on vacation last week.
I think that plan generally sounds reasonable. I'm not sure I understand what would be involved in the "calendars only package" idea you proposed; trading_calendars is already that repo I think. The only other code of note in the repo is a system for registering and fetching calendars by name.
I meant a repo where there would be beta and prod versions of the MarketCalendar() classes. That way you don't need to pollute your repo with calendars that have not met your standard yet, but the community can still contribute to encourage more eyes on the calendars.
Hello @rsheftel , @ssanderson
First of all, thanks a lot for developing and maintaining these projects!
What's your current take on these two projects? Looks like over the last year they were developed independently. Are you still considering to merge?
Best
Vladimir
I am very happy to do a merge, I think that would benefit the community overall to have all eyes looking at the same package to find errors and make improvements. Open to any ideas on how to get this done.
Hey @rsheftel,
I wanted to resurrect this discussion because the trading_calendars project has grown a lot since this thread began. Specifically, I wanted to address your (completely valid) comments related to being hesitant to add calendars, since we may have a particular standard that could make PRs harder to accept. Generally speaking, we make a distinction between calendars we depend on for Quantopian (the list of 44 Q-supported markets is documented here), and all other calendars. If you're looking to add new calendars to the repo, we will still care about having tests, not breaking the build, etc. However, I expect we will be much less strict with requiring sources for new calendars, since we don't depend on them downstream. So if you want to add any new calendars to trading_calendars, we'd be happy to review the PR and fold it into the project!
@jmccorriston, I'm back from a long break. I am still interested in seeing how we can merge the effort here on these projects. I would consider a PR into your repo when I have the time to work on it. If you have free resources to apply I would also be happy to help direct them.
@rsheftel Sorry for taking so long to respond. I don't have a great setup with my GH notifications and completely missed your last message. If you're still interested (and if you have a better notification setup than I do!), I'd love to connect and discuss the idea of merging pandas_market_calendars and trading_calendars. What is your availability like these days?
No problem at all. I am usually available on Friday afternoons. You can email me at rsheftel@alumni.upenn.edu and we can coordinate
Sounds awesome. I've been using both pandas_market_calendars and trading_calendars so I'd love to see them merged.
The way I see it as an end-user, pandas_market_calendars has some useful features not available in trading_calendars yet (e.g. lunch breaks). On the other hand, trading_calendars feels more complete (i.e. more utility/convenience methods), and some methods are better implemented at trading_calendars (e.g. caching/memoization, default start/end dates).
@leonarduschen I appreciate the feedback, thank you! That all makes sense.
@rsheftel Great, I'll reach out via email!