/historical-data-download

downloading historical stock data using Interactive Brokers API

Primary LanguageJava

IB TWS API Historical Data Download

Codes to download US single stock historical data using IB TWS API.

Needs

How to Use

  • Run it in command line java HistoricalDataDownloader and input request parameters
  • Data will be saved in provided directory, named ticker barSize yyyymmdd-yyyymmdd
  • If used from another class, call static HistoricalDataDownloader::getDownloader and HistoricalDataDownloader::start

Input parameters

  • Stock tickers, as String for one ticker or List<String> for multiple tickers
  • End date of data window (year, month, day)
  • Data period (how long to retrieve)
  • Data interval/granularity (from 1 second to 1 week)
  • Path to file output directory

Outputs

  • CSV data, in chronological order (old data first)
  • Intraday data: timestamp (yyyyMMdd HH::mm:ss), bid, ask, open, high, low, close, volume
  • Interday data: timestamp (yyyyMMdd), open, high, low, close, volume
  • OHLC are of traded prices
  • null for data unavailable at a timestamp

Comments

  • All times are defaulted to EST America/New York, 9:30 to 15:59, regular trading hours
  • Bid and ask prices are open prices at timestamps
  • For intraday data, the close of the last data point (like 15:59:00 for 1-min) is different from the daily close which results from closing auction
  • Trading volumes provided by IBKR are lower than other sources (Yahoo Finance etc), often by a substantial margin. Only RTH data are used here. IBKR data feed filters trades that tend to occur away from NBBO such as block trades and excludes odd lot trades
  • When a non-trading date entered, it will be shifted to the last trading day and request period remains unchanged

API Limitations and Workarounds

  • Because request data types are seperated into bid, ask, and traded prices and only one can be sent per request, requests have to be repeatedly sent
  • IB emphasizes it is not a data provider and limits return data points to a few hundreds per request (soft limit), so data windows are directly tied to granularity/interval requested; for 1-min data, 390 (6.5hrs x 60mins) data points per day, so 2-3 days window per request appropriate
  • Maximum 10 simultaneous requests

Logic Overview

  • One class (singleton), whose instance used to connect to TWS and perform all requests using different request identifiers (reqId)
  • Constructors overloaded to take ticker as either String or List<String>
  • start() encapsulates major operations including TWS connect, ticker looping, contract setting, request, msg reading, saving, and disconnect
  • EReader instance, tied to the socket, listens to incoming messages and pushes all messages into the queue
  • Built-in EReader.processMsgs() then called to pass received data and tagged reqId from the queue to relevant callback HistoricalData()
  • HistoricalData is called repeatedly for every message (data point) in a request, related callback HistoricalDataEnd() is called when all messages of a request are sent
  • Data are accumulated into a collection (ArrayList or LinkedList) inside HistoricalData callback
  • Flag isDone used to track if all messages are received as signaled by HistoricalDataEnd, until then keep looping
  • isIntraday flag for intraday or interday data request
  • Because IBKR bid, ask, and trades data require one request each, intraday data need to send 3 separate requests, of different (arbritarily assigned) ids, and results pushed into 3 collections to be combined into one at the end
  • Interday data come only from TRADES request, so uses only one container
  • Custom data types Bid, Ask, Trades defined, with compareTo and toString overriden
  • IB data feed is chronological, so synchronous saving of custom data is in natural order already; but Comparable<Trades> makes possible to sort Trades type based on datetime

Future Works

  • Expand to non-equity contracts, especially FX and futures