csgorham/kafka-connect-stockprice-source

This repository contains a Kafka Connect source connector for importing stock price data from the Alpha Vantage API into Apache Kafka.

kafka-connect-stockprice-source

kafka-connect-stockprice-source is a Kafka Connect source connector for importing data from Alpha Vantage into Apache Kafka.

Building the connector

mvn package

Sample event

Events are produced every minute (during trading hours).

{
    "open"      : 147.288,
    "high"      : 147.37,
    "low"       : 147.12,
    "close"     : 147.35,
    "volume"    : 660905,
    "timestamp" : 1629478800,
    "datetime"  : "2021-08-20 12:00:00"
}

For foreign exchange data (FX), data is produced for the currency pair specified.

{
    "open"      : 1.23,
    "high"      : 1.23,
    "low"       : 1.22,
    "close"     : 1.23,
    "timestamp" : 1655757900,
    "datetime"  : "2022-06-20 20:45:00"
}

Note that Alpha Vantage do not publish as much history for FX data as they do for stocks, so a delay of 168 hours and an event emit interval of 1 minute may produce no data; a 15 minute interval does produce data that far back, but for a 1 minute interval a delay of 24 hours or less would be better.

Configuration

Name	Description	Type	Default	Example
alpha.vantage.api.key	API key for Alpha Vantage	String		AB123CD4EF5G6HIJ
stock.symbol	Stock symbol to retrieve prices for	String		IBM
topic	Kafka topic to produce stock price data to	String		TOPIC.IBM
delay.hours	Number of hours to delay delivering events	Integer	168	24
forex.from.symbol	Source currency (if no stock symbol)	String		GBP
forex.to.symbol	Destination currency (if no stock symbol)	String		USD
event.emit.interval	Event publish interval	String	1min	15min

A sample configuration file called sample-connector.properties is included.

Delay

This was written for a quick prototype proof-of-concept based on processing live stock price events, but I wanted something that I could use with a free API key.

To enable this, the connector is downloading historical events using an Alpha Vantage API that returns several days of one-minute interval time-series records for a stock. It then sends individual price events to Kafka time-shifted by delay.hours.

For example, if you set delay.hours to 24, then it will download the last day of stock price records, and then start sending them to Kafka - each event produced 24 hours after the timestamp in the price record.

This means the Connector only needs to make one API call a day, which is easily within the limits for a free API key. But it produces an event every minute (during trading hours) which makes it useful for demos and proof-of-concepts that need (pseudo-)real-time stock price events.

By default, the delay is set to 168 hours (one week) which means events are published on weekdays and not at weekends, which helps make for a more believable data source.