A small proof-of-concept project that retrieves stock market data from the AlphaVantage API, sends it to a Kafka producer/consumer setup, stores it in AWS S3 and enables AWS Athena to analyze it through SQL.
-
Create an EC2 instance on AWS and edit the security group options such that the rules include all traffic from your local machine's IP and the public IP of the EC2 instance. You can also allow all traffic from any IP, although that is considered bad practice.
-
Install Java
sudo yum install java-1.8.0-amazon-corretto
java -version
- Download Kafka
wget https://downloads.apache.org/kafka/3.3.1/kafka_2.12-3.4.1.tgz
tar -xvf kafka_2.12-3.4.1.tgz
- Start Apache Zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
- Change
server.properties
configuration
The EC2 instance is private by default, so server.properties
has to be edited so that it can run on a public IP address. To do this, edit the ADVERTISED_LISTENERS
line in the config/server.properties
file and plug in the public IP address of the EC2 instance.
- Start Apache Kafka broker/server
export KAFKA_HEAP_OPTS="-Xmx256M -Xms128M"
cd kafka_2.12-3.4.1
bin/kafka-server-start.sh config/server.properties
- Create a topic
cd kafka_2.12-3.4.1
bin/kafka-topics.sh --create --topic mytopic --bootstrap-server 1.2.3.4:9092 --replication-factor 1 --partitions 1
- Start the producer
bin/kafka-console-producer.sh --topic mytopic --bootstrap-server 1.2.3.4:9092
- Start the consumer
cd kafka_2.12-3.4.1
bin/kafka-console-consumer.sh --topic mytopic --bootstrap-server 1.2.3.4:9092
- AWS S3 → stores object data in buckets
- AWS Glue Crawler → scans and discovers the structure, schema, and format of data stored in various data sources like Amazon S3, Amazon RDS, and more. It creates metadata tables in the AWS Glue Data Catalog, facilitating data integration and making it easier to use and process the data in AWS Glue ETL jobs, AWS Athena queries, and other AWS services
- AWS Glue Data Catalog → acts as a centralized catalog for storing and organizing metadata about data sources, tables, and schemas
- AWS Athena → a serverless query service that enables users to run ad-hoc SQL queries on data stored in Amazon S3