This is the repo of the Weather app from my YouTube video
In this project you build a end-to-end pipeline from requesting weather data from an API to visualizing the results on a Dashboard.
As tools we use the weather api, Docker, AWS Elastic container registry, AWS Lambda, AWS EventBridge and Grafana as the dashboard solution.
Here's an overview of the pipeline:
- Create account on weatherapi.com and generate a token
- Clone repository to local and replace the token
- Setup the TDengine cloud database on tdengine.com
- Create the weather database, stable and tables berlin and sanfrancisco
- Install TDengine connector for python with
pip install taospy - Run the code in your VSCode or other dev env and check TDengine if the data is there
- Build the docker container
docker build -f dockerfile-user -t weather-data . - If not done already install the AWS cli (see helpful links below)
- Create a development user and role in IAM with full rights to ecr, create keys for that user
- Do a
aws configureand enter key and secret key - Create ECR, tag the image and push the image up to ECR (find the commands in ECR, top right corner)
- Create Lambda that uses the image
- Create EventBridge schedule that triggers the Lambda function
- Pull the Grafana image from docker hub
docker pull grafana/grafana - Start grafana with
docker run --name=grafana -p 3000:3000 grafana/grafana - Go to localhost:3000 to access Grafana, connect the TDengine datasource and create yourself a Dashboard
- Try out the WeatherAPI interactive explorer: explore the API
- TDEngine documentation of read/write through websocket!! (for TDengine cloud) tdengine docs
- AWS cli installation AWS Documentation
- Ranking time series databases dbengines.com
- TDengine docker image visit Dockerhub
- TDengine stream processing, caching and data subscription Streaming features
- time series extentions like time weighted average rate of change and more functions
- Performance comparison influxdb timescaledb and TDengine Benchmark comparions