- Dev Infras:
- OS : Ubuntu 22.04
- RAM >= 20GB
- Tech stack:
- Source: coinmarketcap API
- Semantic: Kafka
- Orchestration: Airflow
- Backend: Python
- Datalake tier: Postgres
- Frontend: Superset (TBD)
docker compose up -d
git clone https://github.com/apache/superset.git
cd superset
git checkout 3.0.0
TAG=3.0.0 docker compose -f docker-compose-non-dev.yml up -d
docker compose logs -f
- airflow: localhost:8080 (airflow/airflow)
- kafka-ui: localhost: 8000
- superset: localhost:8088 (admin/admin)
Steps: airflow-ui -> admin -> connection -> add a record
Steps: Dag dashboard -> Search Dag -> Trigger The log of consume kafka might be like this
In this context, i use dbvisualizer (cuz its free)
Notice:
- Kafka Consumer inside airflow use: 9092 (communication between 2 images inside a container)