Click here to see general information about application!
With this app, you can see what programming skills are most in demand in the current job market.
Click here to see the technologies used!
- Airflow 2.0
- AWS EMR 5.25
- AWS Lambda
- Boto3 1.20.0
- Docker 20.10.7
- Docker-compose 1.29.2
- Django 3.2.5
- Hadoop 2.8.5
- Pandas 1.3.4
- Python 3.8.5
- Postgres 12.5
- Scrapy 1.7.3
- Serverless 2.64.1
- Terraform 1.0.3
- Write your AWS credentials into /skills_finder_web/.env and also /.aws/credentials files.
- Run this below commands (to execute this commands you must use Linux system!)
chmod +x create_aws_env.sh
./create_aws_env.sh- After that you can start ETL data pipeline by using Airflow (available on http://127.0.0.1:8080)
docker-compose -f .\docker-compose-airflow.yml up --build
- Finally, when all dags complete their activities it is time to launch the web app. Just navigate to /skills_finder_web directory and up another container.
docker-compose -f .\docker-compose-web.yml up --build
Application will be available on http://127.0.0.1:8000
