/most-wanted-programming-skills-finder

With this app, you can see what programming skills are most in-demand in the current job market.

Primary LanguagePython

most-wanted-programming-skills-finder

Table of contents

General info

Click here to see general information about application!
With this app, you can see what programming skills are most in demand in the current job market.

Technologies

Click here to see the technologies used!
  • Airflow 2.0
  • AWS EMR 5.25
  • AWS Lambda
  • Boto3 1.20.0
  • Docker 20.10.7
  • Docker-compose 1.29.2
  • Django 3.2.5
  • Hadoop 2.8.5
  • Pandas 1.3.4
  • Python 3.8.5
  • Postgres 12.5
  • Scrapy 1.7.3
  • Serverless 2.64.1
  • Terraform 1.0.3

Setup


Prepare skills scraper

  1. Write your AWS credentials into /skills_finder_web/.env and also /.aws/credentials files.
  2. Run this below commands (to execute this commands you must use Linux system!)
chmod +x create_aws_env.sh
./create_aws_env.sh

Run ETL data pipeline

  1. After that you can start ETL data pipeline by using Airflow (available on http://127.0.0.1:8080)
docker-compose -f .\docker-compose-airflow.yml up --build

Run web server

  1. Finally, when all dags complete their activities it is time to launch the web app. Just navigate to /skills_finder_web directory and up another container.
docker-compose -f .\docker-compose-web.yml up --build

Application will be available on http://127.0.0.1:8000

Application view

ETL data pipeline

data_pipeline_top

Below charts are generated in "skill-finder-web" app

Most wanted backend skills

backend-chart

Most wanted big-data skills

big-data-chart