/Hadoop-Spark-Airflow

Primary LanguageJupyter Notebook

Project 3

Run pipeline

  1. Install dependency

    pip3 install -r requirement

  2. Run cluster

    docker-compose up -d

  3. Install airflow

    export AIRFLOW_HOME=/working/dir/airflow/dags
./install-airflow.sh

  4. Run pipeline

    airflow standalone

Crawl data

  1. change NAME_NODE_ID in /crawler/constants.py by name node container id.

  2. Run bash

    cd crawler
python3 crawler.py

Run spark application

  1. change containerId in /client/run.sh by spark container id.

  2. change fileName in /client/run.sh by application want run.

  3. Run bash

    /client/run.sh