To deploy your branch to all nodes
-
Git commit, push your branch to github.
-
From repository root directory:
sh utils/deploy_branch.sh [netid] [branch name]
To launch or manage VMs: https://vc.cs.illinois.edu/ui/
Start HDFS cluster (10 nodes) without YARN
start-dfs.sh
With YARN (job scheduling/MapReduce)
start-yarn.sh
Stopping cluster
stop-all.sh
Monitor cluster health, ssh using
ssh -L 9870:localhost:9870 [netid]@sp21-cs525-g14-01.cs.illinois.edu
Go to http://localhost:9870 in the browser.
Start Spark standalone cluster on all workers
/opt/spark/sbin/start-all.sh
Start only master
/opt/spark/sbin/start-master.sh
Start worker on worker node
/opt/spark/sbin/start-worker.sh <master-spark-URL>
Monitor cluster health, ssh using
ssh -L 8080:localhost:8080 [netid]@sp21-cs525-g14-01.cs.illinois.edu
Go to http://localhost:8080 in the browser.
conda activate [netid]_pytorch