One-stop shop for deploying a Pangeo JupyterHub on AWS. This repository uses snakemake
to configure and deploy computing resources on AWS. Following the steps in the README you should have a hub up and running in ~ 30 minutes. Inspired by https://github.com/dask/dask-tutorial-infrastructure. Note that are many alternative approaches to deployment. I was inspired to learn snakemake...
Warning: running this code will incure AWS costs and no gaurantees are made for network security. Use at your own risk
In more detail running the following code will
- create an EKS Kubernetes Cluster
- create an EFS drive for network file system persistent home storage
- create a multiuser Pangeo JupyterHub
- use Pangeo Stacks images by default
- (optionally) tear everything down
** NOTE: tested with conda=4.7.12 eksctl=0.6.0, kubectl=1.16.0, helm=2.14.2 **
git clone https://github.com/scottyhq/deploy-pangeo-jhub
cd deploy-pangeo-jhub
conda env create -f environment.yml
conda activate pangeo-deploy
** If you don't already have conda installed, you'll have to download and run wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh; bash Miniconda3-latest-Linux-x86_64.sh
aws:
accountNumber: XXXXXXX
accountProfile: XXXXXXX
user: pangeo-admin
eksctl:
kubernetesVersion: 1.13
autoscalerVersion: 1.13.7
clusterName: XXXXXXX
region: XXXXXXX
zones: XXXXXXX
sshPublicKey: XXXXXXX.pub
s3Arn: arn:aws:iam::aws:policy/AmazonS3FullAccess
jupyterhub:
namespace: pangeo
secretToken: XXXXXXXXX
pangeo:
version: 19.09.26-dd6574b
Where accountNumber
is your account number (should be known to you) and
accountProfile
is the section in your ~/.aws/credentials file that you would
like to use here for authentication. clusterName
is a string of your choosing
and region
is the AWS region where you would like to deploy your cluster.
zones
should be the AWS availabilty zones to use.
For example ["us-east-1a" , "us-east-1b"]
. It looks like you should
avoid "us-east-1e"
.
See below for generating the sshPublicKey
secretToken
can be generated calling openssl rand -hex 32
, or reusing another
token that was generated thusly.
snakemake iam
Note: this isn't necessary if you identified an existing admin user in config.yml.
If you do run this step you will have to log onto the AWS console, generate
access keys and enter them into the terminal along with your cluster region
and set the default output format to json
.
snakemake pangeo
Note: it typically takes 10+ minutes to create a new EKS cluster. Be patient.
snakemake delete_cluster
Note: this can take several minutes. Double check everything deleted successfully by looking at the cluster stack in the AWS CloudFormation console.
? How to create a new ssh private and public key pair?
KEY_NAME=my-cluster-us-east-1
REGION=us-east-1
aws ec2 create-key-pair --key-name ${KEY_NAME} --region ${REGION} | jq -r ".KeyMaterial" > ${KEY_NAME}.pem
chmod 400 ${KEY_NAME}.pem
ssh-keygen -y -f ${KEY_NAME}.pem > ${KEY_NAME}.pub
? I have multiple kubernetes contexts, how to do I switch between them
alias kube-pangeo='export AWS_DEFAULT_PROFILE=pangeo-admin && aws eks --region us-west-2 update-kubeconfig --name pangeo'