Create EKS Clusters and Deploy Kubeflow from OSX

Single command to create an EKS cluster on AWS and deploy Kubeflow to that cluster from OSX using Docker

Requirements

Overview

Deploying the most recent versions of Kubeflow to AWS is not trivial when using OSX.

Per Kubeflow's documentation, deploys must be done in a Dockerized Ubuntu instance: https://awslabs.github.io/kubeflow-manifests/release-v1.6.1-aws-b1.0.0/docs/deployment/prerequisites/

This script makes this operation a one-liner (minus some configuration)

Quick Start

This creates a single-node EKS cluster named "kubeflow" on us-west-2 and deploys Kubeflow on the cluster:

export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export CLUSTER_NAME=kubeflow

./deploy-kubeflow.sh

Once the deployment is complete, the following command will run to bind the cluster locally from the container:

make port-forward IP_ADDRESS=0.0.0.0

and you can access Kubeflow at http://localhost:8080/

Additional Configuration

The following additional arguments can be specified via environmental variables:

# The AWS Region to create the cluster on
export CLUSTER_REGION="us-west-2"

# The Node group name for the cluster on
export NODEGROUP_NAME="linux-nodes"

# The EC2 instance type for the cluster
export NODE_TYPE="m5.xlarge"

# The starting number of nodes for the cluster
export NODES=1

# The minimum number of nodes for the cluster
export NODES_MIN=1

# The maximum number of nodes for the cluster
export NODES_MAX=1

# The Kubernetes version for the cluster
export CLUSTER_K8S_VERSION="1.23"

# The Kubeflow version to deploy to the cluster
export KUBEFLOW_RELEASE_VERSION="v1.6.1"

# The AWS Kubeflow version to deploy to the cluster
export AWS_RELEASE_VERSION="v1.6.1-aws-b1.0.0"

IAM Configuration

The AWS ACCESS and SECRET keys must belong to an IAM user with sufficient permissions to create an EKS cluster and deploy Kubeflow. This is the configuration that worked for me:

AmazonEC2FullAccess
AmazonEKSClusterPolicy
AmazonEKSWorkerNodePolicy
AmazonS3FullAccess
AmazonVPCFullAccess
AWSCloudFormationFullAccess
IAMFullAccess
IAMUserChangePassword
EKS-Automation

EKS-Automation is a custom policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "eks:CreateCluster",
                "eks:DescribeCluster",
                "eks:UpdateClusterConfig",
                "eks:DeleteCluster",
                "eks:ListClusters",
                "eks:DescribeUpdate",
                "eks:CreateFargateProfile",
                "eks:DeleteFargateProfile",
                "eks:DescribeFargateProfile",
                "eks:ListFargateProfiles",
                "eks:CreateAddon",
                "eks:DescribeAddon",
                "eks:DeleteAddon",
                "eks:DescribeAddonVersions",
                "eks:CreateNodegroup",
                "eks:UpdateNodegroupConfig",
                "eks:DeleteNodegroup",
                "eks:DescribeNodegroup",
                "eks:ListNodegroups",
                "eks:UpdateNodegroupVersion",
                "eks:TagResource",
                "eks:UntagResource",
                "eks:ListTagsForResource"
            ],
            "Resource": "*"
        }
    ]
}