There's currently a pull request open for EMR on EKS support in Terraform.
Until that gets merged, you can build your own version of the AWS Provider to test out the new code.
💁 Please note that this code is a proof-of-concept and not intended for production use
- Terraform v0.14+ (to use the custom provider and deploy your cluster)
- Go 1.16 (to build the provider plugin)
- Do NOT use Go 1.17 as it will error on
gofmt
- Also if you can not use the offical Go module proxy, the project will not build due to this issue
- Do NOT use Go 1.17 as it will error on
Here's the full documentation on building the AWS provider, but in short here's what you need to do.
Clone the repository and change into the directory
git clone git@github.com:terraform-providers/terraform-provider-aws
cd terraform-provider-aws
Checkout the pull request.
git fetch origin pull/20003/head:emr-eks-terraform
git checkout emr-eks-terraform
Run make tools
. This will install the needed tools for the provider.
make tools
To compile the provider, run make build
. This will build the provider and put the provider binary in the $GOPATH/bin
(default: $HOME/go
) directory.
make build
You should now have a terraform-provider-aws
binary in your $GOPATH
which terraform-provider-aws
You'll need this path for the next step.
In order to make use of the change, you need to populate your Terraform CLI configuration file (~/.terraformrc
) with the directory your new binary is in.
provider_installation {
dev_overrides {
"hashicorp/aws" = "/Users/username/go/bin"
}
direct {}
}
One note before getting started. The version of the EMR on EKS module needs to be pinned to 17.9.0
due to a breaking change and the fact this PR is based off a specific version of the AWS provider.
The Terraform example provisions everything you need:
- A new VPC (with the AWS VPC module)
- An EKS cluster with a managed node group (with the AWS EKS module)
- All the necessary IAM and Kubernetes roles as mentioned in the EMR on EKS Getting Started documenation
Please note that I've only run this end-to-end a couple times, so there may be some dependencies that require you to run terraform apply
again.
By default, Terraform will use your default AWS profile and region.
terraform init
terraform apply
The EKS cluster can take about 10-15 minutes to provision.
Now that your cluster is up, you should be able to run a job. Because the EKS cluster autoscaler is not installed by default, if you want to anything more complicated than the example below, you'll need to scale the cluster up manually.
The below example just runs a sample calculation of Pi and enables CloudWatch logging.
The EMR on EKS virtual cluster ID is provided in a emr-virtual-cluster-id
output and the role to run the job is in the emr-eks-job-role
output.
export EMR_EKS_CLUSTER_ID=<CLUSTER_ID>
export EMR_EKS_EXECUTION_ARN=arn:aws:iam::<ACCOUNT_ID>:role/tf_emr_eks_job_role
aws emr-containers start-job-run \
--virtual-cluster-id ${EMR_EKS_CLUSTER_ID} \
--name sample-pi \
--execution-role-arn ${EMR_EKS_EXECUTION_ARN} \
--release-label emr-6.3.0-latest \
--job-driver '{
"sparkSubmitJobDriver": {
"entryPoint": "local:///usr/lib/spark/examples/src/main/python/pi.py"
}
}' \
--configuration-overrides '{
"monitoringConfiguration": {
"cloudWatchMonitoringConfiguration": {
"logGroupName": "/aws/eks/emr-spark",
"logStreamNamePrefix": "pi"
}
}
}'
irsa example: https://github.com/terraform-aws-modules/terraform-aws-eks/tree/a26c9fd0c9c880d5b99c438ad620e91dda957e10/examples/irsa StringLike condition?: https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/examples/irsa/irsa.tf#L8 And another StringLike example: https://github.com/cloudposse/terraform-aws-eks-iam-role/blob/master/main.tf#L55 helpful re: roles: hashicorp/terraform-provider-kubernetes#322
update_config
was broken: terraform-aws-modules/terraform-aws-eks#1570