/terraform-aws-batch

High Throughput Batch Computing in AWS - Terraform module

Primary LanguageHCLMIT LicenseMIT

High-Throughput Batch Computing on AWS.

IaC setup for high-throutput-batch-workflow setup in AWS using Batch written in terrraform.

terraform version

Set up

Get terraform

wget https://releases.hashicorp.com/terraform/0.12.28/terraform_0.12.28_linux_amd64.zip
unzip terraform_0.12.28_linux_amd64.zip
mv terraform /usr/local/bin/

or

brew install terraform # OS X

Batch module usage

Configure AWS Batch with just ~15lines of configs.

module "batch" {
  source      = "iamlmn/batch/aws"
  version     = "1.1.0"
  Name        = "${var.Name}"
  Application = "${var.Application}"

  # Compute environment
  instance_type      = "${var.instance_type}"
  maxvcpus           = "${var.maxvcpus}"
  minvcpus           = "${var.minvcpus}"
  ce_security_groups = "${var.ce_security_groups}"
  ce_subnets         = "${var.ce_subnets}"

  # job definitions
  job_definition_name = "${var.job_definition_name}"
  docker_repo_name    = "${var.docker_repo_name}"
  jd_memory           = "${var.jd_memory}"
  jd_vcpus            = "${var.jd_vcpus}"
  job_command         = "${var.job_command}"

  # job queues
  job_queue_name     = "${var.job_queue_name}"
  job_queue_priority = "${var.job_queue_priority}"

  region = "us-east-1"
}

Individual sub-modules can also be used directly Refer to their respective readme for usages.

Requirements

Name Version
terraform ~> 0.12.6
aws ~> 2.49

Prerequisites

  • VPC default/ custom should be already present.
  • ECR & docker image should be already present
Name Description Type Default Required
Application The instance_type for compute environment to use string n/a yes
ce_security_groups The ce_security_groups list(string) true yes
ce_subnets The ce_subnets list(string) true yes
docker_repo_name The name of the job definitions string true yes
Name Name of the service/pipeline string true yes
instance_type The ce_subnets list(string) false no
jd_memory Batch job definition memory string true no
jd_vcpus VCPUs required by the job definition string true no
job_command Job definition command list true no
job_queue_name The name of the job queue string true no
job_queue_priority Job definition command string true no
maxvcpus Max allowed Vcpus to be spun up by compute environment string true no
minvcpus Min allowed Vcpus to be spun up by compute environment string true no
region AWS availability zone/region string true no

Outputs

Name Description
job_queue_arn The ARN of the Batch Job queue created
job_definition_arn The ARN of the Batch Job definition created
job_definition_name The Name of the Batch Job definition created
job_queue_arn The ARN of the Batch Job queue

Planned work/ TODOs :

  • Base Batch module.
  • Basic Batch submodule.
  • Register as Terraform module.
  • Alpha Test and fix Readme usage definitions.
  • Generalize more with all possible configs.
  • Better variable description and documentation.

Author

License