elasticsearch-on-spot
Use SAM to deploy an ElasticSearch cluster on AWS EC2.
Deployment Steps
cd sam-app/
sam validate
sam build
sam deploy --guidedAMI and Container Build Steps
cd elasticsearch-on-spot
rm ami.zip
zip -r ami.zip ./ami
aws s3 cp ami.zip s3://<bucketname>/ami.zip
rm fargateelasticsearch.zip
zip -r fargateelasticsearch.zip ./fargateelasticsearch
aws s3 cp fargateelasticsearch.zip s3://<bucketname>/fargateelasticsearch.zip
Example for a bucket with name: elasticsearch-on-spot-buildbucket-6kmvqaej409u
cd elasticsearch-on-spot
rm ami.zip
zip -r ami.zip ./ami
rm fargateelasticsearch.zip
zip -r fargateelasticsearch.zip ./fargateelasticsearch
aws s3 cp ami.zip s3://elasticsearch-on-spot-buildbucket-6kmvqaej409u/ami.zip
aws s3 cp fargateelasticsearch.zip s3://elasticsearch-on-spot-buildbucket-6kmvqaej409u/fargateelasticsearch.zip
How It Works
Motivation For The ElasticSearch Cluster Design:
We want to build a highly available ElasticSearch system that has 3 master-eligible nodes and can have a variable number of data nodes.
Typically, the way to perform discovery for an ElasticSearch cluster on AWS is via using the discovery-ec2 plugin. This plugin internally uses the equivalent of aws ec2 describe-instances command to filter nodes based on the region, availability zones, by security group(s), tag values. This is great but it requires use of AWS APIs - which means the ElasticSearch node should either have an IAM role or have credentials present on it.
In this case, we want to have no more assumptions beyond that the initial master node's private IP is referenced by a DNS record. Every instance there after uses the DNS record to discover the ElasticSearch cluster.
We use security group to inform the node's role: master or data. We do this as the security group the instance belongs to is available on the node itself. If we had used tags, it would've required an AWS API call.
The ami/ dir contains the instructions to build the ElasticSearch AMI and launch it. This is used by the SAM template during code build and after starting up the auto-scaling group.
setup_elasticsearch.sh -> provisions the AMI with ElasticSearch, Kibana and Metricbeat.
init_node.sh -> This is the boot time script that sets up the ElasticSearch node based on DNS and sets the node role based on the security group name.