Run an ELK stack using Terraform for the infrastructure related components and ECS-CLI to deploy the ECS services The containers are currently deployed onto EC2 instances rather than Fargate.
ℹ️ The Docker images backing this stack include [Stack Features][stack-features] (formerly X-Pack) with [paid features][paid-features] enabled by default (see How to disable paid features to disable them). The [trial license][trial-license] is valid for 30 days.
Based on the official Docker images from Elastic:
By default, the stack exposes the following ports:
- 5000: Logstash TCP input
- 9200: Elasticsearch HTTP
- 9300: Elasticsearch TCP transport
- 5601: Kibana
All containers are direct from elastic with minor config changes.
To deploy the applications as services is ecs and enable service discovery each service is deployed using the awsvpc network type. This grants each task an IP within the VPC and requires network configuration which is output from Terraform. These must be copied to the relevant variables in docker/deploy.sh and then it must be run before deploying the services with ecs-cli.
This process should be made cleaner.
The Elasticsearch ec2 plugin does not work with ecs when using the awsvpc network mode.
Each application in the ELK stack has been configured to be accessible via an ALB by accessing the ALB using the public DNS name and the application port. This will forward traffic the the repective service.
Terraform is used to provision the infrastructure You will need access to an AWS account with API credentials. terraform/tfvars/terraform.tfvars is the base file for all variables required to provion the infrastucture. This needs to be configured and your public ssh key added to replace mine. Otherwise this will not run.
$ terraform init$ terraform plan -var-file=tfvars/terraform.tfvars -state=tfstate/terraform.tfstate$ terraform apply -var-file=tfvars/terraform.tfvars -state=tfstate/terraform.tfstateOnce terraform has run it will output a few important details that are required to deploy the services using ecs-cli, these must be manually copied to docker/deploy.sh at the moment. This is something that should be improved.
This is used to provision the services from docker-compose and ecs-params files. The ECS params files contain task specific definitions so each task/service has it's own file. To simplify the deployment process environment variables have been used to reduce the number of things that need to be configured.
The current method of deployment is to copy the details output from Terraform into deploy.sh and run the script. This will set environment variables for all the configureable options within the ecs-params.yml files. The rest of the process is quite manual and requires you to run commands.sh in each of the directories for the ELK stack to spin up an ECS service with one task.
Elasticsearch is different to this. It will spin up two services, one with a running task to initialise the cluster and another generic master service with 0 tasks running. This can be used to scale up the masters.
Elasticsearch uses bridge networking as the EC2 plugin does not currently work with ECS tagging to discover other members.
Apache can be ran locally or remotely on ECS with a filebeat sidecar container. Apache has been purposefully configured to log to file and filebeat has been configured to forward this to elasticsearch.
To run remotely on EC2 run the commands within the docker/apache directory and access the ALB on the public DNS name on port 80. This will generate logs that will appear in Kibana.
To run locally you must change the environment variables in docker-compose within docker/apache to the public address of the ALB.
Services may be slow to appear within the ALB due to the length of the health checks.