doi-t/ex-aws-servicediscovery

ex-aws-servicediscovery

Play with AWS Cloud Map and ECS Service Discovery features.

TODOs

• Deploy Prometheus, node-exporter and alertmanager just for fun
• Use ECS Service Discovery as a Prometheus's DNS-based service discovery
  • It supports SRV record query in dns_sd_config
• Use EBS with Docker Volumes as Prometheus's tsdb storage
  • Ref. Data Volumes in Tasks
• Alertmanager
• Grafana
• cAdvisor
• Use Spot Fleet

Preparation

Create a S3 bucket for tfstate.

Deploy cluster

make TF_S3_BUCKET=<your_s3_bucket_name> KEY_NAME=<your_key_pair_name> YOUR_PUBLIC_DOMAIN=<your_public_domain_name> apply

Upload container images to ECR

make push

push example code

Run an example in https://github.com/prometheus-up-and-running/examples (Python only).

export EXAMPLE_CODE=up-and-running-examples/4/4-1-wsgi.py; make push-example

Cleanup

make TF_S3_BUCKET=<your_s3_bucket_name> KEY_NAME=<your_key_pair_name> YOUR_PUBLIC_DOMAIN=<your_public_domain_name> destroy

References

• YouTube: New Features for Building Powerful Containerized Microservices on AWS - AWS Online Tech Talks

Health Checks

Health checks: Perform periodic container-level health checks. If an endpoint does not pass the health check, it is removed from DNS routing and marked as unhealthy. For more information, see How Amazon Route 53 Checks the Health of Your Resources.

Ref. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-discovery.html#service-discovery-concepts

DNS Based Load Balancer in Public

Currently you need to register/deregister an instance to/from a public namespace with a public IP address by yourself.

• The DNS records created for a service discovery service will always register with the private IP address for the task, rather than the public IP address, even when public namespaces are used.

Ref. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-discovery.html#service-discovery-considerations

Understanding Prometheus's TSDB

$ ssh -i <your key pair's pem file path> ec2-user@<instance public ip which Prometheus container is running on>
[ec2] $ docker volume inspect prometheus-storage
[ec2] $ docker exec -it $(docker ps | grep prometheus-server | awk '{ print $1 }') /bin/sh