🐳 Elastic Stack on Docker, with preconfigured security, tools, self-monitoring, and Prometheus Metrics Exporters
Elastic Stack (AKA ELK) Docker Composition, preconfigured with Security, Monitoring, Tools such as ElastAlert for alerting and Curator.
- Configured as Production Single Node Cluster (With a multi-node option for experimenting).
- Use Docker-Compose and
.env
to configure your stack. - Security Enabled (under basic license).
- SSL Enabled for Transport Layer.
- Automated Script that initializes and persist Elasticsearch's Keystore and SSL Certifications.
- Curator Preconfigured for Automated Snapshotting (Need to setup S3 Repository).
- Self-Monitoring Metrics Enabled.
- Filebeat instance for shipping Stack logs to Elasticsearch itself.
- Prometheus Exporters for Stack Metrics.
- ElastAlert preconfigured for Alerting.
- Embedded Container Healthchecks for Stack Images.
- Go to repository directory
- Modify
.env
file for your requirments, most importantlyELASTIC_PASSWORD
that setup your superuserelastic
's password. andELK_VERSION
for, yk, ELK Version. - Initalize Elasticsearch Keystore and SSL Certificates
$ make setup
- Start Elastic Stack
$ make elk
---- OR ----
$ docker-compose up -d
- Visit Kibana at localhost:5601
Username: elastic
Password: changeme
(or ELASTIC_PASSWORD
value in .env
)
$ make monitoring
$ make tools
$ make all
$ make nodes
-
Adding Two Extra Nodes to the cluster will make the cluster depending on them and won't start without them again.
-
Makefile is a wrapper around
Docker-Compose
commands, usemake help
to know every command. -
Elasticsearch will save its data to a volume named
elasticsearch-data
-
Elasticsearch Keystore (that contains passwords and credentials) and SSL Certificate are generated in the
./secrets
directory by the setup command. -
Linux Users must set the following configuration as
root
sysctl -w vm.max_map_count=262144
By default, Virtual Memory is not enough.
- Some Configuration are parameterized in the
.env
file.ELASTIC_PASSWORD
, userelastic
's password (default:changeme
pls).ELK_VERSION
Elastic Stack Version (default:7.3.0
)ELASTICSEARCH_HEAP
, how much Elasticsearch allocate from memory (default: 1GB -good for development only-)LOGSTASH_HEAP
, how much Logstash allocate from memory.- Other configurations which their such as cluster name, and node name, etc.
- Elasticsearch Configuration in
elasticsearch.yml
at./elasticsearch/config
. - Logstash Configuration in
logstash.yml
at./elasticsearch/config/logstash.yml
. - Logstash Pipeline in
main.conf
at./elasticsearch/pipeline/main.conf
. - Kibana Configuration in
kibana.yml
at./kibana/config
. - ElastAlert Configuration in
./tools/elastalert/config
. - ElastAlert Alert rules in
./tools/elastalert/rules
, head to ElastAlert docs to lookup how to create alerts. - Curator Actions at
./tools/curator/actions
and./tools/curator/crontab
.
You can extend the Keystore generation script by adding keys to ./setup/keystore.sh
script. (e.g Add S3 Snapshot Repository Credentials)
To Re-generate Keystore:
make keystore
If you started Prometheus Exporters using make monitoring
command. Prometheus Exporters will expose metrics at the following ports.
Prometheus Exporter | Port | Note |
---|---|---|
elasticsearch-exporter |
9114 |
- |
logstash-exporter |
9304 |
- |
cadvisor-exporter |
8080 |
- To Monitor Each Container stats and metrics. |
Head to Stack Monitoring tab in Kibana to see cluster metrics for all stack components.
In Production, cluster metrics should be shipped to another dedicated monitoring cluster.
MIT License Copyright (c) 2019 Sherif Abdel-Naby
PR(s) are Open and Welcomed.