Docker Elasticsearch with Kibana and Filebeat

Run the latest version of the ELK (Elasticsearch, Filebeat, Kibana) stack with Docker and Docker Compose.

It will give you the ability to analyze any data set by using the searching/aggregation capabilities of Elasticsearch and the visualization power of Kibana.

Based on the official Docker images:

Note: Other branches in this project are available:

ELK 6 with X-Pack support: https://github.com/deviantony/docker-elk/tree/x-pack
ELK 6 in Vagrant: https://github.com/deviantony/docker-elk/tree/vagrant
ELK 6 with Search Guard: https://github.com/deviantony/docker-elk/tree/searchguard

Requirements

Host setup

Install Docker version 1.10.0+
Install Docker Compose version 1.6.0+
Clone this repository

SELinux

On distributions which have SELinux enabled out-of-the-box you will need to either re-context the files or set SELinux into Permissive mode in order for docker-elk to start properly. For example on Redhat and CentOS, the following will apply the proper context:

$ chcon -R system_u:object_r:admin_home_t:s0 docker-elk/

Usage

Bringing up the stack

Note: In case you switched branch or updated a base image - you may need to run docker-compose build first

Start the ELK stack using docker-compose:

$ docker-compose -f docker-compose-elk.yml up

You can also choose to run it in background (detached mode):

$ docker-compose -f docker-compose-elk.yml up -d

Give Kibana a few seconds to initialize, then access the Kibana web UI by hitting http://localhost:5601 with a web browser.

By default, the stack exposes the following ports:

9200: Elasticsearch HTTP
9300: Elasticsearch TCP transport
5601: Kibana

Notice: Kibana will ask you for basic auth username/password saved in filebeat.yml

WARNING: If you're using boot2docker, you must access it via the boot2docker IP address instead of localhost.

WARNING: If you're using Docker Toolbox, you must access it via the docker-machine IP address instead of localhost.

Initial setup

Default Kibana index pattern creation

When Kibana launches for the first time, it is not configured with any index pattern.

Via the Kibana web UI

NOTE: You need to inject data into Logstash before being able to configure a Logstash index pattern via the Kibana web UI. Then all you have to do is hit the Create button.

Refer to Connect Kibana with Elasticsearch for detailed instructions about the index pattern configuration.

Configuration

NOTE: Configuration is not dynamically reloaded, you will need to restart the stack after any change in the configuration of a component.

How can I tune the filebeat configuration?

The Filebeat configuration is stored in filebeat/filebeat.yml.

How can I tune the Elasticsearch configuration?

The Elasticsearch configuration is stored in elasticsearch/config/elasticsearch.yml.

How can I scale out the Elasticsearch cluster?

Storage

How can I persist Elasticsearch data?

The data stored in Elasticsearch will be persisted after container reboot but not after container removal.

In order to persist Elasticsearch data even after removing the Elasticsearch container, you'll have to mount a volume on your Docker host. Update the elasticsearch service declaration to:

elasticsearch:

  volumes:
    - /path/to/storage:/usr/share/elasticsearch/data

This will store Elasticsearch data inside /path/to/storage.

namvu80ap/docker-elk-filebeat