Run the latest version of the ELK (Elasticsearch, Logstash, Kibana) stack with Docker and Docker Compose.
It will give you the ability to analyze any data set by using the searching/aggregation capabilities of Elasticsearch and the visualization power of Kibana.
Based on the official Docker images:
- Clone this repository and create an empty dir data
$ mkdir data
- Start the ELK stack using
docker-compose
:
$ docker-compose up
You can also choose to run it in background (detached mode):
$ docker-compose up -d
Give Kibana a few seconds to initialize, then access the Kibana web UI by hitting
[http://localhost:5601](http://localhost:5601) with a web browser.
By default, the stack exposes the following ports:
* 5000: Logstash TCP input.
* 9200: Elasticsearch HTTP
* 9300: Elasticsearch TCP transport
* 5601: Kibana
There are two default actions defined in the 'curator/action_file.yml'.
First action defined is deleting the old indices X number of days ago.
- To set that X number of days, change the 'unit_count' under 'filters'.
filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 5
The default index name follows this pattern "filebeat-{date}". "filebeat" being the prefix and date being in the format of "YYYY-MM-dd"
- To set either of these fields, modify the "value" of the filter type pattern and "timestring" respectively
- filtertype: pattern
kind: prefix
value: filebeat-
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
Second action defined is deleting the old indices after an Y GB of space is used
- To set that Y GB, change the 'disk_space' under 'filters'.
filtertype: space
disk_space: 0.001
The default index name follows this pattern "filebeat-{date}". "filebeat" being the prefix and date being in the format of "YYYY-MM-dd"
- To set either of these fields, modify the "value" of the filter type pattern and "timestring" respectively
- filtertype: pattern
kind: prefix
value: filebeat-
- filtertype: space
disk_space: 0.001
use_age: True
source: name
timestring: '%Y.%m.%d'
To set the cronjob frequency, go to "curator/Dockerfile" and then set the schedule value to the cronjob configuration you want
ENV SCHEDULE "*/5 * * * * *"
To install any plugin for any of the three components of the ELK Stack (Elastic Search, Kibana or Logstash), go to the intended directory and modify the Dockerfile.
For example, let us say we want to install a plugin on kibana, go to kibana dir and edit the Dockerfile as follows:
FROM docker.elastic.co/kibana/kibana-oss:6.6.0
# Add your kibana plugins setup here
RUN kibana-plugin install <name|url>