/docker-elk

那么，ELK 到底是什么呢？ “ELK”是三个开源项目的首字母缩写，这三个项目分别是：Elasticsearch、Logstash 和 Kibana。Elasticsearch 是一个搜索和分析引擎。Logstash 是服务器端数据处理管道，能够同时从多个来源采集数据，转换数据，然后将数据发送到诸如 Elasticsearch 等“存储库”中。Kibana 则可以让用户在 Elasticsearch 中使用图形和图表对数据进行可视化。 Elastic Stack 是 ELK Stack 的更新换代产品。

Primary LanguageShellMIT LicenseMIT

本项目的目的

本项目是https://github.com/osins/docker-elk的中文翻译，便于尽快了解elk和基于docker部署elk，因此关于elk的最新动态请直接访问https://github.com/osins/docker-elk或者https://www.elastic.co/cn/what-is/elk-stack

ELK 到底是什么呢？

“ELK”是三个开源项目的首字母缩写，这三个项目分别是：Elasticsearch、Logstash 和 Kibana。Elasticsearch 是一个搜索和分析引擎。Logstash 是服务器端数据处理管道，能够同时从多个来源采集数据，转换数据，然后将数据发送到诸如 Elasticsearch 等“存储库”中。Kibana 则可以让用户在 Elasticsearch 中使用图形和图表对数据进行可视化。

Elastic Stack 是 ELK Stack 的更新换代产品。

Elastic stack (ELK) 基于 Docker 的资源

Elastic Stack version Build Status Join the chat at https://gitter.im/deviantony/docker-elk

启动最新 Elastic stack 的版本， 基于 Dokcer 以及 Docker-compose.

它使您能够使用 Elasticsearch 的搜索/聚合功能和 Kibana 的可视化功能来分析任何数据集。

注意：本模板包括 X-Pack 与 paid features 功能， 默认是开启的（参见如何禁用支付功能禁用它们）。该试用许可证的有效期为30天。此许可证到期后，您可以继续无缝使用免费功能，而不会丢失任何数据。

本elk部署模板基于以下三个镜像:

其他可选择的功能及设置:

目的

我们的目标是为任何想尝试这种强大技术组合的人提供最简单的 Elastic 服务入口。这个项目的默认配置是最小化和无个性的。它不依赖于任何外部的依赖或自定义自动化来启动和运行。

相反，我们相信良好的文档，以便您可以将此存储库用作模板，对其进行调整，并使其成为您自己的. sherifabdlnaby/elastdocker是建立在这个想法之上的项目中的一个例子。

因此 https://github.com/osins/docker-elk 也是基于这个目的而尝试建立的Git仓库，也许本仓库会有个性化的修改，所以最终建议大家还是参考 https://github.com/deviantony/docker-elk#host-setup ，并以它为基准来搭建基于Docker的elk服务。

内容目录

  1. Requirements
  2. Usage
  3. Configuration
  4. Extensibility
  5. JVM tuning
  6. Going further

环境要求

主机设置

⚠️Elasticsearch 的引导程序检查被有意禁用，以方便在开发环境中设置 Elastic 服务。对于生产设置，我们建议用户根据 Elasticsearch 文档中的说明设置他们的主机：系统配置参考。

By default, the stack exposes the following ports:

  • 5044: Logstash Beats input
  • 5000: Logstash TCP input
  • 9600: Logstash monitoring API
  • 9200: Elasticsearch HTTP
  • 9300: Elasticsearch TCP transport
  • 5601: Kibana

⚠️ Elasticsearch's bootstrap checks were purposely disabled to facilitate the setup of the Elastic stack in development environments. For production setups, we recommend users to set up their host according to the instructions from the Elasticsearch documentation: Important System Configuration.

SELinux

On distributions which have SELinux enabled out-of-the-box you will need to either re-context the files or set SELinux into Permissive mode in order for docker-elk to start properly. For example on Redhat and CentOS, the following will apply the proper context:

$ chcon -R system_u:object_r:admin_home_t:s0 docker-elk/

Docker for Desktop

Windows

Ensure the Shared Drives feature is enabled for the C: drive.

macOS

The default Docker for Mac configuration allows mounting files from /Users/, /Volumes/, /private/, and /tmp exclusively. Make sure the repository is cloned in one of those locations or follow the instructions from the documentation to add more locations.

Usage

Version selection

This repository tries to stay aligned with the latest version of the Elastic stack. The main branch tracks the current major version (7.x).

To use a different version of the core Elastic components, simply change the version number inside the .env file. If you are upgrading an existing stack, please carefully read the note in the next section.

⚠️ Always pay attention to the official upgrade instructions for each individual component before performing a stack upgrade.

Older major versions are also supported on separate branches:

Bringing up the stack

Clone this repository onto the Docker host that will run the stack, then start services locally using Docker Compose:

$ docker-compose up

You can also run all services in the background (detached mode) by adding the -d flag to the above command.

⚠️ You must rebuild the stack images with docker-compose build whenever you switch branch or update the version of an already existing stack.

If you are starting the stack for the very first time, please read the section below attentively.

Cleanup

Elasticsearch data is persisted inside a volume by default.

In order to entirely shutdown the stack and remove all persisted data, use the following Docker Compose command:

$ docker-compose down -v

Initial setup

Setting up user authentication

ℹ️ Refer to How to disable paid features to disable authentication.

The stack is pre-configured with the following privileged bootstrap user:

  • user: elastic
  • password: changeme

Although all stack components work out-of-the-box with this user, we strongly recommend using the unprivileged built-in users instead for increased security.

  1. Initialize passwords for built-in users

    $ docker-compose exec -T elasticsearch bin/elasticsearch-setup-passwords auto --batch

    Passwords for all 6 built-in users will be randomly generated. Take note of them.

  2. Unset the bootstrap password (optional)

    Remove the ELASTIC_PASSWORD environment variable from the elasticsearch service inside the Compose file (docker-compose.yml). It is only used to initialize the keystore during the initial startup of Elasticsearch.

  3. Replace usernames and passwords in configuration files

    Use the kibana_system user (kibana for releases <7.8.0) inside the Kibana configuration file (kibana/config/kibana.yml) and the logstash_system user inside the Logstash configuration file (logstash/config/logstash.yml) in place of the existing elastic user.

    Replace the password for the elastic user inside the Logstash pipeline file (logstash/pipeline/logstash.conf).

    ℹ️ Do not use the logstash_system user inside the Logstash pipeline file, it does not have sufficient permissions to create indices. Follow the instructions at Configuring Security in Logstash to create a user with suitable roles.

    See also the Configuration section below.

  4. Restart Kibana and Logstash to apply changes

    $ docker-compose restart kibana logstash

    ℹ️ Learn more about the security of the Elastic stack at Tutorial: Getting started with security.

Injecting data

Give Kibana about a minute to initialize, then access the Kibana web UI by opening http://localhost:5601 in a web browser and use the following credentials to log in:

  • user: elastic
  • password: <your generated elastic password>

Now that the stack is running, you can go ahead and inject some log entries. The shipped Logstash configuration allows you to send content via TCP:

# Using BSD netcat (Debian, Ubuntu, MacOS system, ...)
$ cat /path/to/logfile.log | nc -q0 localhost 5000
# Using GNU netcat (CentOS, Fedora, MacOS Homebrew, ...)
$ cat /path/to/logfile.log | nc -c localhost 5000

You can also load the sample data provided by your Kibana installation.

Default Kibana index pattern creation

When Kibana launches for the first time, it is not configured with any index pattern.

Via the Kibana web UI

ℹ️ You need to inject data into Logstash before being able to configure a Logstash index pattern via the Kibana web UI.

Navigate to the Discover view of Kibana from the left sidebar. You will be prompted to create an index pattern. Enter logstash-* to match Logstash indices then, on the next page, select @timestamp as the time filter field. Finally, click Create index pattern and return to the Discover view to inspect your log entries.

Refer to Connect Kibana with Elasticsearch and Creating an index pattern for detailed instructions about the index pattern configuration.

On the command line

Create an index pattern via the Kibana API:

$ curl -XPOST -D- 'http://localhost:5601/api/saved_objects/index-pattern' \
    -H 'Content-Type: application/json' \
    -H 'kbn-version: 7.13.2' \
    -u elastic:<your generated elastic password> \
    -d '{"attributes":{"title":"logstash-*","timeFieldName":"@timestamp"}}'

The created pattern will automatically be marked as the default index pattern as soon as the Kibana UI is opened for the first time.

Configuration

ℹ️ Configuration is not dynamically reloaded, you will need to restart individual components after any configuration change.

How to configure Elasticsearch

The Elasticsearch configuration is stored in elasticsearch/config/elasticsearch.yml.

You can also specify the options you want to override by setting environment variables inside the Compose file:

elasticsearch:

  environment:
    network.host: _non_loopback_
    cluster.name: my-cluster

Please refer to the following documentation page for more details about how to configure Elasticsearch inside Docker containers: Install Elasticsearch with Docker.

How to configure Kibana

The Kibana default configuration is stored in kibana/config/kibana.yml.

It is also possible to map the entire config directory instead of a single file.

Please refer to the following documentation page for more details about how to configure Kibana inside Docker containers: Install Kibana with Docker.

How to configure Logstash

The Logstash configuration is stored in logstash/config/logstash.yml.

It is also possible to map the entire config directory instead of a single file, however you must be aware that Logstash will be expecting a log4j2.properties file for its own logging.

Please refer to the following documentation page for more details about how to configure Logstash inside Docker containers: Configuring Logstash for Docker.

How to disable paid features

Switch the value of Elasticsearch's xpack.license.self_generated.type option from trial to basic (see License settings).

How to scale out the Elasticsearch cluster

Follow the instructions from the Wiki: Scaling out Elasticsearch

How to reset a password programmatically

If for any reason your are unable to use Kibana to change the password of your users (including built-in users), you can use the Elasticsearch API instead and achieve the same result.

In the example below, we reset the password of the elastic user (notice "/user/elastic" in the URL):

$ curl -XPOST -D- 'http://localhost:9200/_security/user/elastic/_password' \
    -H 'Content-Type: application/json' \
    -u elastic:<your current elastic password> \
    -d '{"password" : "<your new password>"}'

Extensibility

How to add plugins

To add plugins to any ELK component you have to:

  1. Add a RUN statement to the corresponding Dockerfile (eg. RUN logstash-plugin install logstash-filter-json)
  2. Add the associated plugin code configuration to the service configuration (eg. Logstash input/output)
  3. Rebuild the images using the docker-compose build command

How to enable the provided extensions

A few extensions are available inside the extensions directory. These extensions provide features which are not part of the standard Elastic stack, but can be used to enrich it with extra integrations.

The documentation for these extensions is provided inside each individual subdirectory, on a per-extension basis. Some of them require manual changes to the default ELK configuration.

JVM tuning

How to specify the amount of memory used by a service

By default, both Elasticsearch and Logstash start with 1/4 of the total host memory allocated to the JVM Heap Size.

The startup scripts for Elasticsearch and Logstash can append extra JVM options from the value of an environment variable, allowing the user to adjust the amount of memory that can be used by each component:

Service Environment variable
Elasticsearch ES_JAVA_OPTS
Logstash LS_JAVA_OPTS

To accomodate environments where memory is scarce (Docker for Mac has only 2 GB available by default), the Heap Size allocation is capped by default to 256MB per service in the docker-compose.yml file. If you want to override the default JVM configuration, edit the matching environment variable(s) in the docker-compose.yml file.

For example, to increase the maximum JVM Heap Size for Logstash:

logstash:

  environment:
    LS_JAVA_OPTS: -Xmx1g -Xms1g

How to enable a remote JMX connection to a service

As for the Java Heap memory (see above), you can specify JVM options to enable JMX and map the JMX port on the Docker host.

Update the {ES,LS}_JAVA_OPTS environment variable with the following content (I've mapped the JMX service on the port 18080, you can change that). Do not forget to update the -Djava.rmi.server.hostname option with the IP address of your Docker host (replace DOCKER_HOST_IP):

logstash:

  environment:
    LS_JAVA_OPTS: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=18080 -Dcom.sun.management.jmxremote.rmi.port=18080 -Djava.rmi.server.hostname=DOCKER_HOST_IP -Dcom.sun.management.jmxremote.local.only=false

Going further

Plugins and integrations

See the following Wiki pages:

Swarm mode

Experimental support for Docker Swarm mode is provided in the form of a docker-stack.yml file, which can be deployed in an existing Swarm cluster using the following command:

$ docker stack deploy -c docker-stack.yml elk

If all components get deployed without any error, the following command will show 3 running services:

$ docker stack services elk

ℹ️ To scale Elasticsearch in Swarm mode, configure seed hosts with the DNS name tasks.elasticsearch instead of elasticsearch.