/snap-plugin-publisher-elasticsearch

Publishes Snap metrics to Elasticsearch

Primary LanguageGoApache License 2.0Apache-2.0

DISCONTINUATION OF PROJECT.

This project will no longer be maintained by Intel.

This project has been identified as having known security escapes.

Intel has ceased development and contributions including, but not limited to, maintenance, bug fixes, new releases, or updates, to this project.

Intel no longer accepts patches to this project.

DISCONTINUATION OF PROJECT

This project will no longer be maintained by Intel. Intel will not provide or guarantee development of or support for this project, including but not limited to, maintenance, bug fixes, new releases or updates. Patches to this project are no longer accepted by Intel. If you have an ongoing need to use this project, are interested in independently developing it, or would like to maintain patches for the community, please create your own fork of the project.

snap-plugin-publisher-elasticsearch

This plugin supports publishing metrics to Elasticsearch endpoint

It is used in the [Snap framework] (http://github.com/intelsdi-x/snap).

  1. Getting Started
  1. Documentation
  1. Community Support
  2. Contributing
  3. License
  4. Acknowledgements

Getting Started

System Requirements

  • golang 1.6+ (needed only for building)
  • Elasticsearch instance to publish data (docker run elasticsearch)

Installation

To build the plugin binary:

Fork https://github.com/intelsdi-x/snap-plugin-publisher-elasticsearch
Clone repo into $GOPATH/src/github.com/intelsdi-x/:

$ git clone https://github.com/<yourGithubID>/snap-plugin-publisher-elasticsearch.git

Build the plugin by running make within the cloned repo:

$ make

This builds the plugin in ./build/

Configuration and Usage

Documentation

The plugin expects you to provide following parameters:

  • protocol http, https, tls
  • address address of elasticsearch endpoint
  • port elasticsearch port
  • index index for publishing data
  • type mapping type

You can also set following options if needed:

  • index_timestamp default is set to true (boolean). Set to false if not want to extend index prefix with timestamp
  • publish_fields default is set to Namespace|Config|Data|Description|Timestamp|Unit|Version (string). Remove field from metric structure which shouldn't be published.

Plugin publishes dynamic elements from metric namespace.

Examples

You can find exemplary task manifests for using this plugin in folder examples.

Example running logs collector plugin, logs-regexp processor, logs-openstack processor, and writing data to elasticsearch.

Prepare docker with elasticsearch

$ docker pull elasticsearch
$ docker run --name elasticsearch -p 9200:9200 -p 9300:9300 elasticsearch

In one terminal window, open the Snap daemon:

$ snapteld -t 0 -l 1 -m 999

The option "-l 1" it is for setting the debugging log level, "-t 0" is for disabling plugin signing, and "-m 999" is for the maximum number of instances of a loaded plugin to run.

In another terminal window:

Download and load collector, processor and publisher plugins:

$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-collector-logs/latest/linux/x86_64/snap-plugin-collector-logs
$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-processor-logs-regexp/latest/linux/x86_64/snap-plugin-processor-logs-regexp
$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-processor-logs-openstack/latest/linux/x86_64/snap-plugin-processor-logs-openstack
$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-publisher-elasticsearch/latest/linux/x86_64/snap-plugin-publisher-elasticsearch
$ snaptel plugin load snap-plugin-collector-logs
$ snaptel plugin load snap-plugin-processor-logs-regexp
$ snaptel plugin load snap-plugin-processor-logs-openstack
$ snaptel plugin load snap-plugin-publisher-elasticsearch

Find logs to collect or use exemplary logs available in logs.

Create task manifests, see examples:

If you are using exemplary logs you need to modify "log_dir" in task manifests and set appropriate paths to directories with logs. If you want to collect your own logs you need to adjust config sections in task manifests to your environment.

Create tasks:

$ snaptel task create -t logs-regexp-elasticsearch.json
$ snaptel task create -t logs-openstack-elasticsearch.json

See running tasks:

$ snaptel task list

Check indexes in elasticsearch:

$ curl 'http://localhost:9200/_cat/indices?v'

See data saved in elasticsearch:

$ curl -XGET 'http://localhost:9200/snap*/message/_search?pretty=true&size=9999'

Exemplary data saved in elasticsearch has the form which is shown in outputs.

Roadmap

If you have a feature request, please add it as an issue and/or submit a pull request.

Community Support

This repository is one of many plugins in the Snap, a powerful telemetry agent framework. See the full project at http://github.com/intelsdi-x/snap. To reach out to other users, head to the main framework.

Contributing

We love contributions! 😍

There is more than one way to give back, from examples to blogs to code updates. See our recommended process in CONTRIBUTING.md.

License

Snap, along with this plugin, is an Open Source software released under the Apache 2.0 License.

Acknowledgements

And thank you! Your contribution, through code and participation, is incredibly important to us.