/logstash-input-kinesis

Logstash Plugin for AWS Kinesis Input

Primary LanguageRubyOtherNOASSERTION

Logstash AWS Kinesis Input Plugin

Build Status

This is a AWS Kinesis input plugin for Logstash. Under the hood uses the Kinesis Client Library.

Installation

This plugin requires Logstash >= 2.0, and can be installed by Logstash itself.

bin/logstash-plugin install logstash-input-kinesis

Usage

input {
  kinesis {
    kinesis_stream_name => "my-logging-stream"
    codec => json { }
  }
}

Using with CloudWatch Logs

If you are looking to read a CloudWatch Logs subscription stream, you'll also want to install and configure the CloudWatch Logs Codec.

Configuration

This are the properties you can configure and what are the default values:

  • application_name: The name of the application used in DynamoDB for coordination. Only one worker per unique stream partition and application will be actively consuming messages.
    • required: false
    • default value: logstash
  • kinesis_stream_name: The Kinesis stream name.
    • required: true
  • region: The AWS region name for Kinesis, DynamoDB and Cloudwatch (if enabled)
    • required: false
    • default value: us-east-1
  • checkpoint_interval_seconds: How many seconds between worker checkpoints to DynamoDB. A low value ussually means lower message replay in case of node failure/restart but it increases CPU+network ussage (which increases the AWS costs).
    • required: false
    • default value: 60
  • metrics: Worker metric tracking. By default this is disabled, set it to "cloudwatch" to enable the cloudwatch integration in the Kinesis Client Library.
    • required: false
    • default value: nil
  • profile: The AWS profile name for authentication. This ensures that the ~/.aws/credentials AWS auth provider is used. By default this is empty and the default chain will be used.
    • required: false
    • default value: ""
  • initial_position_in_stream: The value for initialPositionInStream. Accepts "TRIM_HORIZON" or "LATEST".
    • required: false
    • default value: "TRIM_HORIZON"

Authentication

This plugin uses the default AWS SDK auth chain, DefaultAWSCredentialsProviderChain, to determine which credentials the client will use, unless profile is set, in which case ProfileCredentialsProvider is used.

The default chain follows this order trying to read the credentials:

  • AWS_ACCESS_KEY_ID / AWS_SECRET_KEY environment variables
  • ~/.aws/credentials credentials file
  • EC2 instance profile

The credentials will need access to the following services:

  • AWS Kinesis
  • AWS DynamoDB: the client library stores information for worker coordination in DynamoDB (offsets and active worker per partition)
  • AWS CloudWatch: if the metrics are enabled the credentials need CloudWatch update permisions granted.

Look at the documentation for deeper information on the default chain.

Contributing

  1. https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md#contribution-steps
  2. Fork it ( https://github.com/logstash-plugins/logstash-input-kinesis/fork )
  3. Create your feature branch (git checkout -b my-new-feature)
  4. Commit your changes (git commit -am 'Add some feature')
  5. Push to the branch (git push origin my-new-feature)
  6. Create a new Pull Request