fluentd

Overview
Module Description - What the module does and why it is useful
Setup - The basics of getting started with fluentd
Usage - Configuration options and additional functionality
Todo List - What needs to be done
Development - Guide for contributing to the module

Overview

Manage a FluentD installation from the community managed Gem, and provide configuration for your environment using a set of puppet resources.

Module Description

This module attempts to provide a mechanism to manage a FluentD installation using the gem daemons rather than the td-agent packages which are not as well supported on some systems.

The code has been heavily based on the Puppet module created by mms-srf as I wanted to try and solve a few perceived problems with that implementation:

Avoid use of td-agent and instead use fluentd gem directly
Improve flexibility of configuration by enabling multiple nested elements and arrays without hardcoding

Setup

What fluentd affects

Installs the fluentd gem
Creates a minimum set of configuration files and directories
Provides resources to create additional named configuration files to be included automatically
Manages the fluentd service

Configuration

How to configure the fluentd agent to send data to a centralised Fluentd-Server

Install a Required Plugin

WARNING: Plugin configuration is not complete. Do not attempt to use this yet

Install your fluentd plugin. (Check here for the right plugin name.)

You can choose from a file or gem based installation.

include ::fluentd

fluentd::plugin { 'elasticsearch':
  plugin_type => 'gem',
  plugin_name => 'fluent-plugin-elasticsearch',
}

Configure a Source

Sources describe to fluentd where to obtain its data to process. This can include reading log files, opening tcp ports, running http services etc.

include ::fluentd

fluentd::source { 'apache':
  config => {
    'format'   => 'apache2',
    'path'     => '/var/log/apache2/access.log',
    'pos_file' => '/var/tmp/fluentd.pos',
    'tag'      => 'apache.access_log',
    'type'     => 'tail',
  },
}

fluentd::source { 'syslog':
  config => {
    'format'   => 'syslog',
    'path'     => '/var/log/syslog',
    'pos_file' => '/tmp/td-agent.syslog.pos',
    'tag'      => 'system.syslog',
    'type'     => 'tail',
  },
}

Match configuration

fluentd::match { 'forward':
  pattern  => '**',
  priority => '80',
  config   => {
    'type'    => 'forward',
    'servers' => [
      { 'host' => 'fluentd.example.com', 'port' => '24224' }
    ],
  },
}

Forest Plugin configuration

This is an example of having key/value configuration for nested elements other than the 'server' elements usually seen - This example is based on using the forest configuration plugin to manage the configuration of a fluentd-plugin-elasticsearch gem.

fluentd::match { 'forest-es':
  pattern  => '**',
  priority => '10',
  config   => {
    'type'    => 'forst',
    'subtype' => 'elasticsearch',
    'remove_prefix' => 'my.logs',
    'template' => [
      { 
        'host' => 'elasticsearch.example.com', 
        'port' => 9200 
        'logstash_prefix' => ${tag[1]},
      }
    ],
  },
}

Forwarder with Secondary

This is a very complicated example, based on the documentation of an out forwarder

fluentd::match { 'forwarder-safe':
  pattern  => '**',
  priority => '10',
  config   => {
    'type'  =>  'forward',
    'send_timeout'  =>  '60s',
    'recover_wait'  =>  '10s',
    'heartbeat_interval'  =>  '1s',
    'phi_threshold'  =>  '16',
    'hard_timeout'  =>  '60s',

    'server' => [
    {
       'name'  =>  'myserver1',
       'host'  =>  '192.168.1.3',
       'port'  =>  '24224',
       'weight'  =>  '60',
     },{
       'name'  =>  'myserver2',
       'host'  =>  '192.168.1.4',
       'port'  =>  '24224',
       'weight'  =>  '60',
     }
    ],
    'secondary' => {
     'type'  =>  'file',
     'path'  =>  '/var/log/fluent/forward-failed',
    }
  }
}

TODO

[ ] - Plugin management [ ] - Remove MASSIVE duplication of effort between source and match config - its basically the same thing twice

warmfusion/puppet-fluentd