puppet-elasticsearch: A Ruby repository from bisscuitt

#Elasticsearch Puppet module

####Table of Contents

Overview
Module description - What the module does and why it is useful
Setup - The basics of getting started with Elasticsearch

The module manages the following
Requirements

Usage - Configuration options and additional functionality
Advanced features - Extra information on advanced usage
Limitations - OS compatibility, etc.
Development - Guide for contributing to the module
Support - When you need help with this module

##Overview

This module manages Elasticsearch (http://www.elasticsearch.org/overview/elasticsearch/)

##Module description

The elasticsearch module sets up Elasticsearch instances and can manage plugins and templates.

This module has been tested against all versions of ES 1.x and 2.x

##Setup

###The module manages the following

Elasticsearch repository files.
Elasticsearch package.
Elasticsearch configuration file.
Elasticsearch service.
Elasticsearch plugins.
Elasticsearch templates.
Elasticsearch Shield users, roles, and certificates.

###Requirements

The stdlib Puppet library.
ceritsc/yum For yum version lock.
richardc/datacat
Augeas
puppetlabs-java for Java installation (optional).
puppetlabs-java_ks for Shield certificate management (optional).

Repository management

When using the repository management you will need the following dependency modules:

Debian/Ubuntu: Puppetlabs/apt
OpenSuSE/SLES: Darin/zypprepo

##Usage

###Main class

####Install a specific version

class { 'elasticsearch':
  version => '1.4.2'
}

Note: This will only work when using the repository.

Automatically restarting the service (default set to false)

By default, the module will not restart elasticsearch when the configuration file, package, or plugins change. This can be overridden globally with the following option:

class { 'elasticsearch':
  restart_on_change => true
}

Or controlled with the more granular options: restart_config_change, restart_package_change, and restart_plugin_change.

####Automatic upgrade of the software ( default set to false )

class { 'elasticsearch':
  autoupgrade => true
}

####Removal/decommissioning

class { 'elasticsearch':
  ensure => 'absent'
}

####Install everything but disable service(s) afterwards

class { 'elasticsearch':
  status => 'disabled'
}

###Instances

This module works with the concept of instances. For service to start you need to specify at least one instance.

####Quick setup

elasticsearch::instance { 'es-01': }

This will set up its own data directory and set the node name to $hostname-$instance_name

####Advanced options

Instance specific options can be given:

elasticsearch::instance { 'es-01':
  config => { },        # Configuration hash
  init_defaults => { }, # Init defaults hash
  datadir => [ ],       # Data directory
}

See Advanced features for more information

###Plug-ins

Install a variety of plugins. Note that module_dir is where the plugin will install itself to and must match that published by the plugin author; it is not where you would like to install it yourself.

####From official repository

elasticsearch::plugin{'lmenezes/elasticsearch-kopf':
  instances  => 'instance_name'
}

####From custom url

elasticsearch::plugin{ 'jetty':
  url        => 'https://oss-es-plugins.s3.amazonaws.com/elasticsearch-jetty/elasticsearch-jetty-1.2.1.zip',
  instances  => 'instance_name'
}

####Using a proxy You can also use a proxy if required by setting the proxy_host and proxy_port options:

elasticsearch::plugin { 'lmenezes/elasticsearch-kopf',
  instances  => 'instance_name',
  proxy_host => 'proxy.host.com',
  proxy_port => 3128
}

#####Plugin name could be:

elasticsearch/plugin/version for official elasticsearch plugins (download from download.elasticsearch.org)
groupId/artifactId/version for community plugins (download from maven central or oss sonatype)
username/repository for site plugins (download from github master)

####Upgrading plugins When you specify a certain plugin version, you can upgrade that plugin by specifying the new version.

elasticsearch::plugin { 'elasticsearch/elasticsearch-cloud-aws/2.1.1':
}

And to upgrade, you would simply change it to

elasticsearch::plugin { 'elasticsearch/elasticsearch-cloud-aws/2.4.1':
}

Please note that this does not work when you specify 'latest' as a version number.

####ES 2.x official plugins For the Elasticsearch commercial plugins you can refer them to the simple name.

See the Plugin installation for more details.

###Scripts

Install scripts to be used by Elasticsearch. These scripts are shared across all defined instances on the same host.

elasticsearch::script { 'myscript':
  ensure => 'present',
  source => 'puppet:///path/to/my/script.groovy'
}

###Templates

Add a new template using a file

This will install and/or replace the template in Elasticsearch:

elasticsearch::template { 'templatename':
  file => 'puppet:///path/to/template.json',
  # Defaults:
  # host => 'localhost',
  # port => 9200,
}

Add a new template using content

This will install and/or replace the template in Elasticsearch:

elasticsearch::template { 'templatename':
  content => '{"template":"*","settings":{"number_of_replicas":0}}'
}

Delete a template

elasticsearch::template { 'templatename':
  ensure => 'absent'
}

Host

By default it uses localhost:9200 as host. you can change this with the host and port variables

elasticsearch::template { 'templatename':
  host => $::ipaddress,
  port => 9200
}

###Bindings / Clients

Install a variety of clients/bindings:

####Python

elasticsearch::python { 'rawes': }

####Ruby

elasticsearch::ruby { 'elasticsearch': }

###Connection Validator

This module offers a way to make sure an instance has been started and is up and running before doing a next action. This is done via the use of the es_instance_conn_validator resource.

es_instance_conn_validator { 'myinstance' :
  server => 'es.example.com',
  port   => '9200',
}

A common use would be for example :

class { 'kibana4' :
  require => Es_Instance_Conn_Validator['myinstance'],
}

###Package installation

There are 2 different ways of installing the software

####Repository

This option allows you to use an existing repository for package installation. The repo_version corresponds with the major.minor version of Elasticsearch for versions before 2.x.

class { 'elasticsearch':
  manage_repo  => true,
  repo_version => '1.4',
}

For 2.x versions of Elasticsearch, use repo_version => '2.x'.

class { 'elasticsearch':
  manage_repo  => true,
  repo_version => '2.x',
}

####Remote package source

When a repository is not available or preferred you can install the packages from a remote source:

#####http/https/ftp

class { 'elasticsearch':
  package_url       => 'https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.2.deb',
  proxy_url         => 'http://proxy.example.com:8080/',
}

Setting proxy_url to a location will enable download using the provided proxy server. This parameter is also used by elasticsearch::plugin. Setting the port in the proxy_url is mandatory. proxy_url defaults to undef (proxy disabled).

#####puppet://

class { 'elasticsearch':
  package_url => 'puppet:///path/to/elasticsearch-1.4.2.deb'
}

#####Local file

class { 'elasticsearch':
  package_url => 'file:/path/to/elasticsearch-1.4.2.deb'
}

###Java installation

Most sites will manage Java separately; however, this module can attempt to install Java as well. This is done by using the puppetlabs-java module.

class { 'elasticsearch':
  java_install => true
}

Specify a particular Java package/version to be installed:

class { 'elasticsearch':
  java_install => true,
  java_package => 'packagename'
}

###Service management

Currently only the basic SysV-style init and Systemd service providers are supported, but other systems could be implemented as necessary (pull requests welcome).

####Defaults File

The defaults file (/etc/defaults/elasticsearch or /etc/sysconfig/elasticsearch) for the Elasticsearch service can be populated as necessary. This can either be a static file resource or a simple key value-style hash object, the latter being particularly well-suited to pulling out of a data source such as Hiera.

#####file source

class { 'elasticsearch':
  init_defaults_file => 'puppet:///path/to/defaults'
}

#####hash representation

$config_hash = {
  'ES_HEAP_SIZE' => '30g',
}

class { 'elasticsearch':
  init_defaults => $config_hash
}

Note: init_defaults hash can be passed to the main class and to the instance.

Advanced features

Shield

Shield users, roles, and certificates can be managed by this module.

Note: If you are planning to use these features, it is highly recommended you read the following documentation to understand the caveats and extent of the resources available to you.

Getting Started

Although this module can handle several types of Shield resources, you are expected to manage the plugin installation and versions for your deployment. For example, the following manifest will install Elasticseach with a single instance running shield:

class { 'elasticsearch':
  java_install => true,
  manage_repo  => true,
  repo_version => '1.7',
}

elasticsearch::instance { 'es-01': }

Elasticsearch::Plugin { instances => ['es-01'], }
elasticsearch::plugin { 'elasticsearch/license/latest': }
elasticsearch::plugin { 'elasticsearch/shield/latest': }

The following examples will assume the preceding resources are part of your puppet manifest.

Roles

Roles in the esusers realm can be managed using the elasticsearch::shield::role type. For example, to create a role called myrole, you could use the following resource:

elasticsearch::shield::role { 'myrole':
  privileges => {
    'cluster' => 'monitor',
    'indices' => {
      '*' => 'read'
    }
  }
}

This role would grant users access to cluster monitoring and read access to all indices. See the Shield documentation for your version to determine what privileges to use and how to format them (the Puppet hash representation will simply be translated into yaml.)

Note: The Puppet provider for esusers has fine-grained control over the roles.yml file and thus will leave the default roles Shield installs in-place. If you would like to explicitly purge the default roles (leaving only roles managed by puppet), you can do so by including the following in your manifest:

resources { 'elasticsearch_shield_role':
  purge => true,
}

Mappings

Associating mappings with a role is done by passing an array of strings to the mappings parameter of the elasticsearch::shield::role type. For example, to define a role with mappings using Shield >= 2.3.x style role definitions:

elasticsearch::shield::role { 'logstash':
  mappings   => [
    'cn=group,ou=devteam',
  ],
  privileges => {
    'cluster' => 'manage_index_templates',
    'indices' => [{
      'names'      => ['logstash-*'],
      'privileges' => [
        'write',
        'delete',
        'create_index',
      ],
    }],
  },
}

Note: Observe the brackets around indices in the preceding role definition; which is an array of hashes per the format in Shield 2.3.x. Follow the documentation to determine the correct formatting for your version of Shield.

If you'd like to keep the mappings file purged of entries not under Puppet's control, you should use the following resources declaration because mappings are a separate low-level type:

resources { 'elasticsearch_shield_role_mapping':
  purge => true,
}

Users

Users can be managed using the elasticsearch::shield::user type. For example, to create a user mysuser with membership in myrole:

elasticsearch::shield::user { 'myuser':
  password => 'mypassword',
  roles    => ['myrole'],
}

The password parameter will also accept password hashes generated from the esusers utility and ensure the password is kept in-sync with the Shield users file for all Elasticsearch instances.

elasticsearch::shield::user { 'myuser':
  password => '$2a$10$IZMnq6DF4DtQ9c4sVovgDubCbdeH62XncmcyD1sZ4WClzFuAdqspy',
  roles    => ['myrole'],
}

Note: When using the esusers provider (the default for plaintext passwords), Puppet has no way to determine whether the given password is in-sync with the password hashed by Shield. In order to work around this, the elasticsearch::shield::user resource has been designed to accept refresh events in order to update password values. This is not ideal, but allows you to instruct the resource to change the password when needed. For example, to update the aforementioned user's password, you could include the following your manifest:

notify { 'update password': } ~>
elasticsearch::shield::user { 'myuser':
  password => 'mynewpassword',
  roles    => ['myrole'],
}

Certificates

SSL/TLS can be enabled by providing an elasticsearch::instance type with paths to the certificate and private key files, and a password for the keystore.

elasticsearch::instance { 'es-01':
  ssl                  => true,
  ca_certificate       => '/path/to/ca.pem',
  certificate          => '/path/to/cert.pem',
  private_key          => '/path/to/key.pem',
  keystore_password    => 'keystorepassword',
}

Note: Setting up a proper CA and certificate infrastructure is outside the scope of this documentation, see the aforementioned Shield guide for more information regarding the generation of these certificate files.

The module will set up a keystore file for the node to use and set the relevant options in elasticsearch.yml to enable TLS/SSL using the certificates and key provided.

###Package version pinning

The module supports pinning the package version to avoid accidental upgrades that are not done by Puppet. To enable this feature:

class { 'elasticsearch':
  package_pin => true,
  version     => '1.5.2',
}

In this example we pin the package version to 1.5.2.

###Data directories

There are 4 different ways of setting data directories for Elasticsearch. In every case the required configuration options are placed in the elasticsearch.yml file.

####Default By default we use:

/usr/share/elasticsearch/data/$instance_name

Which provides a data directory per instance.

####Single global data directory

class { 'elasticsearch':
  datadir => '/var/lib/elasticsearch-data'
}

Creates the following for each instance:

/var/lib/elasticsearch-data/$instance_name

####Multiple Global data directories

class { 'elasticsearch':
  datadir => [ '/var/lib/es-data1', '/var/lib/es-data2']
}

Creates the following for each instance: /var/lib/es-data1/$instance_name and /var/lib/es-data2/$instance_name

####Single instance data directory

class { 'elasticsearch': }

elasticsearch::instance { 'es-01':
  datadir => '/var/lib/es-data-es01'
}

Creates the following for this instance: /var/lib/es-data-es01

####Multiple instance data directories

class { 'elasticsearch': }

elasticsearch::instance { 'es-01':
  datadir => ['/var/lib/es-data1-es01', '/var/lib/es-data2-es01']
}

Creates the following for this instance: /var/lib/es-data1-es01 and /var/lib/es-data2-es01

###Main and instance configurations

The config option in both the main class and the instances can be configured to work together.

The options in the instance config hash will merged with the ones from the main class and override any duplicates.

Simple merging

class { 'elasticsearch':
  config => { 'cluster.name' => 'clustername' }
}

elasticsearch::instance { 'es-01':
  config => { 'node.name' => 'nodename' }
}
elasticsearch::instance { 'es-02':
  config => { 'node.name' => 'nodename2' }
}

This example merges the cluster.name together with the node.name option.

Overriding

When duplicate options are provided, the option in the instance config overrides the ones from the main class.

class { 'elasticsearch':
  config => { 'cluster.name' => 'clustername' }
}

elasticsearch::instance { 'es-01':
  config => { 'node.name' => 'nodename', 'cluster.name' => 'otherclustername' }
}

elasticsearch::instance { 'es-02':
  config => { 'node.name' => 'nodename2' }
}

This will set the cluster name to otherclustername for the instance es-01 but will keep it to clustername for instance es-02

####Configuration writeup

The config hash can be written in 2 different ways:

Full hash writeup

Instead of writing the full hash representation:

class { 'elasticsearch':
  config                 => {
   'cluster'             => {
     'name'              => 'ClusterName',
     'routing'           => {
        'allocation'     => {
          'awareness'    => {
            'attributes' => 'rack'
          }
        }
      }
    }
  }
}

Short hash writeup

class { 'elasticsearch':
  config => {
    'cluster' => {
      'name' => 'ClusterName',
      'routing.allocation.awareness.attributes' => 'rack'
    }
  }
}

##Limitations

This module has been built on and tested against Puppet 3.2 and higher.

The module has been tested on:

Debian 6/7/8
CentOS 6/7
Ubuntu 12.04, 14.04
OpenSuSE 13.x
SLES 12

Other distro's that have been reported to work:

RHEL 6
OracleLinux 6
Scientific 6

Testing on other platforms has been light and cannot be guaranteed.

##Development

Please see the [CONTRIBUTING.md][CONTRIBUTING.md] file for instructions regarding development environments and testing.

##Support

Need help? Join us in #elasticsearch on Freenode IRC or on the discussion forum.

bisscuitt/puppet-elasticsearch