Add support for elastic search, log stash, kibana stack

[danielcompton]

It would be really handy to be able to view aggregated logs from all of the machines. Is it within the scope of wirbelsturm to add support for the ELK stack or any other log aggregator?

[miguno]

Sure, ELK support would be a really nice addition.

A general note: In some some you don't add ELK support "to Wirbelsturm". You can deploy any kind of software via Wirbelsturm as long as you have a Puppet module, and your custom Puppet module(s) don't need to be officially "added" to Wirbelsturm. Simply add your modules to puppet/Puppetfile for librarian-puppet, and make sure you have e.g. a working Hiera setup.

Hence the question is rather: Should the additional Puppet modules etc. be bundled out of the box in Wirbelsturm?

All these clarifications aside, in the particular case of ELK I'd really like to bundle the ELK stack as part of the default Wirbelsturm setup. :-)

You can take a look at the current "bundled" Puppet modules such as puppet-kafka or puppet-storm for examples of the code style/Puppet patterns I would be looking for. I don't want to enforce "our" Puppet style etc. on other people (like you), but I'd prefer that all bundled Puppet modules in Wirbelsturm follow the same style so that Wirbelsturm users have an easier time to navigate through and understand the Puppet code.

Would that work for you?

[danielcompton]

Using hieradata and your general puppet coding style is the only one I've seen that seems sane to me so that's not a problem :)

Configuring logstash on each of the boxes would be a matter of reaching in to the other puppet modules to find out where their logs are being stored. I'm not sure if this is kosher in Puppet land. A more advanced setup would be to have the applications send their logs to syslog, though this probably isn't necessary.

I'm not sure how to structure this code as the log stash binary will need to be running on every host, and I think config added to every module. Would we add a conditional flag in the top level to switch this on or off? It doesn't feel quite right but I'm not sure how else you could do this.

[miguno]

Configuring logstash on each of the boxes would be a matter of reaching in to the other puppet modules to find out where their logs are being stored. I'm not sure if this is kosher in Puppet land.

Yeah, such inter-dependencies between Puppet modules are unfortunately not a place you'd like to be in. :-)

What we could start out with is to provide a default configuration for logstash that is compatible with the default Wirbelsturm setup (e.g. location of Storm logs). To make this work once you start customizing your Wirbelsturm setup ("Let me also deploy a Kafka machine!") we'd need a way to configure logstash -- and where it will look for log files -- through Hiera.

I'm not sure how to structure this code as the log stash binary will need to be running on every host, and I think config added to every module.

We can install logstash by default using the Hiera data of the "default environment": default-environment.yaml. Here's how this could look like (I'm totally making this up):

classes:
  - supervisor
  - timezone
  - logstash     # <<<<  install logstash binary on all machines
  ...

Again, the key will be to write the Puppet module for logstash in such a way that you can control its behavior properly through Hiera.

Does that feedback help to get you started?

[Iheartweb]

@danielcompton Have you made any headway with this? I'm going down a similar path and would love to know if this worked for you.

[danielcompton]

@Iheartweb I haven't got to it yet, still on my list of things I'd really like to do though. Happy to help you on it as well if you get there first.

[Iheartweb]

@danielcompton Thanks. I'm trying it out now - If I am successful - I'll let you know.

[Iheartweb]

@miguno, @danielcompton - Unfortunately ES fails to start on CentOS 6. See voxpupuli/puppet-elasticsearch/issues/176

Here's a few code snippets for ya:

In wirbelstrum.yamal:

  # Deploys ES.
  elasticsearch_node:
    count: 1
    hostname_prefix: elasticsearch
    ip_range_start: 10.0.0.30
    node_role: elasticsearch_node
    providers:
      virtualbox:
        memory: 1536
        forwarded_ports:
          - guest: 9200
            host: 9200
      aws:
        instance_type: m2.2xlarge
        ami: ami-abc12345
        security_groups:
          - wirbelsturm

In Puppetfile

mod 'elasticsearch/elasticsearch',
  :git => 'https://github.com/elasticsearch/puppet-elasticsearch.git',
  :ref => '0.4.0'

In roles/elasticsearch_node.yaml

---
classes:
  - elasticsearch

elasticsearch::manage_repo: true
elasticsearch::repo_version: '1.3'
elasticsearch::java_install: true

Do you see any user errors here? Is there something I overlooked?

[miguno]

@Iheartweb Robert, your last comment slipped under my radar. I don't see any error that jumps at me. Have you been able to make further progress?

[miguno]

Closed because of inactivity. Please reopen if you want to continue working on this!