/monitoring

Scripts to configure various monitoring services on Ubuntu 14.04

Primary LanguageShell

Monitoring

Set of scripts for evaluating various monitoring stacks (log aggregation, metrics collection, metrics correlation, etc.)

Currently, the following stacks can be installed.

This repository started off from the ideas on my Monitoring Everything blog series.

Goals

Scripts included in this project must:

  • Follow best practices for Ubuntu and the applications being installed
  • Document service boundaries (ports, sockets, etc.) and default credentials
  • Be as minimal as possible
  • Be as reusable as possible (middleware and persistence, for example, can be reused between applications)
  • Try to stay close to each application's defaults (specifically port numbers and file locations)

This makes it easier for people to solve issues during evaluation, as well as prepare for deploying the applications into staging and production environments.

I won't accept scripts for the following applications. These applications are either too old or too bloated for cloud monitoring.

  • Nagios
  • Zabbix

Contributing

Want to add additional scripts? Improve existing scenarios? Just fork it, and submit a pull request. It's that simple.

Usage

Two Vagrant boxes are provided with this script.

  • monitoring on 192.168.12.10
  • app1 on 192.168.12.11

The monitoring box is intended for the different monitoring stacks, while app1 is intended to try out clients.

Sensu/Logstash

The monitoring box provides the following:

For client nodes, it provides:

  • AMQP (TCP/5672)
  • AMQP over SSL (TCP/5671)
  • Redis (TCP/6379)
  • Statsd (UDP/8125)
  • Lumberjack receiver (TCP/5043)
  • Graphite line receiver (TCP/2013)
  • Graphite Pickle receiver (TCP/2014)

OpenTSDB

This package provides scripts to install OpenTSDB and TCollector. OpenTSDB depends on HBase, which is installed in pseudo-distributed mode alongside a standalone ZooKeeper.

vagrant up --no-provision monitoring
vagrant ssh monitoring
sudo -i
cd /vagrant && ./install-all-opentsdb.sh

This script also installs Grafana with the OpenTSDB backend configured. Elasticsearch is installed as a dashboard store for Grafana.

TCollector is installed on the monitoring host to provide some sample metrics. Note that this script can take more than 10 minutes to install, depending on your bandwidth.

To start collecting metrics from app1, simply SSH to it and run the following.

sudo -i
cd /vagrant && ./install-tcollector.sh

Sentry

This package provides scripts to install Sentry with the following configuration.

  • memcached
  • supervisord
  • Redis as work queue, update buffer and TSDB
  • PostgreSQL as node store

You can use the following to perform the installation.

vagrant up --no-provision monitoring
vagrant ssh monitoring
sudo -i
cd /vagrant && ./install-all-sentry.sh

After installation, the Sentry web interface should be available. Login with the username admin and the password secret.

Graylog2

To install Graylog2 with the web interface and stream dashboard, simply use the following.

vagrant up --no-provision monitoring
vagrant ssh monitoring
sudo -i
cd /vagrant && ./install-all-graylog2.sh

After installation, you can access one of the following resources. Use the username admin and the passwod password.

Note that Graylog2 requires its own Elasticsearch cluster, so do not use this script on the same node as other installation scripts.

Heka

Heka was created by Mozilla as a lighter alternative to Logstash. This package provides scripts for installing a Heka router that outputs to Elasticsearch.

vagrant up --no-provision monitoring
vagrant ssh monitoring
sudo -i
cd /vagrant && ./install-all-heka.sh

This package provides the following:

InfluxDB

InfluxDB is an open-source distributed time series database with no external dependencies. This package provides scripts for using InfluxDB as a general replacement for Graphite. It will install Sensu and InfluxDB, configuring Sensu to relay metrics to InfluxDB.

vagrant up --no-provision monitoring
vagrant ssh monitoring
sudo -i
cd /vagrant && ./install-all-influxdb.sh

This package provides the following:

Use vagrant up app1 to start collecting metrics

Flume

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. This package provides scripts for using Flume with Elasticsearch and receivers for Avro and Syslog protocols.

vagrant up --no-provision monitoring
vagrant ssh monitoring
sudo -i
cd /vagrant && ./install-all-flume.sh

This package provides the following:

  • Kibana
  • ElasticHQ
  • Avro input on 192.168.12.10 at TCP port 41414
  • Syslog input on 192.168.12.10 at TCP port 1514

When using Kibana, you will need to change the index pattern to [flume-]YYYY-MM-DD.

Fluentd

Fluentd is an open source data collector to unify log management. This package provides scripts for using Fluentd with Elasticsearch and receivers for Syslog and HTTP protocols.

vagrant up --no-provision monitoring
vagrant ssh monitoring
sudo -i
cd /vagrant && ./install-all-fluentd.sh

This package provides the following:

  • Kibana
  • ElasticHQ
  • HTTP input on 192.168.12.10 at TCP port 9880
  • Syslog input on 192.168.12.10 at TCP port 1514

Testing Syslog Receivers

To quickly test the functionality of a syslog-compatible receiver, you can use the logger command on Ubuntu.

message="hello world"

# With TCP syslog receiver
logger -n localhost -P 1514 $message

# With UDP syslog receiver
logger -n localhost -P 1514 -d $message