/mod-mongo-logs

Store Shinken logs in MongoDB without LiveStatus module

Primary LanguagePythonGNU Affero General Public License v3.0AGPL-3.0

Shinken logs MongoDB storage

Shinken broker module used to :

  • store Shinken logs in a MongoDB database
  • store hosts/services availability data in a MongoDB database

No more need for Livestatus to store Shinken logs and hosts availability data!

Note: This module is a must-have for the new Web UI features: host/service history, Shinken activity, hosts availability, ...

Main features

This module intercepts Shinken logs and analyses each log to store it in MongoDB collection. The logs are then available for an application such as Web UI: see https://github.com/shinken-monitoring/mod-webui.

This module also intercepts Shinken hosts and services checks results to store daily data in MongoDB collection. For each concerned host and service, the data stored allow to have a daily availability for the host/service.

Concerned hosts and services are defined as is:

  • host availability is always stored
  • service availability is stored depending upon a configurable filter:
    • regexp on the service description
    • rules on the service business impact For more information see, examples hereunder and configuration file that is documented.

For each host/service, the availability follow the rules:

  • one record per service per day
  • for each day, 86400 seconds
  • first check result received is stored (state and timestamp)
  • last check result received is stored (state and timestamp)
  • day period outside of first/last check is considered as unchecked (daily_4 seconds)
  • each state duration seconds is stored (daily_0, daily_1, daily_2)

With this information it is possible to know how many seconds is spent in a certain state

Note: the pros of this method is that is is really a simple and 'real time' time to get availability for an host/service. Np need to parse a huge amount of logs to get some data!

Note: the only cons of this method is that it is not very precise ... based on the first/last daily received check, it does not care about the previous day last state!

Requirements

Use pymongo version > 3.0.

   pip install pymongo>=3

Enabling Mongo logs

To use the mongo-logs module you must declare it in your broker configuration.

   define broker {
      ...

      modules    	 ..., mongo-logs

   }

The module configuration is defined in the file: mongo-logs.cfg.

Default configuration needs to be tuned up to your MongoDB configuration.

## Module:      mongo-logs
## Loaded by:   Broker
# Store the Shinken logs in a mongodb database
# Store hosts/services availability in a mongodb database
#
# This module is necessary if you intend to use the logs and availability features offered
# by the Shinken WebUI2
#
# -----------------
# IMPORTANT ADVICE:
# -----------------
# If you change the default configuration in this file, you MUST copy the same configuration
# parameters in your webui2.cfg file.
#
# Please note that the max_logs_age parameter is not used in the WebUI
#
define module {
   module_name         mongo-logs
   module_type         mongo-logs

   # MongoDB connection string
   # EXAMPLE
   # To describe a connection to a replica set named test, with the following mongod hosts:
   #   db1.example.net on port 27017 with sysop credentials and
   #   db2.example.net on port 2500.
   # You would use a connection string that resembles the following:
   #   uri     mongodb://sysop:password@db1.example.net,db2.example.net:2500/?replicaSet=test
   #
   # Default is a non replicated localhost server
   #uri                  mongodb://localhost

   # Database name where to store the logs/availability collection
   # Default is shinken
   #database             shinken

   # DB connection test period
   # Every db_test_period seconds, the module tests if the DB connection is alive
   # Default is 0 to skip this test
   #db_test_period    300

   ### ------------------------------------------------------------------------
   ### Logs management
   ### ------------------------------------------------------------------------
   # Logs collection name
   # Default is a collection named logs
   #logs_collection      logs

   # Logs rotation
   #
   # Remove logs older than the specified value
   # Value is specified as :
   # 1d: 1 day
   # 3m: 3 months ...
   # d = days, w = weeks, m = months, y = years
   # Default is 3 months
   #max_logs_age    3m

   # Commit volume
   # The module commits at most commit_volume logs in the DB at every commit period
   # Default is 1000 lines
   #commit_volume     1000

   # Commit period
   # Every commit_period seconds, the module stores the received logs in the DB
   # Default is to commit every 60 seconds
   #commit_period     60

   ### ------------------------------------------------------------------------
   ### Hosts/services availability management
   ### ------------------------------------------------------------------------
   # Hosts/services availability collection name
   # Default is a collection named availability
   #hav_collection      availability

   # Services filtering
   # Filter is declared as a comma separated list of items:
   # An item can be a regexp which is matched against service description (hostname/service)
   #  ^test*, matches all hosts which name starts with test
   #  /test*, matches all services which name starts with test
   #
   # An item containing : is a specific filter (only bi is supported currently)
   #  bi:>x, bi:>=x, bi:<x, bi:<=x, bi:=x to match business impact

   # default is to consider only the services which business impact is > 4
   # 3 is the default value for business impact if property is not explicitely declared
   # Default is only bi>4 (most important services)
   #services_filter bi:>4
}

Doc

Logs collection

Logs are stored in a collection which default name is logs

Each document in the collection contain always the same fields wichever is the stored log line:

  • _id: added by mongodb when document is inserted

  • type:

    • INFO, DEBUG, WARNING, ERROR for specific Shinken logs, or
    • NAGIOS log type (eg. SERVICE ALERT, PASSIVE HOST CHECK, ...)

  • logobject: monitoring object concerned by the log line

    LOGOBJECT_INFO = 0
    LOGOBJECT_HOST = 1
    LOGOBJECT_SERVICE = 2
    LOGOBJECT_CONTACT = 3
  • logclass:
    LOGCLASS_INFO = 0          # all messages not in any other class
    LOGCLASS_ALERT = 1         # alerts: the change service/host state
    LOGCLASS_PROGRAM = 2       # important program events (restart, ...)
    LOGCLASS_NOTIFICATION = 3  # host/service notifications
    LOGCLASS_PASSIVECHECK = 4  # passive checks
    LOGCLASS_COMMAND = 5       # external commands
    LOGCLASS_STATE = 6         # initial or current states
  • time: log line UTC timestamp
  • message: complete log line message

The following fields are always present but their value are completed depending upon the message logclass value:

  • comment
  • plugin_output
  • attempt
  • options
  • state_type
  • state
  • host_name
  • service_description
  • contact_name
  • command_name

Logs stored in the mongodb collection (example):

    { "_id" : "shinken-test", "last_test" : 1441968270.48028 } ,
    { "_id" : { "$oid" : "55f118cca5d69827ccea9520" }, "comment" : "", "plugin_output" : "", "attempt" : 0, "message" : "[1441863874] TIMEPERIOD TRANSITION: workhours;-1;0", "logclass" : 2, "options" : "", "state_type" : "", "lineno" : 614, "state" : 0, "host_name" : "", "time" : 1441863874, "service_description" : "", "logobject" : 0, "type" : "TIMEPERIOD TRANSITION", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f118cca5d69827ccea9521" }, "comment" : "", "plugin_output" : "", "attempt" : 0, "message" : "[1441863870] INFO: [Shinken] OK, all schedulers configurations are dispatched :)\n", "logclass" : 2, "options" : "", "state_type" : "", "lineno" : 615, "state" : 0, "host_name" : "", "time" : 1441863870, "service_description" : "", "logobject" : 0, "type" : "INFO", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f118cca5d69827ccea9522" }, "comment" : "", "plugin_output" : "", "attempt" : 0, "message" : "[1441863870] INFO: [Shinken] [All] Trying to send configuration to receiver receiver-master", "logclass" : 2, "options" : "", "state_type" : "", "lineno" : 616, "state" : 0, "host_name" : "", "time" : 1441863870, "service_description" : "", "logobject" : 0, "type" : "INFO", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f118eaa5d69827ccea95cf" }, "comment" : "", "plugin_output" : "ERROR : this plugin needs the python-paramiko module. Please install it", "attempt" : 1, "message" : "[1441863885] SERVICE ALERT: pi2;CPU Stats;CRITICAL;SOFT;1;ERROR : this plugin needs the python-paramiko module. Please install it", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 789, "state" : 2, "host_name" : "pi2", "time" : 1441863885, "service_description" : "CPU Stats", "logobject" : 2, "type" : "SERVICE ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f118eaa5d69827ccea95d0" }, "comment" : "", "plugin_output" : "ERROR : this plugin needs the python-paramiko module. Please install it", "attempt" : 1, "message" : "[1441863885] SERVICE ALERT: pi2;Reboot;CRITICAL;SOFT;1;ERROR : this plugin needs the python-paramiko module. Please install it", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 790, "state" : 2, "host_name" : "pi2", "time" : 1441863885, "service_description" : "Reboot", "logobject" : 2, "type" : "SERVICE ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f118eaa5d69827ccea95d1" }, "comment" : "", "plugin_output" : "check_ping: Invalid hostname/address - pi2", "attempt" : 1, "message" : "[1441863885] HOST ALERT: pi2;DOWN;SOFT;1;check_ping: Invalid hostname/address - pi2", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 791, "state" : 1, "host_name" : "pi2", "time" : 1441863885, "service_description" : "", "logobject" : 1, "type" : "HOST ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f118f5a5d69827ccea95d2" }, "comment" : "", "plugin_output" : "check_ping: Invalid hostname/address - pi2", "attempt" : 2, "message" : "[1441863887] HOST ALERT: pi2;DOWN;HARD;2;check_ping: Invalid hostname/address - pi2", "logclass" : 1, "options" : "", "state_type" : "HARD", "lineno" : 792, "state" : 1, "host_name" : "pi2", "time" : 1441863887, "service_description" : "", "logobject" : 1, "type" : "HOST ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f118f5a5d69827ccea95d3" }, "comment" : "", "plugin_output" : "", "attempt" : 0, "message" : "[1441863887] HOST NOTIFICATION: admin;pi2;DOWN;notify-host-by-email;check_ping: Invalid hostname/address - pi2", "logclass" : 3, "options" : "", "state_type" : "DOWN", "lineno" : 793, "state" : 1, "host_name" : "pi2", "time" : 1441863887, "service_description" : "", "logobject" : 1, "type" : "HOST NOTIFICATION", "contact_name" : "admin", "command_name" : "notify-host-by-email" } ,
    { "_id" : { "$oid" : "55f118f5a5d69827ccea95d4" }, "comment" : "", "plugin_output" : "check_ping: Invalid hostname/address - pi1", "attempt" : 1, "message" : "[1441863895] HOST ALERT: pi1;DOWN;SOFT;1;check_ping: Invalid hostname/address - pi1", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 794, "state" : 1, "host_name" : "pi1", "time" : 1441863895, "service_description" : "", "logobject" : 1, "type" : "HOST ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f118f5a5d69827ccea95d5" }, "comment" : "", "plugin_output" : "ERROR : this plugin needs the python-paramiko module. Please install it", "attempt" : 1, "message" : "[1441863901] SERVICE ALERT: pi1;NET Stats;CRITICAL;SOFT;1;ERROR : this plugin needs the python-paramiko module. Please install it", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 795, "state" : 2, "host_name" : "pi1", "time" : 1441863901, "service_description" : "NET Stats", "logobject" : 2, "type" : "SERVICE ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f118f5a5d69827ccea95d6" }, "comment" : "", "plugin_output" : "check_ping: Invalid hostname/address - pi1", "attempt" : 1, "message" : "[1441863903] HOST ALERT: pi1;DOWN;SOFT;1;check_ping: Invalid hostname/address - pi1", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 796, "state" : 1, "host_name" : "pi1", "time" : 1441863903, "service_description" : "", "logobject" : 1, "type" : "HOST ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f118f5a5d69827ccea95d7" }, "comment" : "", "plugin_output" : "check_ping: Invalid hostname/address - graphite", "attempt" : 1, "message" : "[1441863911] HOST ALERT: graphite;DOWN;SOFT;1;check_ping: Invalid hostname/address - graphite", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 797, "state" : 1, "host_name" : "graphite", "time" : 1441863911, "service_description" : "", "logobject" : 1, "type" : "HOST ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f118f5a5d69827ccea95d8" }, "comment" : "", "plugin_output" : "ERROR : this plugin needs the python-paramiko module. Please install it", "attempt" : 1, "message" : "[1441863915] SERVICE ALERT: pi1;Kernel Stats;CRITICAL;SOFT;1;ERROR : this plugin needs the python-paramiko module. Please install it", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 798, "state" : 2, "host_name" : "pi1", "time" : 1441863915, "service_description" : "Kernel Stats", "logobject" : 2, "type" : "SERVICE ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f118f5a5d69827ccea95d9" }, "comment" : "", "plugin_output" : "check_ping: Invalid hostname/address - pi1", "attempt" : 1, "message" : "[1441863917] HOST ALERT: pi1;DOWN;SOFT;1;check_ping: Invalid hostname/address - pi1", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 799, "state" : 1, "host_name" : "pi1", "time" : 1441863917, "service_description" : "", "logobject" : 1, "type" : "HOST ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f118f5a5d69827ccea95da" }, "comment" : "", "plugin_output" : "ERROR : this plugin needs the python-paramiko module. Please install it", "attempt" : 1, "message" : "[1441863924] SERVICE ALERT: pi1;NFS Stats;CRITICAL;SOFT;1;ERROR : this plugin needs the python-paramiko module. Please install it", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 800, "state" : 2, "host_name" : "pi1", "time" : 1441863924, "service_description" : "NFS Stats", "logobject" : 2, "type" : "SERVICE ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f118ffa5d69827ccea95db" }, "comment" : "", "plugin_output" : "check_ping: Invalid hostname/address - pi1", "attempt" : 1, "message" : "[1441863927] HOST ALERT: pi1;DOWN;SOFT;1;check_ping: Invalid hostname/address - pi1", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 801, "state" : 1, "host_name" : "pi1", "time" : 1441863927, "service_description" : "", "logobject" : 1, "type" : "HOST ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f118ffa5d69827ccea95dc" }, "comment" : "", "plugin_output" : "", "attempt" : 0, "message" : "[1441863933] INFO: [broker-master] We have our schedulers: {0: {'broks': {}, 'data_timeout': 120, 'name': u'scheduler-master', 'last_connection': 0, 'hard_ssl_name_check': False, 'uri': u'http://localhost:7768/', 'instance_id': 0, 'running_id': 1441863854.7471528, 'timeout': 3, 'address': u'localhost', 'active': True, 'use_ssl': False, 'push_flavor': 138339, 'port': 7768}}", "logclass" : 2, "options" : "", "state_type" : "", "lineno" : 802, "state" : 0, "host_name" : "", "time" : 1441863933, "service_description" : "", "logobject" : 0, "type" : "INFO", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f1190aa5d69827ccea968d" }, "comment" : "", "plugin_output" : "check_ping: Invalid hostname/address - pi1", "attempt" : 1, "message" : "[1441863942] HOST ALERT: pi1;DOWN;SOFT;1;check_ping: Invalid hostname/address - pi1", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 979, "state" : 1, "host_name" : "pi1", "time" : 1441863942, "service_description" : "", "logobject" : 1, "type" : "HOST ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f1190aa5d69827ccea968e" }, "comment" : "", "plugin_output" : "ERROR : this plugin needs the python-paramiko module. Please install it", "attempt" : 1, "message" : "[1441863944] SERVICE ALERT: pi2;Memory;CRITICAL;SOFT;1;ERROR : this plugin needs the python-paramiko module. Please install it", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 980, "state" : 2, "host_name" : "pi2", "time" : 1441863944, "service_description" : "Memory", "logobject" : 2, "type" : "SERVICE ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f11914a5d69827ccea968f" }, "comment" : "", "plugin_output" : "ERROR : this plugin needs the python-paramiko module. Please install it", "attempt" : 1, "message" : "[1441863946] SERVICE ALERT: pi2;Disks;CRITICAL;SOFT;1;ERROR : this plugin needs the python-paramiko module. Please install it", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 981, "state" : 2, "host_name" : "pi2", "time" : 1441863946, "service_description" : "Disks", "logobject" : 2, "type" : "SERVICE ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f11914a5d69827ccea9690" }, "comment" : "", "plugin_output" : "ERROR : this plugin needs the python-paramiko module. Please install it", "attempt" : 1, "message" : "[1441863947] SERVICE ALERT: pi2;Read-only Filesystems;CRITICAL;SOFT;1;ERROR : this plugin needs the python-paramiko module. Please install it", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 982, "state" : 2, "host_name" : "pi2", "time" : 1441863947, "service_description" : "Read-only Filesystems", "logobject" : 2, "type" : "SERVICE ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f11914a5d69827ccea9691" }, "comment" : "", "plugin_output" : "check_ping: Invalid hostname/address - pi2", "attempt" : 1, "message" : "[1441863947] HOST ALERT: pi2;DOWN;SOFT;1;check_ping: Invalid hostname/address - pi2", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 983, "state" : 1, "host_name" : "pi2", "time" : 1441863947, "service_description" : "", "logobject" : 1, "type" : "HOST ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f11914a5d69827ccea9692" }, "comment" : "", "plugin_output" : "check_ping: Invalid hostname/address - pi2", "attempt" : 1, "message" : "[1441863949] HOST ALERT: pi2;DOWN;SOFT;1;check_ping: Invalid hostname/address - pi2", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 984, "state" : 1, "host_name" : "pi2", "time" : 1441863949, "service_description" : "", "logobject" : 1, "type" : "HOST ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f11914a5d69827ccea9693" }, "comment" : "", "plugin_output" : "check_ping: Invalid hostname/address - pi2", "attempt" : 1, "message" : "[1441863951] HOST ALERT: pi2;DOWN;SOFT;1;check_ping: Invalid hostname/address - pi2", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 985, "state" : 1, "host_name" : "pi2", "time" : 1441863951, "service_description" : "", "logobject" : 1, "type" : "HOST ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f1191ea5d69827ccea9694" }, "comment" : "", "plugin_output" : "ERROR : this plugin needs the python-paramiko module. Please install it", "attempt" : 1, "message" : "[1441863965] SERVICE ALERT: pi2;Load Average;CRITICAL;SOFT;1;ERROR : this plugin needs the python-paramiko module. Please install it", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 986, "state" : 2, "host_name" : "pi2", "time" : 1441863965, "service_description" : "Load Average", "logobject" : 2, "type" : "SERVICE ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f11929a5d69827ccea9695" }, "comment" : "", "plugin_output" : "check_ping: Invalid hostname/address - pi2", "attempt" : 1, "message" : "[1441863967] HOST ALERT: pi2;DOWN;SOFT;1;check_ping: Invalid hostname/address - pi2", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 987, "state" : 1, "host_name" : "pi2", "time" : 1441863967, "service_description" : "", "logobject" : 1, "type" : "HOST ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f11929a5d69827ccea9696" }, "comment" : "", "plugin_output" : "ERROR : this plugin needs the python-paramiko module. Please install it", "attempt" : 1, "message" : "[1441863973] SERVICE ALERT: pi2;Disks Stats;CRITICAL;SOFT;1;ERROR : this plugin needs the python-paramiko module. Please install it", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 988, "state" : 2, "host_name" : "pi2", "time" : 1441863973, "service_description" : "Disks Stats", "logobject" : 2, "type" : "SERVICE ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f11929a5d69827ccea9697" }, "comment" : "", "plugin_output" : "check_ping: Invalid hostname/address - pi2", "attempt" : 1, "message" : "[1441863976] HOST ALERT: pi2;DOWN;SOFT;1;check_ping: Invalid hostname/address - pi2", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 989, "state" : 1, "host_name" : "pi2", "time" : 1441863976, "service_description" : "", "logobject" : 1, "type" : "HOST ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f1193ea5d69827ccea9698" }, "comment" : "", "plugin_output" : "ERROR : this plugin needs the python-paramiko module. Please install it", "attempt" : 1, "message" : "[1441863988] SERVICE ALERT: pi1;NFS Stats;CRITICAL;SOFT;1;ERROR : this plugin needs the python-paramiko module. Please install it", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 990, "state" : 2, "host_name" : "pi1", "time" : 1441863988, "service_description" : "NFS Stats", "logobject" : 2, "type" : "SERVICE ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f1193ea5d69827ccea9699" }, "comment" : "", "plugin_output" : "check_ping: Invalid hostname/address - pi2", "attempt" : 2, "message" : "[1441863990] HOST ALERT: pi2;DOWN;HARD;2;check_ping: Invalid hostname/address - pi2", "logclass" : 1, "options" : "", "state_type" : "HARD", "lineno" : 991, "state" : 1, "host_name" : "pi2", "time" : 1441863990, "service_description" : "", "logobject" : 1, "type" : "HOST ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f1193ea5d69827ccea969a" }, "comment" : "", "plugin_output" : "", "attempt" : 0, "message" : "[1441863990] HOST NOTIFICATION: admin;pi2;DOWN;notify-host-by-email;check_ping: Invalid hostname/address - pi2", "logclass" : 3, "options" : "", "state_type" : "DOWN", "lineno" : 992, "state" : 1, "host_name" : "pi2", "time" : 1441863990, "service_description" : "", "logobject" : 1, "type" : "HOST NOTIFICATION", "contact_name" : "admin", "command_name" : "notify-host-by-email" } ,
    { "_id" : { "$oid" : "55f1193ea5d69827ccea969b" }, "comment" : "", "plugin_output" : "check_ping: Invalid hostname/address - pi1", "attempt" : 1, "message" : "[1441863991] HOST ALERT: pi1;DOWN;SOFT;1;check_ping: Invalid hostname/address - pi1", "logclass" : 1, "options" : "", "state_type" : "SOFT", "lineno" : 993, "state" : 1, "host_name" : "pi1", "time" : 1441863991, "service_description" : "", "logobject" : 1, "type" : "HOST ALERT", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f1193ea5d69827ccea969c" }, "comment" : "", "plugin_output" : "", "attempt" : 0, "message" : "[1441863993] INFO: [Shinken] [reactionner-master] We already got the conf 0 (scheduler-master)", "logclass" : 2, "options" : "", "state_type" : "", "lineno" : 994, "state" : 0, "host_name" : "", "time" : 1441863993, "service_description" : "", "logobject" : 0, "type" : "INFO", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f1193ea5d69827ccea969d" }, "comment" : "", "plugin_output" : "", "attempt" : 0, "message" : "[1441863993] INFO: [Shinken] [reactionner-master] Using max workers: 15", "logclass" : 2, "options" : "", "state_type" : "", "lineno" : 995, "state" : 0, "host_name" : "", "time" : 1441863993, "service_description" : "", "logobject" : 0, "type" : "INFO", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f1193ea5d69827ccea969e" }, "comment" : "", "plugin_output" : "", "attempt" : 0, "message" : "[1441863993] INFO: [Shinken] [reactionner-master] Using min workers: 1", "logclass" : 2, "options" : "", "state_type" : "", "lineno" : 996, "state" : 0, "host_name" : "", "time" : 1441863993, "service_description" : "", "logobject" : 0, "type" : "INFO", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f1193ea5d69827ccea969f" }, "comment" : "", "plugin_output" : "", "attempt" : 0, "message" : "[1441863993] INFO: [Shinken] We have our schedulers: {0: {'wait_homerun': {}, 'data_timeout': 120, 'name': u'scheduler-master', 'hard_ssl_name_check': False, 'uri': u'http://localhost:7768/', 'actions': {}, 'instance_id': 0, 'running_id': 0, 'timeout': 3, 'address': u'localhost', 'active': True, 'use_ssl': False, 'push_flavor': 797380, 'port': 7768}}\n", "logclass" : 2, "options" : "", "state_type" : "", "lineno" : 997, "state" : 0, "host_name" : "", "time" : 1441863993, "service_description" : "", "logobject" : 0, "type" : "INFO", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f1193ea5d69827ccea96a0" }, "comment" : "", "plugin_output" : "", "attempt" : 0, "message" : "[1441863993] INFO: [Shinken] [reactionner-master] Init connection with scheduler-master at http://localhost:7768/ (3s,120s)", "logclass" : 2, "options" : "", "state_type" : "", "lineno" : 998, "state" : 0, "host_name" : "", "time" : 1441863993, "service_description" : "", "logobject" : 0, "type" : "INFO", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f1193ea5d69827ccea96a1" }, "comment" : "", "plugin_output" : "", "attempt" : 0, "message" : "[1441863993] INFO: [Shinken] [reactionner-master] Connection OK with scheduler scheduler-master", "logclass" : 2, "options" : "", "state_type" : "", "lineno" : 999, "state" : 0, "host_name" : "", "time" : 1441863993, "service_description" : "", "logobject" : 0, "type" : "INFO", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f1193ea5d69827ccea96a2" }, "comment" : "", "plugin_output" : "", "attempt" : 0, "message" : "[1441863993] INFO: [broker-master] We have our schedulers: {0: {'broks': {}, 'data_timeout': 120, 'name': u'scheduler-master', 'last_connection': 0, 'hard_ssl_name_check': False, 'uri': u'http://localhost:7768/', 'instance_id': 0, 'running_id': 1441863854.7471528, 'timeout': 3, 'address': u'localhost', 'active': True, 'use_ssl': False, 'push_flavor': 797380, 'port': 7768}}", "logclass" : 2, "options" : "", "state_type" : "", "lineno" : 1000, "state" : 0, "host_name" : "", "time" : 1441863993, "service_description" : "", "logobject" : 0, "type" : "INFO", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f1193ea5d69827ccea96a3" }, "comment" : "", "plugin_output" : "", "attempt" : 0, "message" : "[1441863993] INFO: [broker-master] We have our schedulers: {0: {'broks': {}, 'data_timeout': 120, 'name': u'scheduler-master', 'last_connection': 0, 'hard_ssl_name_check': False, 'uri': u'http://localhost:7768/', 'instance_id': 0, 'running_id': 1441863854.7471528, 'timeout': 3, 'address': u'localhost', 'active': True, 'use_ssl': False, 'push_flavor': 797380, 'port': 7768}}", "logclass" : 2, "options" : "", "state_type" : "", "lineno" : 1001, "state" : 0, "host_name" : "", "time" : 1441863993, "service_description" : "", "logobject" : 0, "type" : "INFO", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f1193ea5d69827ccea96a4" }, "comment" : "", "plugin_output" : "", "attempt" : 0, "message" : "[1441863993] INFO: [broker-master] We have our schedulers: {0: {'broks': {}, 'data_timeout': 120, 'name': u'scheduler-master', 'last_connection': 0, 'hard_ssl_name_check': False, 'uri': u'http://localhost:7768/', 'instance_id': 0, 'running_id': 1441863854.7471528, 'timeout': 3, 'address': u'localhost', 'active': True, 'use_ssl': False, 'push_flavor': 797380, 'port': 7768}}", "logclass" : 2, "options" : "", "state_type" : "", "lineno" : 1002, "state" : 0, "host_name" : "", "time" : 1441863993, "service_description" : "", "logobject" : 0, "type" : "INFO", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f1193ea5d69827ccea96a5" }, "comment" : "", "plugin_output" : "", "attempt" : 0, "message" : "[1441863993] INFO: [broker-master] We have our schedulers: {0: {'broks': {}, 'data_timeout': 120, 'name': u'scheduler-master', 'last_connection': 0, 'hard_ssl_name_check': False, 'uri': u'http://localhost:7768/', 'instance_id': 0, 'running_id': 1441863854.7471528, 'timeout': 3, 'address': u'localhost', 'active': True, 'use_ssl': False, 'push_flavor': 797380, 'port': 7768}}", "logclass" : 2, "options" : "", "state_type" : "", "lineno" : 1003, "state" : 0, "host_name" : "", "time" : 1441863993, "service_description" : "", "logobject" : 0, "type" : "INFO", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f1193ea5d69827ccea96a6" }, "comment" : "", "plugin_output" : "", "attempt" : 0, "message" : "[1441863993] INFO: [broker-master] We have our schedulers: {0: {'broks': {}, 'data_timeout': 120, 'name': u'scheduler-master', 'last_connection': 0, 'hard_ssl_name_check': False, 'uri': u'http://localhost:7768/', 'instance_id': 0, 'running_id': 1441863854.7471528, 'timeout': 3, 'address': u'localhost', 'active': True, 'use_ssl': False, 'push_flavor': 797380, 'port': 7768}}", "logclass" : 2, "options" : "", "state_type" : "", "lineno" : 1004, "state" : 0, "host_name" : "", "time" : 1441863993, "service_description" : "", "logobject" : 0, "type" : "INFO", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f1193ea5d69827ccea96a7" }, "comment" : "", "plugin_output" : "", "attempt" : 0, "message" : "[1441863993] INFO: [broker-master] We have our schedulers: {0: {'broks': {}, 'data_timeout': 120, 'name': u'scheduler-master', 'last_connection': 0, 'hard_ssl_name_check': False, 'uri': u'http://localhost:7768/', 'instance_id': 0, 'running_id': 1441863854.7471528, 'timeout': 3, 'address': u'localhost', 'active': True, 'use_ssl': False, 'push_flavor': 797380, 'port': 7768}}", "logclass" : 2, "options" : "", "state_type" : "", "lineno" : 1005, "state" : 0, "host_name" : "", "time" : 1441863993, "service_description" : "", "logobject" : 0, "type" : "INFO", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f1193ea5d69827ccea96a8" }, "comment" : "", "plugin_output" : "", "attempt" : 0, "message" : "[1441863993] INFO: [broker-master] We have our arbiters: {0: {'broks': {}, 'last_connection': 0, 'name': u'arbiter-master', 'hard_ssl_name_check': False, 'uri': u'http://localhost:7770/', 'instance_id': 0, 'running_id': 0, 'address': u'localhost', 'use_ssl': False, 'port': 7770}}", "logclass" : 2, "options" : "", "state_type" : "", "lineno" : 1006, "state" : 0, "host_name" : "", "time" : 1441863993, "service_description" : "", "logobject" : 0, "type" : "INFO", "contact_name" : "", "command_name" : "" } ,
    { "_id" : { "$oid" : "55f1193ea5d69827ccea96a9" }, "comment" : "", "plugin_output" : "", "attempt" : 0, "message" : "[1441863993] INFO: [broker-master] We have our arbiters: {0: {'broks': {}, 'last_connection': 0, 'name': u'arbiter-master', 'hard_ssl_name_check': False, 'uri': u'http://localhost:7770/', 'instance_id': 0, 'running_id': 0, 'address': u'localhost', 'use_ssl': False, 'port': 7770}}", "logclass" : 2, "options" : "", "state_type" : "", "lineno" : 1007, "state" : 0, "host_name" : "", "time" : 1441863993, "service_description" : "", "logobject" : 0, "type" : "INFO", "contact_name" : "", "command_name" : "" } ,

Availability collection

Hosts/services daily availability are stored in a collection which default name is availability

Each document in the collection contain always the same fields:

  • _id: added by mongodb when document is inserted

  • first_check_state: state of first received check for the day

  • first_check_timestamp: timestamp of first received check

  • last_check_state: state of last received check for the day

  • last_check_timestamp: timestamp of last received check

  • day_ts: timestamp for 00:00 of the day

  • day: day in YYYY-MM-DD format

  • hostname: hostname

  • service: service description (empty for host check)

  • daily_0: number of seconds in state 0 (UP/OK)

  • daily_1: number of seconds in state 1 (DOWN/WARNING)

  • daily_2: number of seconds in state 2 (UNREACHABLE/CRITICAL)

  • daily_3: number of seconds in state 3 (UNKNOWN)

  • daily_4: number of seconds in state 4 (OTHER) - unchecked period ...

  • is_downtime: currently in downtime

Example:

    { "_id" : { "$oid" : "55f118eac4e7774e6d845809" }, "first_check_state" : 1, "day_ts" : 1441836000, "service" : "", "first_check_timestamp" : 1441863885, "daily_4" : 59028, "hostname" : "pi2", "daily_1" : 27372, "daily_0" : 0, "daily_3" : 0, "daily_2" : 0, "is_downtime" : "0", "last_check_timestamp" : 1441891257, "day" : "2015-09-10", "last_check_state" : 1 } ,
    { "_id" : { "$oid" : "55f118eac4e7774e6d84580a" }, "first_check_state" : 1, "day_ts" : 1441836000, "service" : "", "first_check_timestamp" : 1441863894, "daily_4" : 59044, "hostname" : "pi1", "daily_1" : 27356, "daily_0" : 0, "daily_3" : 0, "daily_2" : 0, "is_downtime" : "0", "last_check_timestamp" : 1441891250, "day" : "2015-09-10", "last_check_state" : 1 } ,
    { "_id" : { "$oid" : "55f118eac4e7774e6d84580b" }, "first_check_state" : 1, "day_ts" : 1441836000, "service" : "", "first_check_timestamp" : 1441863910, "daily_4" : 59728, "hostname" : "graphite", "daily_1" : 26672, "daily_0" : 0, "daily_3" : 0, "daily_2" : 0, "is_downtime" : "0", "last_check_timestamp" : 1441890582, "day" : "2015-09-10", "last_check_state" : 1 } ,
    { "_id" : { "$oid" : "55f118ebc4e7774e6d84580c" }, "first_check_state" : 0, "day_ts" : 1441836000, "service" : "", "first_check_timestamp" : 1441863912, "daily_4" : 59273, "hostname" : "webui", "daily_1" : 0, "daily_0" : 27127, "daily_3" : 0, "daily_2" : 0, "is_downtime" : "0", "last_check_timestamp" : 1441891039, "day" : "2015-09-10", "last_check_state" : 0 } ,
    { "_id" : { "$oid" : "55f1289a72777c74656f0d56" }, "first_check_state" : 0, "day_ts" : 1441836000, "service" : "", "first_check_timestamp" : 1441867909, "daily_4" : 83545, "hostname" : "localhost", "daily_1" : 0, "daily_0" : 2855, "daily_3" : 0, "daily_2" : 0, "is_downtime" : "0", "last_check_timestamp" : 1441870764, "day" : "2015-09-10", "last_check_state" : 0 }