watcher
is a simple monitoring tool that checks the health of a number of
remote servers. It does this by regularly executing a set of pingers, each
performing a check against a remote server using a certain ping protocol.
Currently, pingers based on SSH and HTTP(S) are supported.
Should a ping check fail to produce an expected result, watcher
will send
out an alert email to any configured recipients. Should the remote endpoint
yet again become availble, a new alert will be sent to notify that the endpoint
has recovered. Should the ping check continue fail for a sufficiently long
time, reminder alerts will be sent (after a configurable time interval).
Besides sending out email alerts, the watcher
also publishes a simple REST
API, through which the status of its configured endpoints can be viewed.
The watcher
binary can be built as follows:
# get dependencies
go get -u
# compile
go build
watcher
expects a JSON-formatted configuration file, which describes the
ping checks to perform, at what interval, and how to alert (if at all) when
a monitored server's state changes.
The configuration file has the following structure:
{
"defaultSchedule": {
"interval": "1m",
"retries": {
"attempts": 3,
"delay": "10s",
"exponentialBackoff": false
}
},
"pingers": [
{
"name": "pinger-name",
"description": "pinger description",
"type": "<protocol>",
"check": {
<protocol-specific>
},
"schedule": {
"interval": "90s",
"retries": { "attempts": 3, "delay": "5s", "exponentialBackoff": false}
}
},
...
],
"alerter": {
"reminderDelay": "12h",
"email": {
"smtpHost": "some.smtp.host",
"smtpPort": 587,
"auth": { "username": "foo", "password": "bar" },
"from": "noreply@watcher.org",
"to": ["admin@company.com"]
}
}
}
defaultSchedule
(optional): specifies a default schedule to use for running pingers and also the default number of attempts to make on each ping. The default schedule can be overridden for each pinger, since a separate schedule can be specified per-pinger. Default isinterval
:10m
,attempts
:3
,delay
:3s
,exponentialBackoff
:false
.interval
: The default duration between two successive ping checks. This is specified as a golang duration. For example,1h
(1 hour),5m30s
(5 minutes and 30 seconds).retries
: The default retry behavior of pingers.attempts
: The number of attempts to try before deeming a ping check a failure.delay
: Delay between each retry. Also given as a golang duration. For example,2m
(2 minutes).exponentialBackoff
: Iftrue
, double the delay for each new retry attempt.
pingers
: The set of pingers to run.- Each pinger is an object with the following fields:
name
: The name of the pinger. Can only contain alphanumeric characters and-
,.
, and_
.description
(optional): A short description of the pinger and its purpose.type
: The type (protocol) of the pinger. One ofssh
andhttp
.check
: Protocol-specific details on how to perform each "ping". See below.schedule
(optional): The schedule to use for this pinger. If no schedule is given, thedefaultSchedule
is used.
- Each pinger is an object with the following fields:
alerter
(optional): How to notify interested parties of state changes in monitored endpoints.reminderDelay
: The duration to wait before sending out a reminder alert for an endpoint that keeps failing to respond properly on ping attempts. This is specified as a golang duration. For example,1h
(1 hour),90m
(90 minutes).email
: Configures an email alerter that will send alerts to a set of email recipients.smtpHost
: The SMTP server to send mails through.smtpPort
: The port of the SMTP server (typically587
,25
or465
)auth
(optional): Authentication credentials (ausername
andpassword
).from
: TheFrom:
address to set on sent alerts.to
: a list of email addresses to send alerts to (To:
).
A http
pinger, which tries to contact a URL (via a GET
request) and
expects a certain HTTP status code in the response, is configured as follows:
{
"name": "<name>",
"type": "http",
"check": {
"url": "https://...",
"verifyCert": false,
"basicAuth": {"username": "foo", "password": "bar"},
"expect": {
"statusCode": 200
},
"timeout": "10s"
}
}
The check
is the only part specific to the http
pinger. Its fields
carry the following semantics:
url
: The URL to try and contact.verifyCert
: Iftrue
, the server's certificate will be verified. Iffalse
no such verification is made (similar tocurl
's--insecure
flag).basicAuth
(optional): Specifies username and password to use.expect
: The expected response for the pinger to deem a ping attempt a success.statusCode
: The HTTP status code that the endpoint needs to respond with.
timeout
(optional): The connection timeout to use. Default:30s
.
A ssh
pinger, which executes a shell command/script against a remote server
and expects a certain exit code, is configured as shown below:
{
"name": "<name>",
"type": "ssh",
"check": {
"host": "some.host",
"port": 22,
"auth": { "username": "foo", "agent": true },
"command": "service docker status | grep running",
"timeout": "10s",
"expect": {
"exitCode": 0
}
}
}
The check
is the only part specific to the ssh
pinger. Its fields
carry the following semantics:
host
: The remote SSH server to monitor.port
: The port number of the SSH server (typically22
).auth
:username
: The username to log in with.
An authentication mechanism also needs to be specified. Must be at least one of the following:agent
:true
if the agent connection authentication method is to be used.password
: Specifies a password to use.key
: Specifies the path to a private key to use with the public key authentication method.
- The
check
must also specify a shell command/script to execute. It is either given directly as acommand
or as a file path viacommandFile
. timeout
(optional): The connection timeout to use. Default:30s
.expect
: The expected response for the pinger to deem a ping attempt a success.exitCode
: The exit code that the script must produce for the ping to be successful.
Sample configurations are given under etc/
.
To run watcher
according to a given configuration just run:
./watcher --certfile cert.pem --keyfile key.pem config.json
This will start a watcher publishing a HTTPS server on port 8443
(see below
for a description of its REST API). A self-signed certificate and key can be
generated using the etc/cert/generate-server-cert.sh
script.
For a complete list of command-line options, run ./watcher --help
.
watcher
publishes the following endpoints:
$ curl --insecure https://localhost:8443/pingers/
[
"https://localhost:8443/pingers/google.com",
"https://localhost:8443/pingers/jenkins",
]
$ curl --insecure https://localhost:8443/pingers/google.com
{
"LatestResult": {
"Status": 2,
"Error": {}
},
"Consecutive": 2,
"LatestOK": null,
"LatestNOK": "2016-05-26T09:38:57.686217751Z"
}
Status 0
means Unknown
, 1
means OK
, and 2 means NOK
.
$ curl --insecure https://localhost:8443/pingers/google.com/output
...