grunf 0.4.02 (Beta)

(simple clojure-based http monitoring tool)

Install

Clone the project to your target machine git clone https://github.com/SupplyFrame/grunf
Install leiningen if you don't have it on your target machine. you can either
1. Use the package manager to install it, ex sudo apt-get install leiningen
2. Download the script and put it in your path (ie. /usr/local/bin), then chmod 755 ~/usr/local/bin/lein.
In the grunf source code directory, run lein deps to install all the dependency libraries.
Checkout graphite setup guide on our wiki!

Usage

read config file and log in console.

lein run --log-level info --config conf.example.clj

Send events to graphite

lein run --config conf.example.clj --graphite-host localhost

mkdir logs
lein trampoline run --log logs/foo.log -c conf.example.clj &
tail -f logs/foo.log

Read smtp config, will send mails when receive error

lein run -c conf.example.clj -s smtp.example.clj

export csv in logs/bar.csv

lein run -c conf.example.clj --csv logs/bar.csv

Forward events to Riemann

lein run -c conf.example.clj --riemann-host 127.0.0.1 --riemann-port 5555

Run a script when receive error. The arguments passed to the script are url, status code ("null" when there's no status code), and exception message.

lein run -c conf.example.clj --script ./my-script.sh

Command line options

The command line options for grunf are:

Usage:

Switches                 Default    Desc
--------                 -------    ----
-c, --config                        Path to the config file
--log                               log path for log4j. If not specified, log to console 
--log-level              debug      log level for log4j, (fatal|error|warn|info|debug)
--no-rolling, --rolling  false      Use fixed size rolling file
--graphite-host                     Graphite server host
--graphite-port          2003       Graphite server port
--graphite-prefix                   prefix namespace for graphite
--riemann-host                      Riemann host
--riemann-port           5555
-s, --script                        a script to execute when receive error. The args are: url, status code, and java exception message
--hostname               127.0.0.1  This server's hostname
--csv                               csv log path
--interval               60000      Default interval for each url request
--user-agent             Grunf      Default user agent string
--timeout                6000       Default timeout per request
-s, --smtp-config                   Path to smtp config file 
-h, --no-help, --help    false      Print this message

For graphite and riemann, must specify graphite-host and riemann-host to trigger the feature.

Configuration files format

URL configurations

The configuration file format for conf.example.clj is

[{:url "http://www.yahoo.com/"
  :interval 1000                      ;; optional
  :validator #(re-find #"yahoo" %)    ;; optional
  :http-options {:timeout 2000        ;; :http-options itself is also optional 
                 :user-agent "Mozilla"}
  :graphite-ns "com.yahoo.www"        ;; defualt to reverse domain name
  :params-fn (map #(hash-map :id %) (iterate inc 0)) ;; programatically control query params
  :riemann-tags ["Yahoo" "Homepage"]
  }
  {:url "http://www.google.com/?search=abc"
   :name "search"                     ;; append to graphite namespace
                                      ;; in this case it would be
                                      ;; "com.google.www.search"
  }      ;; only url is required
]

Experimental features

:params-fn is a function that generate a lazy sequence, and every element in that lazy sequence will be transformed into query params in each query. For example:

  :params-fn (map #(hash-map :id %) (iterate inc 0)) ;; programatically control query params

will transformed into these urls:

http://www.google.com/?id=1
http://www.google.com/?id=2
... # and so on

And (cycle [{:search "abc"} {:search "def"}]) will transformed into

http://www.google.com/?search=abc
http://www.google.com/?search=def
http://www.google.com/?search=abc
http://www.google.com/?search=def
... # as you expected, a cycle

SMTP configuration

The smtp configuration file for smtp.example.clj is

^{:host "smtp.gmail.com"
  :user "example@gmail.com"
  :pass "password"
  :port 1234
  :tls true
  }
{:from "sysalerts@example.com"
 :to ["user1@gmail.com" "user2@gmail.com"]
 :subject "Will be overwirtten by grunf!"
 :body "Will be overwritten by grunf!"
 }

For full documentaion of SMTP setup, please visit postal -- internet email library for clojure.

Output format

CSV

The CSV fields are

HH:MM:ss,SSS,*log type*,*http status*,*url*,*response time (milliseconds)*

Note

This tool is still in experimental status, but all the example configs should work just fine.

Test covered

SMTP send mail error and timeout error
log4j
csv
utility functions

News and changes

v0.4.02
- Export http headers to riemann
v0.3.12
- Added a option that can execute a script when receive an error.
v0.4.01
- Added Rolling file option.
v0.3.11
- Added UnknownHostException and ConnectException to Riemann adapter.
v0.3.10
- Use thread executor to send message to riemann asynchronously.
v0.3.9
- Custom thread executor to get more robust control over graphite connection
v0.3.8
- Fix graphite exception (should use a thunk)
v0.3.7
- Fix log4j issues
v0.3.6
- Added Riemann support
v0.3.5
- Fixed memory issue by using constant threads
v0.3.4
- IO exception handling improved in graphite adapter.
v0.3.3
- Experimental feature :params-fn
v0.3.2
- Wrap (eval validator) to let block, hopefully can solve memory out of space error
v0.3.1
- Fix graphite CLI option bug
v0.3
- Automatic graphite namespace generator now covers the full uri path
- Can use ``--graphite-prefix` to setup prefix namespace for graphite
- Graphite namespace is now under test coverages
- Refactored bin.clj, many functions go to utils.clj
v0.2.12
- Tests for most adapters (log4j, csv, smtp)
v0.2.10
- Socket exception handling for graphite
v0.2.9
- :name option in config file
v0.2.8
- command line options for global request interval, user-agent, and request timeout

TODOs

graphite adapter test
Tutorial of writing verification in configuration file.

License

Distributed under the Eclipse Public License, the same as Clojure.

Supplyframe/grunf