romsnl/logstash

Configurations of my logstash: logstash, filebeat, grok patterns: sshd, postfix, apache, sysdig, zimbra mailbox.log, zimbra zimbra.log, Datadog Dogstatsd, fail2ban

logstash

my logsash config

ELK (Elasticsearch + Logstash + Kibana) is fun!

Logstash is super flexible, most operations can be.

memo

Start separately Java process, shipper indexer. (divided into two by copying the startup script that is distributed in the package version)

reference

• postfix grok patterns :

• https://github.com/whyscream/postfix-grok-patterns

• https://gist.github.com/poolski/9911628

• https://gist.github.com/jamtur01/4385667

• https://gist.github.com/randywallace/6983588

• sshd grok patterns :

• https://github.com/autosportlabs/docker-logstash/blob/master/src/conf/520-mogrify-sshd.conf

• Lightweight log shipper : logstash-forwarder (aka lumberjack)

• https://github.com/elasticsearch/logstash-forwarder

• https://www.digitalocean.com/community/tutorial_series/centralized-logging-with-logstash-and-kibana-on-ubuntu-14-04

• https://www.digitalocean.com/community/tutorials/adding-logstash-filters-to-improve-centralized-logging

• grok filter ruby :

• https://groups.google.com/forum/#!topic/logstash-users/iEYRv7bCqdM

• http://stackoverflow.com/questions/20512416/adding-tags-to-logstash-events-based-on-the-md5-of-the-filename

• kibana geoip BetterMap :

• https://beingasysadmin.wordpress.com/2014/04/07/near-realtime-dashboard-with-kibana-and-elasticsearch/

• http://dev.maxmind.com/geoip/legacy/geolite/

• grok apache User-Agent :

• http://untergeek.com/2013/09/11/getting-apache-to-output-json-for-logstash-1-2-x/

• https://github.com/ua-parser/uap-core/blob/master/regexes.yaml

• Integrating DataDog

• http://ifdattic.com/integrating-datadog-and-logstash-on-aws-ec2/

• zimbra mailbox.log & zimbra.log (amavis)

• http://blog.itlinux.cl/blog/2015/05/25/buscando-mensajes-de-correo-con-kibana/

• https://github.com/ITLinuxCL/zimbra_logstash

• http://antisp.in/2014/04/01/useful-logstash-grok-patterns/

• https://github.com/Autobase/Zimbra/blob/4bf3dc250c68a38e38286bdd972c8d5469d40e34/ZimbraCommon/src/java/com/zimbra/common/util/ZimbraLog.java

• https://wiki.zimbra.com/wiki/Centralized_Logs_-_Elasticsearch,_Logstash_and_Kibana

• https://blog.zimbra.com/2007/05/mailboxlog-the-king-of-zimbra-log-files/

• https://www.zimbra.com/docs/os/5.0.19/administration_guide/9_Monitoring.11.1.html