/f5elastic

A fast syslog server that receives high volumes of traffic logs from F5 Load Balancers and indexes them in Elasticsearch

Primary LanguageGoMIT LicenseMIT

f5elastic

A fast system tool for receiving high volumes of traffic logs sent from F5 LTM load balancers and indexes these requests inside ES.

It is today battle tested in a production environment handling loads of 500k req/min on a single host.

Features

  • Works with Elasticsearch 7.x
  • Support for clusters of ES nodes, for distribution of indexing load.
  • Health checks for each ES node is enabled by default.
  • Uses bulk indexing for better performance with large amount of requests.
  • GeoIP lookup of each request, adds long/lat, city, and country for each client.
  • Allows to obfuscate client ip if needed.
  • ...

Getting started

  1. Create a Request Logging profile on your BigIP, and add the following Response Settings template (set HSL Protocol to TCP):
<34> $CLIENT_IP || $HTTP_METHOD || ${Host} || $HTTP_URI || $HTTP_STATCODE || $RESPONSE_SIZE || ${Referer} || ${User-Agent} || $SERVER_IP:$SERVER_PORT || $VIRTUAL_POOL_NAME || $VIRTUAL_NAME

  1. Download compiled binary from https://github.com/martensson/f5elastic/releases or go get github.com/martensson/f5elastic

  2. Download the latest Maxmind GeoLite2-City or GeoIP2-City db.

  3. Edit f5elastic.toml (check example in repo)

  4. Apply the ES template inside the repo. (optional but recommended). Create a corresponding ILM policy named f5elastic with the retention rules of your liking.

  5. Run f5elastic -f /path/to/f5elastic.toml

  6. Take a cup of coffee and make some nice dashboards inside Kibana :)