Vamp Gateway Agent

HAProxy with configuration from ZooKeeper, etcd or Consul.

HAProxy is a tcp/http load balancer, the purpose of this agent is to:

read the HAProxy configuration from ZooKeeper, etcd or Consul and reloads the HAProxy on each configuration change with as close to zero client request interruption as possible.
read the logs from HAProxy over socket and push them to Logstash over UDP.
handle and recover from ZooKeeper, etcd, Consul and Logstash outages without interrupting the haproxy process and client requests.

It is possible to specify a custom configuration (based on arguments configurationPath/configurationBasicFile). In that case any configuration read from KV store is appended to the content of custom configuration, stored as configurationPath/haproxy.cnf and used for the next HAProxy reload.

Usage

$ ./vamp-gateway-agent: -h
                                       
Usage of ./vamp-gateway-agent:
  -configurationBasicFile string
        Basic HAProxy configuration. (default "haproxy.basic.cfg")
  -configurationPath string
        HAProxy configuration path. (default "/opt/vamp/")
  -debug
        Switches on extra log statements.
  -help
        Print usage.
  -logo
        Show logo. (default true)
  -logstashHost string
        Address of the Logstash instance (default "127.0.0.1")
  -logstashPort int
        The UDP input port of the Logstash instance (default 10001)
  -retryTimeout int
        Default retry timeout in seconds. (default 5)
  -scriptPath
        HAProxy validation and reload script path. (default "/opt/vamp/")
  -storeConnection string
        Key-value store connection string.
  -storeKey string
        HAProxy configuration store key. (default "/vamp/gateways/haproxy/1.6")
  -storeType string
        zookeeper, consul or etcd.

Logstash example configuration:

input {
  udp {
    port => 10001
    type => haproxy
  }
}

filter {
  grok {
    match => ["message", "^.+?]: (?<metrics>{.*})$"]
    match => ["message", "^.*$"]
  }
  if [metrics] =~ /.+/ {
    json {
      source => "metrics"
    }
    if [t] =~ /.+/ {
      date {
        match => ["t", "dd/MMM/YYYY:HH:mm:ss.SSS"]
      }
    }
  }
}

output {
  elasticsearch {
    hosts => "elasticsearch:9200"
  }
  stdout {
    codec => rubydebug
  }
}

Note: Logstash configuration depends on HAProxy log configuration and that is not in the scope of the agent (HAProxy configuration is retrieved from ZooKeeper, etcd or Consul).

Building Binary

Using the build.sh script:

  ./build.sh --make

Alternatively:

go get github.com/tools/godep
godep restore
go install
CGO_ENABLED=0 go build -v -a -installsuffix cgo

Building Docker Images

Directory haproxy contains Dockerfiles for HAProxy 1.5.16 and 1.6.4 Alpine based images.

$ ./build.sh -h

Usage of ./build.sh:

  -h|--help   Help.
  -l|--list   List built Docker images.
  -r|--remove Remove Docker image.
  -m|--make   Build the binary and copy it to the Docker directories.
  -b|--build  Build Docker image.
  -a|--all    Build all binaries, by default only linux:amd64.

Docker images after the build (e.g. ./build.sh -b):

magneticio/vamp-gateway-agent_1.6.4:0.8.5
magneticio/vamp-gateway-agent_1.5.16:0.8.5

Travis CI Build

Build is performed on each push to master branch and all directories from target/docker are pushed to specific version branch (e.g. 0.8.5). After that Docker Hub Automated Build is triggered.

Docker Images

Docker Hub Repo

1.6.4-0.8.5

1.5.16-0.8.5

e.g.