leia

What?

A service backed by Ktor, writing incoming HTTP requests to one or serveral topics of one or more message brokers (Kafka, Redis or Cloud Pub/Sub).

Why?

Sometimes a call to a webhook is only received once. Writing a HTTP request to a semi-permanent message broker log allows for repeated processing.

How?

Defining a route with the same syntax used for Ktor routing

Example

title = "My routes config"

[[routes]]
        path = "/status/mail"
        topic = "mail"
        verify = false
        format = "proto"
        methods = [ "POST", "PUT", "HEAD", "GET" ]

[[routes]]
        path = "/api/{id}"
        topic = "api_calls"
        format = "raw_body"
        verify = true
        validateJson = true
        jsonSchema = """
<put your JSON schema here>
"""

Routing is saved in files with .toml extension, and their location is set with the environment varaiable CONFIG_DIRECTORY. If this is not set it will default to /etc/config/.

Configuration

Configuration consists of few tables:

routes
sink providers
auth providers

Configuration is loaded on startup and then refreshed every 1 minute.

It can be retrieved either from a file or from kubernetes custom resources.

Configuration in files

Configuration in a file is stored in TOML file format. See example configuration here. You can have several configuration files stored in /etc/config/ or in path defined in CONFIG_DIRECTORY environment variable.

Configuration in kubernetes

Configuration in kubernetes requires creating custom resource defitnions first:

kubectl create -f leiaroute_crd.yaml
kubectl create -f leiasinkprov_crd.yaml

Then add your objects:

kubectl create -f object1.yaml
...

Custom resource definitions and sample object definitions YAML files can be found here.

Routes

Route defines where given request should be sent to.

path - is mandatory, HTTP URL path to match
topic - is mandatory, topic where the message will be sent
sink - is optional, name of sink where message will be sent, if not provided uses default sink.
verify - is optional, default is false. Decides whether only request with a verified json web token is allowed. Requires ktor-jwt.
format - is optional, default is proto a format using protobuf defined in zensum/webhook-proto. Also available is raw_body - writes HTTP body as is.
methods - is optional, default is all seven HTTP verbs.
response - is optional, HTTP response code which should be sent to the sender on success, default is 204.
validateJson - is optional, wether to validate body of the request as JSON before sending, default is false. 204 code is returned on success, 400 code on error.
jsonSchema - is optional, validates request body against this JSON schema if validateJson is set to true
cors - is optional, list of hosts to check incoming request's origin against for CORS. By default all traffic is allowed. If contains * then all hosts are allowed. When non empty, OPTIONS method is allowed implicitly.
hosts - is optional, list of hosts to check incoming request's host field against.
authenticateUsing - is optional, list of auth providers to verify request against

Sink providers

Leia needs at least one sink configured and marked as default.

name - is mandatory, name to identify the sink in routes configuration sink field
type - is optional, one of these types: kafka, redis, gpubsub, null, default is kafka
isDefault - is optional, must be set to true for one sink, default is false
options - is optional, additional options to configure sink provider, it is a list of key/value pairs

For testing purposes you can use null sink type which does not forward messages and only logs them.

Options

Kafka

To set kafka hostname and port use following option in configuration:

host = "<hostname>:<port>"

Redis

To set up hostname and port for redis Pub/Sub use following options in configuration:

host = "<hostname>"
port = "<port>"

Cloud Pub/Sub

To use other project than the default for Cloud Pub/Sub one use following option in configuration:

projectId = "<your-project-id>"

Before writing to sinks in Cloud Pub/Sub you need to create them first:

gcloud pubsub topics list-subscriptions --project <your-project-id> <topic>

GOOGLE_APPLICATION_CREDENTIALS environment variable needs to point to your cloud service account key in json file. Create service account key.

Auth providers

This table is optional in configuration.

name - is mandatory, name to identify the auth provider in routes configuration authenticateUsing field
type - is optional, one of these types: jwk, basic_auth, no_auth, default is no_auth
options - is optional, additional options to configure auth provider, it is a list of key/value pairs

Basic_auth

Option basic_auth_users is a map of users and passwords.

Jwk

Option jwk_configis mandatory, it is a map of key/values. The map must contain jwk_url and jwk_issuer keys.

Environment variables

CONFIG_DIRECTORY - location of toml files (default /etc/config/)
PORT - port on which leia listens for requests (default 80)
KUBERNETES_SERVICE_HOST - hostname of kubernetes API (default localhost)
KUBERNETES_SERVICE_PORT - port of kubernetes API (default 8080)
KUBERNETES_ENABLE - whether to read configuration from Kubernetes custom resources (default true). To disable set it to false.
PROMETHEUS_ENABLE - if set to true enables /metrics endpoint on port 9090 exposing metrics for prometheus (uses zensum/ktor-prometheus-feature), default false
LOG_REMOTE_HOST - if set to true enables logging of remote host (client address or host name), default false. Note the value may come from headers and could be falsified.

Health check

To check status of leia make request to http://leiahost/leia/health. Sample output:

sink redis1: ERROR
sink kafka1: OK
leia: ERROR

Error messages will be written to logs. You can also get them using verbose parameter in the request. Here is sample output for http://leiahost/leia/health?verbose:

sink redis1: ERROR redis.clients.jedis.exceptions.JedisConnectionException: Failed connecting to host redis:6379
sink kafka1: OK
leia: ERROR

zensum/leia