/hercules

Vostok Hercules is an open-source distributed system based on Apache Kafka and used for reliable delivery of telemetry data from microservices to storage and analytics engines.

Primary LanguageJavaMIT LicenseMIT

Build Status

Vostok Hercules

Vostok Hercules is an open-source distributed system based on Apache Kafka and used for reliable delivery of telemetry data from microservices to storage and analytics engines.

Vostok Hercules is developed by Kontur as a part of Vostok project — an ecosystem for easy and fast development of microservices primarily using .NET Core / C#.

Vostok Hercules provides abstractions for application code to encapsulate specific data format in telemetry engines depending on a telemetry type. Vostok Hercules provides out of the box 4 telemetry types:

  • logs,
  • metrics,
  • distributed traces,
  • annotations.

Telemetry types

Telemetry is an event which has a timestamp and tags. Set of tags depends on a telemetry type.

Logs

Vostok Hercules delivers logs to ElasticSearch. Also, logs with level ERROR or FATAL are delivered to Sentry. See supported log event schema for details.

Metrics

Vostok Hercules delivers metrics to Graphite, Moira or any other tool which supports Graphite format. Metrics can be in the plaintext format, when a metric name consists of several components joined by a dot or in the modern tagged format. See supported metric event schema for details.

Distributed traces

Vostok Hercules stores distributed traces into ClickHouse. To analyze traces use Contrails web interface. Contrails is a tool for tree span analysis. See supported trace span schema for details.

Annotations

Annotation marks point on graphs in Grafana and Kibana. Vostok Hercules delivers annotations to ElasticSearch. See supported annotation event schema for details.

Other telemetry

Vostok Hercules supports an extendable processing of telemetry events and custom telemetry types as well. Hercules can be extended by one of the following ways:

  • Use one of provided Sinks (e.g. see Elastic Sink),
  • Write an integration via universal HTTP Stream API or Timeline API in any language you prefer,
  • Write a new Sink in Java.

System Design

System Design

Minimal data unit of Vostok Hercules is an Event. Events do not have predefined schema except well-known tags in some cases. To support schemaless processing Vostok Hercules stores and transfers events in the binary format with a self-described schema (see Hercules Protocol).

A Stream is a collection of events of the same type. Each stream has a backing topic in Apache Kafka.

An application uses hercules client to write events to Gate into streams. Other applications can read these events from Stream API or Timeline API using the same client.

Various sink daemons can deliver events to various backends at the same time. List of supported sink daemons:

Also, there are several auxiliary sink daemons:

To analyze distributed traces use Contrails. Contrails read tracing spans from Tracing API.

Management

There are several applications to manage streams and timelines:

Other components

Vostok Hercules uses internally this list of tools:

  • Apache Kafka for storing streams,
  • Apache ZooKeeper for meta-data (configuration for streams, timelines and others),
  • Apache Cassandra for storing timelines,
  • ClickHouse for storing tracing spans.

How to Use

Spaceport is an all-in-one development infrastructure.

Also, each component of Vostok Hercules can be run separately using Docker or as simple java application.

The official Docker images are on Docker Hub.

How to Build

Build jars

You need to have Java 8 or higher and Maven 3.3.9 or higher.

Build project with tests:

mvn package

Build project without tests:

mvn -DskipTests=true package

Build Docker images

You need make and Docker 19.03 or higher.

Build Docker images:

make build_images