/NetMeta

NetMeta is a scalable network observability toolkit optimized for performance.

Primary LanguageCUEApache License 2.0Apache-2.0

NetMeta

github issues github forks github stars github license

⚠️ PRE-RELEASE: This is a work in progress - please watch this repo for news.

NetMeta is a scalable network observability toolkit optimized for performance.

Flows are not pre-aggregated and stored with one second resolution. This allows for queries in arbitrary dimensions with high-fidelity graphs.

⚡️ Features

NetMeta captures, aggregates and analyzes events from a variety of data sources:

  • sFlow
  • NetFlow/IPFIX
  • Port Mirror (AF_PACKET)
  • Linux NFLOG (soon)
  • Linux conntrack (soon)
  • Scamper traces (soon)
  • GCP VPC Flow Logs (soon)
  • AWS VPC Flow Logs (soon)

The different pros/cons between these and a more informations about the ingest of events can be found here

NetMeta also allows to integrate FastNetMon Attack notifications on the Dashboards. The docs to setup this and other Grafana related settings can be found here

Sampling rate is detected automatically. Different devices with different sampling rates can be mixed. IPv6 is fully supported throughout the stack.

👀 Demo

💥 API Stability

NetMeta is beta software and subject to change. It exposes the following APIs:

  • The cluster configuration file for single-node deployments.
  • ClickHouse SQL schema for raw database access.
  • Protobuf schemas for ingestion for writing custom processors.

One NetMeta has stabilized, these APIs will be stable and backwards compatible.

🛠 Deployment

NetMeta includes a production-ready single node deployment that scales to up to ~100k events/s and billions of database rows. More infos can be found here

Ingestion performance is limited by CPU performance and disk bandwidth. Query performance is limited by disk and memory bandwidth, as well as total amount of available memory for larger in-memory aggregations.

Most reads/writes are sequential due to heavy use of batching in all parts of the stack, and it works fine even on network storage or spinning disks. We recommend local NVMe drives for best performance.

NetMeta can scale to millions of events per seconds in multi-node deployments.

Multi-node deployment

We are currently finalizing the design for multi-node deployments. Please contact us if you're interested in large-scale deployments - we want your feedback!

Monogon OS

NetMeta will be a first-class citizen on Monogon OS - stay tuned!

☸️ Kubernetes

NetMeta works on any Kubernetes cluster that supports LoadBalancer and Ingress objects and can provision storage. It's up to you to carefully read the deployment code and cluster role assigments to make sure it works with your cluster. Note that we use two operators, which require cluster-admin permissions since CRDs are global (Strimzi for Kafka and clickhouse-operator).

All pieces of NetMeta are installed into a single namespace. By default, this is default, which is probably not what you want. You can change the target namespace in the deployment config.

Please contact us if you need help porting NetMeta to an existing k8s cluster.

💼 Support

Please contact us for support and consulting. If you are using NetMeta in production, we'd love to hear from you!

🧩 Related

NetMeta is powered by a number of great open source projects, we use:

🏰 Architecture

flowchart TD;
    sFlow --> goflow
    IPFIX --> goflow
    Netflow --> goflow

    kafka[Kafka Broker]
    clickhouse["ClickHouse Server</br>MergeTree</br>(hourly partitions)"]
    
    goflow --> kafka
    ntm-agent --> kafka
    kafka --> clickhouse
    clickhouse --> |SQL| grafana[Grafana]
    asmap --> clickhouse
    nexthop --> clickhouse
    resolver --> clickhouse

Loading

(C) 2022 Monogon SE.

This software is provided "as-is" and without any express or implied warranties, including, without limitation, the implied warranties of merchantability and fitness for a particular purpose.