NetMeta is a scalable network observability toolkit optimized for performance.
Flows are not pre-aggregated and stored with one second resolution. This allows for queries in arbitrary dimensions with high-fidelity graphs.
NetMeta captures, aggregates and analyzes events from a variety of data sources:
- sFlow
- NetFlow/IPFIX
- Port Mirror (AF_PACKET)
- Linux NFLOG (soon)
- Linux conntrack (soon)
- Scamper traces (soon)
- GCP VPC Flow Logs (soon)
- AWS VPC Flow Logs (soon)
The different pros/cons between these and a more informations about the ingest of events can be found here
NetMeta also allows to integrate FastNetMon Attack notifications on the Dashboards. The docs to setup this and other Grafana related settings can be found here
Sampling rate is detected automatically. Different devices with different sampling rates can be mixed. IPv6 is fully supported throughout the stack.
NetMeta is beta software and subject to change. It exposes the following APIs:
- The cluster configuration file for single-node deployments.
- ClickHouse SQL schema for raw database access.
- Protobuf schemas for ingestion for writing custom processors.
One NetMeta has stabilized, these APIs will be stable and backwards compatible.
NetMeta includes a production-ready single node deployment that scales to up to ~100k events/s and billions of database rows. More infos can be found here
Ingestion performance is limited by CPU performance and disk bandwidth. Query performance is limited by disk and memory bandwidth, as well as total amount of available memory for larger in-memory aggregations.
Most reads/writes are sequential due to heavy use of batching in all parts of the stack, and it works fine even on network storage or spinning disks. We recommend local NVMe drives for best performance.
NetMeta can scale to millions of events per seconds in multi-node deployments.
We are currently finalizing the design for multi-node deployments. Please contact us if you're interested in large-scale deployments - we want your feedback!
NetMeta will be a first-class citizen on Monogon OS - stay tuned!
NetMeta works on any Kubernetes cluster that supports LoadBalancer and Ingress objects and can provision storage. It's up to you to carefully read the deployment code and cluster role assigments to make sure it works with your cluster. Note that we use two operators, which require cluster-admin permissions since CRDs are global (Strimzi for Kafka and clickhouse-operator).
All pieces of NetMeta are installed into a single namespace. By default, this is default, which is
probably not what
you want.
You can change the target namespace in the deployment config.
Please contact us if you need help porting NetMeta to an existing k8s cluster.
Please contact us for support and consulting. If you are using NetMeta in production, we'd love to hear from you!
NetMeta is powered by a number of great open source projects, we use:
- ClickHouse as the main database
- Kafka as a queue in front of ClickHouse
- Grafana with
- clickhouse-grafana as frontend
- goflow as the sFlow/Netflow collector
- Strimzi to deploy Kafka,
- clickhouse-operator to deploy ClickHouse, as well as
- Kubernetes and Rancher's k3s.
flowchart TD;
sFlow --> goflow
IPFIX --> goflow
Netflow --> goflow
kafka[Kafka Broker]
clickhouse["ClickHouse Server</br>MergeTree</br>(hourly partitions)"]
goflow --> kafka
ntm-agent --> kafka
kafka --> clickhouse
clickhouse --> |SQL| grafana[Grafana]
asmap --> clickhouse
nexthop --> clickhouse
resolver --> clickhouse
(C) 2022 Monogon SE.
This software is provided "as-is" and without any express or implied warranties, including, without limitation, the implied warranties of merchantability and fitness for a particular purpose.