/observability-collection

An ongoing & curated collection of awesome software, frameworks and libraries, learning tutorials and videos, technical guidelines and best practices on the Observability Ecosystem

MIT LicenseMIT

Obsevability Collection

An ongoing & curated collection of awesome software, frameworks and libraries, learning tutorials and videos, technical guidelines and best practices on the Observability Ecosystem. Thanks to our daily readers and contributors. The goal is to build a categorized community-driven collection of very well-known resources. Sharing, suggestions and contributions are always welcome!

About Observability

Table of Contents

1. Best Practices

2. General Tools

Before to start with huge observability solution. If you just need to control some application aspects, visualize how is working your system, or just identify a problem, may be usefull start with one, or a collection application, that help you to get this information in a easy and cheap way.

Additional to this, start with tools to get information about your system to determine if it's working well, can help you to define the final stack if you want to install a corporative solution to any project. I know some stories abot people that install, configure and even evolution some monitoring tools as a corporative solution, an when the solution is in production, they realize that the tools don't cover all the necessaries to control their applications :-D

Following you can see an interesting post from Netflix writteb by Brendan Gregg that show this very clear.

https://netflixtechblog.com/linux-performance-analysis-in-60-000-milliseconds-accc10403c55

In the article you can see how with a few tools and in a short time, you can get a lot of information about your system ;-)

 $ uptime
 $ dmesg | tail
 $ vmstat 1
 $ mpstat -P ALL 1
 $ pidstat 1
 $ iostat -xz 1
 $ free -m
 $ sar -n DEV 1
 $ sar -n TCP,ETCP 1
 $ top

There are many more commands and methodologies you can apply to drill deeper.

3. Collect

Get any data – metrics, events, logs, traces – from everywhere – systems, sensors, queues, databases and networks.

Metrics

  • Opentelemetry - OpenTelemetry is made up of an integrated set of APIs and libraries as well as a collection mechanism via an agent and collector.
  • top - Allows users to monitor processes and system resource usage on Linux. It is one of the most useful tools in a sysadmin's toolbox, and it comes pre-installed on every distribution.
  • htop - Command line utility that allows you to interactively monitor your system's vital resources or server's processes in real time.
  • ctop - Top-like interface for container metrics.
  • sls-dev-tools - The Developer Tools for the Serverless World - think Chrome Dev Tools but for Serverless.
  • OpenCensus - OpenCensus is a set of libraries for various languages that allow you to collect application metrics and distributed traces, then transfer the data to a backend of your choice in real time.
  • Opentracing - Vendor-neutral APIs and instrumentation for distributed tracing.
  • Openmetrics - An effort to create an open standard for transmitting metrics at scale, with support for both text representation and Protocol Buffers.
  • Micrometer - Micrometer provides a simple facade over the instrumentation clients for the most popular monitoring systems, allowing you to instrument your JVM-based application code without vendor lock-in. Think SLF4J, but for metrics.
  • cAdvisor - cAdvisor (Container Advisor) provides container users an understanding of the resource usage and performance characteristics of their running containers.
  • Node-exporter - Prometheus stack, Exporter for machine metrics.
  • Elastic Beats - Lightweight shippers for Elasticsearch & Logstash, Elastic stack.
  • Collectd - The system statistics collection daemon.
  • Tcollector - Data collection framework for OpenTSDB.
  • Performance Co-Pilot - Performance Co-Pilot is a system performance analysis toolkit.
  • inspectIT Ocelot - Java agent for collecting performance, tracing and business data.
  • Kamon - Monitoring applications running on the JVM.
  • Netflix Vector - An on-host performance monitoring framework which exposes hand picked high resolution metrics to every engineer's browser.
  • Express State Metrics - Simple, self-hosted module based on Socket.io and Chart.js to report realtime server metrics for Express-based node servers.
  • Kube State Metrics - The kube-state-metrics is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects.
  • MyPerf4J - High performance Java APM. Powered by ASM. Try it. Test it. If you feel its better, use it.
  • SkyAPM-dotnet - SkyAPM-dotnet provides the native support agent in C# and .NETStandard platform, with the helps from Apache SkyWalking committer team.
  • pktvisor - Observability agent for summarizing high volume, information dense data streams down to lightweight, immediately actionable observability data directly at the edge.

Tracing

  • Sleuth - Spring Cloud Sleuth implements a distributed tracing solution for Spring Cloud, borrowing heavily from Dapper, Zipkin and HTrace.
  • inspectIT Ocelot - Java agent for collecting performance, tracing and business data.

Logging

  • Elastic Beats - Lightweight shippers for Elasticsearch & Logstash, Elastic stack.
  • mTAIL - Windows program that extract internal monitoring data from application logs for collection in a timeseries database.

Events & Problems

  • kubernetes-event-exporter - This tool allows exporting the often missed Kubernetes events to various outputs so that they can be used for observability or alerting purposes.
  • kspan - Turning Kubernetes Events into spans.
  • KubeEye - KubeEye aims to find various problems on Kubernetes, such as application misconfiguration(using Polaris), cluster components unhealthy and node problems(using Node-Problem-Detector).

4. Load Generators and Synthetic Traffic

  • JMeter - Java application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions.
  • Taurus - Taurus relies on JMeter, Gatling, Locust.io, Grinder and Selenium WebDriver as its underlying tools. Free and open source under Apache 2.0 License.
  • K6 - k6 is a developer-centric, free and open-source load testing tool built for making performance testing a productive and enjoyable experience.
  • Vegeta - HTTP load testing tool built out of a need to drill HTTP services with a constant request rate. It can be used both as a command line utility and a library.
  • Trunks - Trunks, like every son, is derived from the father Vegeta with some enhanced skills.
  • Yandex Tank - Yandex.Tank is an extensible open source load testing tool for advanced linux users which is especially good as a part of an automated load testing suite.
  • ghz - Simple gRPC benchmarking and load testing tool inspired by hey and grpcurl.
  • Locust - Locust is an easy-to-use, distributed, user load testing tool. It is intended for load-testing web sites (or other systems) and figuring out how many concurrent users a system can handle.
  • Pandora - Pandora is a high-performance load generator in Go language. It has built-in HTTP(S) and HTTP/2 support and you can write your own load scenarios in Go, compiling them just before your test.
  • Gatling - Load test as code.
  • GoReplay - Open-source tool for capturing and replaying live HTTP traffic into a test environment in order to continuously test your system with real data.
  • phantom - Evgeniy Mamchits' phantom is a very fast (100 000+ RPS) shooter written in C++ (default).
  • BFG - A modular tool and framework for load generation that supports HTTP/2.
  • Bender - Bender makes it easy to build load testing applications for services using protocols like HTTP, Thrift, Protocol Buffers and many more. Bender provides a library of flexible, powerful primitives that can be combined (with plain Go code) to build load testers customized to any use case and that evolve with your service over time.

5. Transport

The transport tools simply serve as transport pipelines for data. This includes messaging systems, proprietary protocols and exchange formats.

  • Apache Kafka - Publish-subscribe messaging rethought as a distributed commit log.
  • Stanza - Fast and lightweight log transport and processing agent. It's designed as a modern replacement for Fluentd, Fluent Bit, and Logstash.
  • Redis - Redis is an open source, in-memory data structure store, used as a database, cache and message broker. It supports many different data structures such as stringes, hashes, list, etc.
  • Rsyslog - RSYSLOG is the rocket-fast system for log processing.
  • ØMQ - Brokerless intelligent transport layer.
  • ActiveMQ - Powerful open source messaging and integration patterns server.
  • Aeron - Efficient reliable UDP unicast, UDP multicast, and IPC message transport.
  • Apollo - Faster, more reliable, easier to maintain messaging broker built from the foundations of the original ActiveMQ.
  • Ascoltatori - Pub/sub library for Node.
  • Beanstalk - Simple, fast work queue.
  • Disque - Distributed message broker.
  • Eventuate - A platform for developing asynchronous microservices solving the distributed data management problems.
  • Malamute - ZeroMQ enterprise messaging broker.
  • Mist - A distributed, tag-based pub/sub service.
  • Mosca - MQTT broker as a module.
  • Mosquitto - Open source message broker that implements the MQTT protocol.
  • Nanomsg - Socket library that provides several common communication patterns for building distributed systems.
  • NATS - Open source, high-performance, lightweight cloud messaging system.
  • NSQ - A realtime distributed messaging platform.
  • Pulsar - Distributed pub-sub messaging system.
  • Qpid - Cross-platform messaging components built on AMQP.
  • RabbitMQ - Open source Erlang-based message broker that just works.
  • RocketMQ - A low latency, reliable, scalable, easy to use message oriented middleware born from alibaba massive messaging business.
  • VerneMQ - Open source software, extendable, and enterprise support is available.
  • Zenoh - A low overhead, low latency, high throughput open-source protocol that blends traditional pub/sub with geo distributed storage, queries and computations for unifying data in motion, data at rest and computations.

6. Collector

Receive data from the agents or instrumentation frameworks. The received data is usually persisted to some kind of storage or piped to another tool.

Depending on the collector type, performance data enhancement and modification is also possible inside of the collector.

In addition, collectors can have other responsibilities. For example, some expose the data access API, configuration points for the agents or user interface for interaction with the stored data.

Metrics

  • Telegraf - TICK stack, The plugin-driven server agent for collecting & reporting metrics.
  • Prometheus - The Prometheus monitoring system and time series database.
  • statsd - A network daemon that runs on the Node.js platform and listens for statistics, like counters and timers, sent over UDP or TCP and sends aggregates to one or more pluggable backend services.

Logging

  • Graylog - Simply great centralized log management.
  • Loki - Horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus.
  • Brubeck - Statsd-compatible stats aggregator written in C.
  • GoAccess - GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser. It provides fast and valuable HTTP statistics for system administrators that require a visual server report on the fly.
  • Logbook - Extensible Java library to enable complete request and response logging for different client- and server-side technologies.

Events

  • Nothing for the moment :-P

7. Storage

Time Series Database

  • Prometheus - The Prometheus monitoring system and time series database.
  • VictoriaMetrics - VictoriaMetrics is a fast, cost-effective and scalable monitoring solution and time series database.
  • InfluxDB - InfluxDB is an open-source time series database developed by InfluxData.
  • OpenTSDB - OpenTSDB, written in java.
  • kairosDB - Fast Time Series Database on Cassandra.
  • Graphite - Store numeric time-series data and render graphs of this data on demand.
  • M3DB - Fully open source metrics platform built on M3DB, a distributed timeseries database.
  • Netflix Atlas - Atlas features in-memory data storage, allowing it to gather and report very large numbers of metrics, very quickly.
  • TimescaleDB - PostgreSQL for time‑series.
  • QuestDB - QuestDB is the fastest open source time series database.
  • Metric Tank - Provides high availability and efficient long-term storage, retrieval, and processing for large-scale environments.

Time Series Cache

  • Trickster - Open Source HTTP Reverse Proxy Cache and Time Series Dashboard Accelerator.

"Meta Projects" (data storage, multi-tenant, aggregation, high availability, etc)

  • Thanos - Highly available Prometheus setup with long term storage capabilities.
  • Cortex - Horizontally scalable, highly available, multi-tenant, long term storage for Prometheus.
  • Observatorium - This system will allow you to ingest, store and use common observability signals like metrics, logging and tracing.
  • Opstrace - The Opstrace Distribution is a secure, horizontally-scalable, open source observability platform that you can install in your cloud account.
  • Apache HBase - Apache HBase is the Hadoop database, a distributed, scalable, big data store.

Tracing

  • Zipkin - A distributed tracing system.
  • Jaeger - Monitor and troubleshoot transactions in complex distributed systems.
  • Grafana Tempo - Open source, easy-to-use and high-scale distributed tracing backend.
  • Haystack - A resilient, scalable tracing and analysis system.

Search Engine

  • Apache Lucene - Java library providing powerful indexing and search features.
  • Apache Solr - Solr is the popular, blazing-fast, open source enterprise search platform built on Apache Lucene.
  • Elasticsearch - Free and Open, Distributed, RESTful Search Engine.

Graph Database

  • neo4j - Graphs for Everyone.
  • ArangoDB - Natively store data for graph, document and search needs.

SQL Database

  • MySQL - Relational database management system.
  • MariaDB - Open source relational database.
  • PostgreSQL - Open source relational database.
  • CockroachDB - CockroachDB delivers Distributed SQL, combining the familiarity of relational data with limitless, elastic cloud scale, bulletproof resilience… and more.
  • ClickHouse - ClickHouse is a fast open-source OLAP database management system.

NoSQL Database (The Others :-P)

  • Apache Cassandra - Scalability and high availability with linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure.
  • MongoDB - MongoDB is a document database with the scalability and flexibility that you want with the querying and indexing that you need.
  • Redis - Multi-model NoSQL database server enables search, messaging, streaming, graph, and other capabilities.
  • RethinkDB - RethinkDB pushes JSON to your apps in realtime.
  • SQLite - C-language library that implements a small, fast, self-contained, high-reliability, full-featured, SQL database engine.
  • CouchDB - Seamless multi-master sync, that scales from Big Data to Mobile, with an Intuitive HTTP/JSON API and designed for Reliability.

8. Visualization

General & Tools

  • ExplorViz - Live trace visualization for large software landscapes.
  • Flame Graph - Visualization of profiled software, allowing the most frequent code-paths to be identified quickly and accurately.
  • Flame Scope - FlameScope is a visualization tool for exploring different time ranges as Flame Graphs.

Dashboarding

  • Grafana - The first really good dashboard for displaying metrics.
  • SLO-Tracker - A simple but effective way to track SLO's and Error budgets. SLO-tracker can be integrated with few alerting tools via webhook integration to receive SLO voilating incidents.
  • Chronograf - Chronograf is the user interface and administrative component of the InfluxDB platform.
  • Kibana - Elastic stack.
  • Prometheus - The Prometheus monitoring system and time series database.
  • PromViz - Promviz is an application that helps you visualize the traffic of your cluster from Prometheus data.
  • Uchiwa - Uchiwa is a simple dashboard for the Sensu monitoring framework, built with Go and AngularJS.
  • Grafterm - Visualize metrics dashboards on the terminal, like a simplified and minimalist version of Grafana for terminal.

Tracing

  • Zipkin - A distributed tracing system.
  • Jaeger - Monitor and troubleshoot transactions in complex distributed systems.
  • Grafana Tempo - Open source, easy-to-use and high-scale distributed tracing backend.
  • Haystack - A resilient, scalable tracing and analysis system.

Graph of Nodes

  • Vizceral - Vizceral is a component for displaying traffic data on a webgl canvas. If a graph of nodes and edges with data about traffic volume is provided, it will render a traffic graph animating the connection volume between nodes.
  • Novatec Service Dependency Graph Panel - A Grafana panel for visualizing a Service Dependency Graph.

Uptime

  • Monitive - Free for 1 service, checked every 10 minutes with unlimited email & twitter alerts.
  • UptimeRobot - Free for 50 monitors, checked every 5 minutes.
  • Nodown - Free up to 5 monitors, with email notifications.
  • OverOps - OverOps provides Automated Root Cause (ARC) analysis to reduce the time to identify and fix critical production application errors.

9. Processing and Analyze and Act

Tools for rocessing the system data.

  • Pipeline tools that receive system data in one format, buffer or generate additional value on the raw data, and usually output or store it in another or the same format
  • Usually ingests the data from a multitude of sources and also sends the results to different destinations
  • Can setup alerts with a simple click or perform complex anomaly detection based on machine learning algorithms
  • Send alerts to popular services like Slack, Email, SMS or PagerDuty
  • Create custom triggers to perform any action. Integration with the corporative systems like Jira, CI/CD environment, source code anayze tools, etc.

Processing

  • Logstash - Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite "stash".
  • Fluentd - Fluentd is an open source data collector for unified logging layer.
  • Vector - Vector is a high-performance observability data router. It makes collecting, transforming, and sending logs, metrics, and events easy. It decouples data collection & routing from your services, giving you control and data ownership, among many other benefits.
  • Haystack - A resilient, scalable tracing and analysis system.
  • Kapacitor - Kapacitor is a real-time streaming data processing engine.

Alerts

  • Sloth - Easy and simple Prometheus SLO (service level objectives) generator.
  • Bosun - Is an open-source, MIT licensed, monitoring and alerting system.
  • ElastAlert - Easy & Flexible Alerting With Elasticsearch.
  • Bosun - Time Series Alerting Framework.
  • Moira - Most powerful alerting system, backed by Graphite.
  • Alerta - Tool used to consolidate and de-duplicate alerts from multiple sources for quick ‘at-a-glance’ visualisation.
  • Flapjack - Monitoring notification routing & event processing system.
  • Seyren - An alerting dashboard for Graphite.

Triggers

  • Odin - A programmable, observable and distributed job orchestration system.

Anomalies Detection

  • Failure Mode and Effects Analysis (FMEA) - Documents current knowledge and actions about the risks of failures, for use in continuous improvement.
  • Banshee - Real-time anomalies(outliers) detection system for periodic metrics.
  • Project Scorpio - Log Anomaly Detector.
  • Anomaly Detection in Prometheus Metrics - Prototype for a Prometheus Anomaly Detector (PAD) which can be deployed on OpenShift. The PAD is a framework to deploy a metric prediction model to detect anomalies in prometheus metrics.
  • Prophet - Prophet is a forecasting procedure implemented in R and Python. It is fast and provides completely automated forecasts that can be tuned by hand by data scientists and analysts.
  • Anomaly Detection Toolkit (ADTK) - Python package for unsupervised / rule-based time series anomaly detection.

10. Application Performance Monitoring Solutions (APM)

  • swagger-stats - API Telemetry and APM.
  • Cloudprober - An active monitoring software to detect failures before your customers do.
  • Hubble - Network, Service & Security Observability for Kubernetes.
  • Datav - A modern apm solution for enterprise, an open-source alternative to DataDog, New Relic, etc.
  • SigNoz - Monitor your applications and troubleshoot problems in your deployed applications, an open-source alternative to DataDog, New Relic, etc.
  • SkyWalking - Application performance monitor tool for distributed systems, especially designed for microservices, cloud native and container-based (Docker, K8s, Mesos) architectures.
  • Pinpoint - Pinpoint is an APM (Application Performance Management) tool for large-scale distributed systems written in Java / PHP/PYTHON.
  • Falcon Plus - An open-source and enterprise-level monitoring system.
  • dynatrace APM - Best-in-class APM from the category leader. Ensure application performance, innovate faster, collaborate efficiently, and deliver more value with dramatically less effort.
  • Elastic APM - Application performance monitoring system built on the Elastic Stack.
  • DataDog - Unified Monitoring For Metrics, Traces, & Logs.
  • Honeycomb - Give all software engineering teams the observability they need to eliminate toil and delight their users.
  • NewRelic - Complete view of your applications and operating environment.
  • AppDynamics - Business and application performance monitoring.
  • sematext - Solutions for performance monitoring.
  • Instrumental - Real-time application and server monitoring.
  • Kamon apm - Point and click to find the endpoints, database queries, and API calls that affect your user's experience.
  • Scouter - Scouter is an open source APM (Application Performance Management) tool.
  • Netdata - Troubleshoot slowdowns and anomalies in your infrastructure with thousands of per-second metrics, meaningful visualizations, and insightful health alarms with zero configuration.
  • Stagemonitor - An open source solution to application performance monitoring for java server applications.
  • Checkmk Server - Monitor your entire hybrid IT infrastructure.
  • Icinga - The Icinga stack spans six core strengths that cover all aspects of monitoring.
  • Sentry - From error tracking to performance monitoring, developers can see what actually matters, solve quicker, and learn continuously about their applications - from the frontend to the backend.
  • Nagios - Computer system, network and infrastructure monitoring software application.
  • Sensu - The Observability Pipeline that delivers monitoring as code on. any cloud.
  • Kieker - Monitoring, analysis and tool integration.
  • Soda Sql - Data profiling, testing, and monitoring for SQL accessible data.

11. Service Mesh

  • Istio - Istio generates detailed telemetry for all service communications within a mesh.
  • Kiali - Observability console for Istio with service mesh configuration capabilities. It helps you to understand the structure of your service mesh by inferring the topology, and also provides the health of your mesh.

12. Observability as a Service

  • Grafana Cloud - Composable observability platform, integrating metrics, traces and logs with Grafana.
  • NexClipper - NexClipper is Open Source software for Cloud Native monitoring and operation , especailly for Kubernetes, to support enterprise environments and integrate with Prometheus.
  • Sysdig Prometheus - Cloud scale monitoring solution with full Prometheus compatibility.
  • WhaTap - Provides an integrated monitoring service for DevOps that analyzes application performance issue running on kubernetes in real time.
  • CloudWatch - Observability of your AWS resources and applications on AWS and on-premises.
  • Google Cloud Monitoring - Gain visibility into the performance, availability, and health of your applications and infrastructure.
  • Azure Monitor - Full observability into your applications, infrastructure, and network.
  • Guance - China local "All in one" observability platform, it can integrate any open source collecting method.
  • Alibaba Cloud Logs Service - Complete real-time data logging service that has been developed by Alibaba Group.
  • Graylog Cloud - Seamless log data collection, faster analysis, and the answers you need when you need them.
  • humio - Modern log management with streaming observability and affordable Unlimited Plans.
  • logdna - A comprehensive log solution to control all of your log data.
  • loggly - See it all in one place. Dozens of log sources, no proprietary agents.
  • logiq.ai - Platform to connect observability data from any source to any destination.
  • SCALYR - Log analytics - blazing fast at petabyte scale, feature rich at surprisingly low cost.
  • splunk - Extensible data platform powers unified security, full-stack observability and limitless custom applications.
  • sumo logic - Reduce downtime with real-time alerting, dashboards, and machine-learning-powered analytics for all three types of telemetry — logs, metrics, and traces.
  • Tencent Cloud Log Service - Tencent is an internet service portal offering value-added internet, mobile, telecom, and online advertising services.

13. Examples and Sandbox's

14. References

15. Contributing

Contributions welcome! Read the contribution guidelines first.

16. License

MIT License

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

Back to top