/vector

A lightweight and ultra-fast tool for building observability pipelines

Primary LanguageRustApache License 2.0Apache-2.0

Components  •  Guides  •  Docs  •  Blog  •  Download v0.9.2

Vector

What is Vector?

Vector is a lightweight, ultra-fast, open-source tool for building observability pipelines. Compared to Logstash and friends, Vector improves throughput by ~10X while significanly reducing CPU and memory usage.

Principles

  • Reliability First. - Built in Rust, Vector's primary design goal is reliability.
  • One Tool. All Data. - One simple tool gets your logs, metrics, and traces (coming soon) from A to B.
  • Single Responsibility. - Vector is a data router, it does not plan to become a distributed processing framework.

Who should use Vector?

  • You SHOULD use Vector to replace Logstash, Fluent*, Telegraf, Beats, or similar tools.
  • You SHOULD use Vector as a daemon or sidecar.
  • You SHOULD use Vector as a Kafka consumer/producer for observability data.
  • You SHOULD use Vector in resource constrained environments (such as devices).
  • You SHOULD NOT use Vector if you need an advanced distributed stream processing framework.
  • You SHOULD NOT use Vector to replace Kafka. Vector is designed to work with Kafka!
  • You SHOULD NOT use Vector for non-observability data such as analytics data.

Community

  • Vector is downloaded over 100,000 times per day.
  • Vector's largest user processes over 10TB daily.
  • Vector is used by multiple fortune 500 companies with stringent production requirements.
  • Vector has over 15 active contributors and growing.

Documentation

About

Setup

Reference

Administration

Resources

Performance

The following performance tests demonstrate baseline performance between common protocols with the exception of the Regex Parsing test.

Test Vector Filebeat FluentBit FluentD Logstash SplunkUF SplunkHF
TCP to Blackhole 86mib/s n/a 64.4mib/s 27.7mib/s 40.6mib/s n/a n/a
File to TCP 76.7mib/s 7.8mib/s 35mib/s 26.1mib/s 3.1mib/s 40.1mib/s 39mib/s
Regex Parsing 13.2mib/s n/a 20.5mib/s 2.6mib/s 4.6mib/s n/a 7.8mib/s
TCP to HTTP 26.7mib/s n/a 19.6mib/s <1mib/s 2.7mib/s n/a n/a
TCP to TCP 69.9mib/s 5mib/s 67.1mib/s 3.9mib/s 10mib/s 70.4mib/s 7.6mib/s

To learn more about our performance tests, please see the Vector test harness.

Correctness

The following correctness tests are not exhaustive, but they demonstrate fundamental differences in quality and attention to detail:

Test Vector Filebeat FluentBit FluentD Logstash Splunk UF Splunk HF
Disk Buffer Persistence ✅ ✅ ❌ ❌ ⚠️ ✅ ✅
File Rotate (create) ✅ ✅ ✅ ✅ ✅ ✅ ✅
File Rotate (copytruncate) ✅ ❌ ❌ ❌ ❌ ✅ ✅
File Truncation ✅ ✅ ✅ ✅ ✅ ✅ ✅
Process (SIGHUP) ✅ ❌ ❌ ❌ ⚠️ ✅ ✅
JSON (wrapped) ✅ ✅ ❌ ✅ ✅ ✅ ✅

To learn more about our correctness tests, please see the Vector test harness.

The Little Details

Data Model

Control Flow

Data Processing

Operations

Reliability

  • Memory safety - Vector is built in Rust and is memory safe, avoiding a large class of memory related errors.
  • Decoupled buffer design - Buffers are per-sink; a bad sink won't bring the entire pipeline to a halt.
  • Intelligent retries - A fibonacci backoff algorithsm with jitter makes Vector a good citizen during outages.
  • Backpressure & load shedding - Buffers can be configured to provide backpressure or shed load.
  • Rate-limited internal logging - Vector's internal logging is rate-limited avoiding IO saturation if errors occur.
  • Sink healthchecks - Healthchecks provide startup safety and prevent deploys with bad configuration.
  • Robust disk buffering - Vector uses leveldb for robust data durability across restarts.

UX

Installation

Run the following in your terminal, then follow the on-screen instructions.

curl --proto '=https' --tlsv1.2 -sSf https://sh.vector.dev | sh

Or use your own preferred method.

Latest Highlights

view all...

Developed with ❤️ by Timber.io - Security Policy - Privacy Policy