What is this?
An open-source, high-performance observability data pipeline: it collects, transforms, and routes your logs and metrics to any vendors you choose, so you keep control of your observability data. It is maintained by Datadog's Community Open Source Engineering team.
An open-source, high-performance observability data pipeline: it collects, transforms, and routes your logs and metrics to any vendors you choose, so you keep control of your observability data. It is maintained by Datadog's Community Open Source Engineering team.Is this a cloud service I sign up for, or software I run myself?Neither SaaS nor library — you deploy it yourself in one of two roles: as an agent on each host, or as a central aggregator. Together they form a complete end-to-end pipeline between your data sources and destinations.
Neither SaaS nor library — you deploy it yourself in one of two roles: as an agent on each host, or as a central aggregator. Together they form a complete end-to-end pipeline between your data sources and destinations.How do I get it installed and running?Install it for your platform — the docs cover manual installs, OS packages, package managers, and platform guides (Kubernetes included) — then follow the quickstart. Everything is documented at vector.dev/docs.
Install it for your platform — the docs cover manual installs, OS packages, package managers, and platform guides (Kubernetes included) — then follow the quickstart. Everything is documented at vector.dev/docs.Do I need to write code to build pipelines with it?No coding required: you define pipelines by configuring sources, transforms, and sinks. For reshaping data there is a remap transform driven by VRL (Vector Remap Language), plus a Lua transform when you need scripted logic.
No coding required: you define pipelines by configuring sources, transforms, and sinks. For reshaping data there is a remap transform driven by VRL (Vector Remap Language), plus a Lua transform when you need scripted logic.Does it handle only logs, or metrics and traces too?It unifies all three signal types, but with caveats: logs and metrics today — metrics still in beta — while traces are listed as coming soon.
It unifies all three signal types, but with caveats: logs and metrics today — metrics still in beta — while traces are listed as coming soon.Is it battle-tested, or still niche?Battle-tested: startups and enterprises like Atlassian, T-Mobile, Comcast, Zendesk, Discord, Fastly, Visa, and Instacart run it in production, and it is downloaded over 100,000 times per day.
Battle-tested: startups and enterprises like Atlassian, T-Mobile, Comcast, Zendesk, Discord, Fastly, Visa, and Instacart run it in production, and it is downloaded over 100,000 times per day.Where does my observability data go — do I have to route it through a vendor's cloud?You decide: Vector is built around keeping data control on your side, routing to any vendor or to your own storage — sinks include AWS S3, CloudWatch Logs, GCP Cloud Storage, ClickHouse, and Elasticsearch.
You decide: Vector is built around keeping data control on your side, routing to any vendor or to your own storage — sinks include AWS S3, CloudWatch Logs, GCP Cloud Storage, ClickHouse, and Elasticsearch.How does it compare to FluentBit, Logstash, or Filebeat?The README benchmarks it at up to 10x faster than every alternative in the space: TCP-to-blackhole at 86 MiB/s vs FluentBit 64.4, Logstash 40.6, and FluentD 27.7 MiB/s. It is built in Rust with reliability as the primary design goal.
The README benchmarks it at up to 10x faster than every alternative in the space: TCP-to-blackhole at 86 MiB/s vs FluentBit 64.4, Logstash 40.6, and FluentD 27.7 MiB/s. It is built in Rust with reliability as the primary design goal.What sources and destinations can it connect to?Documented sources include files, Docker logs, journald, HTTP, Kafka, and sockets. Sinks include AWS S3, AWS CloudWatch Logs, GCP Cloud Storage, ClickHouse, and Elasticsearch — and it is designed to route to 'any vendors you want today and any other vendors you may want tomorrow'.
Documented sources include files, Docker logs, journald, HTTP, Kafka, and sockets. Sinks include AWS S3, AWS CloudWatch Logs, GCP Cloud Storage, ClickHouse, and Elasticsearch — and it is designed to route to 'any vendors you want today and any other vendors you may want tomorrow'.Can it handle serious production volume?Yes — its largest single user processes over 500TB of data daily, and the project counts over 500 contributors, so heavy production use is well represented.
Yes — its largest single user processes over 500TB of data daily, and the project counts over 500 contributors, so heavy production use is well represented.What are similar projects?openobserve, telegraf, signoz, fluentd, quickwit, kafka, nightingale, questdb
openobserve, telegraf, signoz, fluentd, quickwit, kafka, nightingale, questdbWhich topics is it in?pipelines, logs, stream-processing, events, etl, rust-lang, telemetry, clickhouse, kafka, metrics, elasticsearch, lua
pipelines, logs, stream-processing, events, etl, rust-lang, telemetry, clickhouse, kafka, metrics, elasticsearch, lua