vector README

The project's README from github.com/vectordotdev/vector. As one markdown file: README.md.

Nightly Integration/E2E Test Suite Component Features

Vector

Quickstart  •   Docs  •   Guides  •   Integrations  •   Chat  •   Download  •   Container Images  •   Rust Crate Docs

What is Vector?

Vector is a high-performance, end-to-end (agent & aggregator) observability data pipeline that puts you in control of your observability data. Collect, transform, and route all your logs and metrics to any vendors you want today and any other vendors you may want tomorrow. Vector enables dramatic cost reduction, novel data enrichment, and data security where you need it, not where it is most convenient for your vendors. Additionally, it is open source and up to 10x faster than every alternative in the space.

To get started, follow our quickstart guide or install Vector.

Vector is maintained by Datadog's Community Open Source Engineering team.

Principles

Use cases

Community

Documentation

All user documentation is available at vector.dev/docs.

Other Resources:

Comparisons

Performance

The following performance tests demonstrate baseline performance between common protocols with the exception of the Regex Parsing test.

Test Vector Filebeat FluentBit FluentD Logstash SplunkUF SplunkHF
TCP to Blackhole 86mib/s n/a 64.4mib/s 27.7mib/s 40.6mib/s n/a n/a
File to TCP 76.7mib/s 7.8mib/s 35mib/s 26.1mib/s 3.1mib/s 40.1mib/s 39mib/s
Regex Parsing 13.2mib/s n/a 20.5mib/s 2.6mib/s 4.6mib/s n/a 7.8mib/s
TCP to HTTP 26.7mib/s n/a 19.6mib/s <1mib/s 2.7mib/s n/a n/a
TCP to TCP 69.9mib/s 5mib/s 67.1mib/s 3.9mib/s 10mib/s 70.4mib/s 7.6mib/s

To learn more about our performance tests, please see the Vector test harness.

Correctness

The following correctness tests are not exhaustive, but they demonstrate fundamental differences in quality and attention to detail:

Test Vector Filebeat FluentBit FluentD Logstash Splunk UF Splunk HF
Disk Buffer Persistence ✓ ✓ ⚠ ✓ ✓
File Rotate (create) ✓ ✓ ✓ ✓ ✓ ✓ ✓
File Rotate (copytruncate) ✓ ✓ ✓
File Truncation ✓ ✓ ✓ ✓ ✓ ✓ ✓
Process (SIGHUP) ✓ ⚠ ✓ ✓
JSON (wrapped) ✓ ✓ ✓ ✓ ✓ ✓ ✓

To learn more about our correctness tests, please see the Vector test harness.

Features

Vector is an end-to-end, unified, open data platform.

Vector Beats Fluentbit Fluentd Logstash Splunk UF Splunk HF Telegraf
End-to-end ✓ ✓
Agent ✓ ✓ ✓ ✓ ✓
Aggregator ✓ ✓ ✓ ✓ ✓
Unified ✓ ✓
Logs ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Metrics ✓ ⚠ ⚠ ⚠ ⚠ ⚠ ⚠ ✓
Traces 🚧
Open ✓ ✓ ✓ ✓
Open-source ✓ ✓ ✓ ✓ ✓ ✓
Vendor-neutral ✓ ✓ ✓ ✓
Reliability ✓
Memory-safe ✓ ✓
Delivery guarantees ✓ ✓ ✓
Multi-core ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

⚠ = Not interoperable, metrics are represented as structured logs


Developed with ❤️ by Datadog - Security Policy - Privacy Policy

vector (vectordotdev/vector)

Searching Database…

22.7K ·GitHub
这是什么?一个开源的高性能可观测性数据管道:它会收集、转换你的日志和指标,并路由到你选择的任意厂商,让你始终掌控自己的可观测性数据。它由 Datadog 的社区开源工程团队维护。
这是需要注册使用的云服务,还是我自己运行的软件?既不是 SaaS 也不是程序库——你需要自己部署它,角色有两种:要么作为 agent 部署在每台主机上,要么作为中心聚合器。两者配合起来,就在你的数据源和目的地之间构成一条完整的端到端管道。
我怎么把它装好并跑起来?先为你的平台安装——文档涵盖手动安装、操作系统软件包、包管理器以及各平台指南(包括 Kubernetes)——然后跟着快速入门走。所有内容都在 vector.dev/docs 上有说明。
用它搭建管道需要写代码吗?不需要写代码:你通过配置 sources(数据源)、transforms(转换)和 sinks(输出)来定义管道。要调整数据格式,有一个由 VRL(Vector Remap Language)驱动的 remap 转换;需要脚本逻辑时还可以用 Lua 转换。
它只处理日志,还是指标和链路追踪也能处理?它统一支持这三类信号,但有保留:目前支持日志和指标——指标仍处于 beta 阶段——而链路追踪还列为即将推出。
它经受过大规模考验,还是比较小众?久经考验:Atlassian、T-Mobile、Comcast、Zendesk、Discord、Fastly、Visa、Instacart 等初创公司和大型企业都在生产环境使用它,每天下载量超过 10 万次。
我的可观测性数据会流向哪里——必须经过某个厂商的云吗?由你决定:Vector 的设计初衷就是把数据控制权留在你自己手里,可以路由到任意厂商或你自己的存储——输出目标包括 AWS S3、CloudWatch Logs、GCP Cloud Storage、ClickHouse 和 Elasticsearch。
它和 FluentBit、Logstash 或 Filebeat 相比怎么样?README 中的基准测试显示,它比同类所有替代方案最多快 10 倍:TCP 到黑洞的吞吐为 86 MiB/s,而 FluentBit 是 64.4,Logstash 是 40.6,FluentD 是 27.7 MiB/s。它用 Rust 编写,可靠性是首要设计目标。
它能连接哪些数据源和目的地?文档中的数据源包括文件、Docker 日志、journald、HTTP、Kafka 和 socket。输出目标包括 AWS S3、AWS CloudWatch Logs、GCP Cloud Storage、ClickHouse 和 Elasticsearch——而且它的设计理念是“今天能路由到你想要的任何厂商,明天也能路由到你将来想要的任何厂商”。
它能应付真正的生产环境流量吗?可以——它最大的单一用户每天处理超过 500TB 的数据,项目拥有 500 多名贡献者,重度生产使用很有代表性。
Size
146.5 MB
Stars
22.7K
Age
8y
Commits · 90 Days