What is this?
Apache Arrow is a universal columnar format plus a multi-language toolbox for fast data interchange and in-memory analytics — technologies that let data systems store, process, and move tabular data efficiently.
Apache Arrow is a universal columnar format plus a multi-language toolbox for fast data interchange and in-memory analytics — technologies that let data systems store, process, and move tabular data efficiently.Is it one single library or a whole suite of technologies?A suite. The core is a standard in-memory columnar format, accompanied by an IPC serialization format, the Flight RPC protocol, the ADBC database-access API, and reference libraries in about a dozen languages.
A suite. The core is a standard in-memory columnar format, accompanied by an IPC serialization format, the Flight RPC protocol, the ADBC database-access API, and reference libraries in about a dozen languages.How do I get Arrow into my project?You don't build this monorepo yourself — add the official library for your language: C++, Java, Python, R, Ruby, Rust, Go, .NET, JavaScript, Julia, or Swift. arrow.apache.org links to each implementation.
You don't build this monorepo yourself — add the official library for your language: C++, Java, Python, R, Ruby, Rust, Go, .NET, JavaScript, Julia, or Swift. arrow.apache.org links to each implementation.Do I need a running database or server to use it?No — the columnar format and its containers work in-process on plain buffers. Servers only enter the picture when you deliberately build on Flight or connect to databases through ADBC drivers.
No — the columnar format and its containers work in-process on plain buffers. Servers only enter the picture when you deliberately build on Flight or connect to databases through ADBC drivers.Is every language library equally complete?No — the official libraries implement the format at different stages, so check the feature matrix on the main branch before relying on a specific feature in your language.
No — the official libraries implement the format at different stages, so check the feature matrix on the main branch before relying on a specific feature in your language.Who is Arrow built for?Developers of data systems — databases, query engines, analytics pipelines — that need to store, process, and move tabular data across processes and languages fast.
Developers of data systems — databases, query engines, analytics pipelines — that need to store, process, and move tabular data across processes and languages fast.What is Arrow Flight?An RPC protocol built on top of the Arrow IPC format. It's a building block for remote services that exchange Arrow data with application-defined semantics — a storage server or a database, for example.
An RPC protocol built on top of the Arrow IPC format. It's a building block for remote services that exchange Arrow data with application-defined semantics — a storage server or a database, for example.What is ADBC?Arrow Database Connectivity: an Arrow-powered API with drivers and libraries for access to databases and query engines — Arrow's take on standard database connectivity.
Arrow Database Connectivity: an Arrow-powered API with drivers and libraries for access to databases and query engines — Arrow's take on standard database connectivity.How does Arrow relate to Parquet?Parquet is a file format on disk; Arrow is the in-memory representation. Arrow libraries ship readers and writers for widely-used formats like Parquet and CSV, so files load straight into columnar containers.
Parquet is a file format on disk; Arrow is the in-memory representation. Arrow libraries ship readers and writers for widely-used formats like Parquet and CSV, so files load straight into columnar containers.What is Gandiva?An LLVM-based Arrow expression compiler that lives inside the C++ codebase — one of the project's less visible but interesting components.
An LLVM-based Arrow expression compiler that lives inside the C++ codebase — one of the project's less visible but interesting components.What are similar projects?flatbuffers, modin, grpc, doris, polars, logstash, spark, aws-doc-sdk-examples
flatbuffers, modin, grpc, doris, polars, logstash, spark, aws-doc-sdk-examplesWhich topics is it in?zero-copy, dataframe, parquet, interoperability, serialization, apache, data-processing, rpc, data-engineering, big-data, ruby, analytics
zero-copy, dataframe, parquet, interoperability, serialization, apache, data-processing, rpc, data-engineering, big-data, ruby, analytics