What is this?
3FS (Fire-Flyer File System) is a high-performance distributed file system built by DeepSeek to handle AI training and inference storage workloads at scale, pooling thousands of SSDs and hundreds of storage nodes behind RDMA networking into one shared storage layer.
3FS (Fire-Flyer File System) is a high-performance distributed file system built by DeepSeek to handle AI training and inference storage workloads at scale, pooling thousands of SSDs and hundreds of storage nodes behind RDMA networking into one shared storage layer.Do I mount it like a normal file system, or use a special API?It exposes a standard, well-known file interface backed by stateless metadata services and a transactional key-value store like FoundationDB, so most apps just use ordinary file calls; a separate USRBIO API is available for maximum throughput.
It exposes a standard, well-known file interface backed by stateless metadata services and a transactional key-value store like FoundationDB, so most apps just use ordinary file calls; a separate USRBIO API is available for maximum throughput.How do I get a test cluster running?Clone the repo with its submodules, install the required build prerequisites, build 3FS (optionally using the provided TencentOS-4 or OpenCloudOS-9 Docker build images), then follow the project's Setup Guide to bring up a test cluster.
Clone the repo with its submodules, install the required build prerequisites, build 3FS (optionally using the provided TencentOS-4 or OpenCloudOS-9 Docker build images), then follow the project's Setup Guide to bring up a test cluster.What do I need installed before building it?You need libfuse 3.16.1+, FoundationDB 7.1+, and a Rust toolchain (minimum 1.75.0, recommended 1.85.0+), on top of a working C++/CMake build environment.
You need libfuse 3.16.1+, FoundationDB 7.1+, and a Rust toolchain (minimum 1.75.0, recommended 1.85.0+), on top of a working C++/CMake build environment.Any build pitfalls I should know about?Binaries compiled with different GCC versions (g++10 vs g++11+) can be incompatible because of how std::shuffle behaves, so you must explicitly pin -DSHUFFLEMETHOD and keep using the same setting for the lifetime of a deployed cluster.
Binaries compiled with different GCC versions (g++10 vs g++11+) can be incompatible because of how std::shuffle behaves, so you must explicitly pin -DSHUFFLEMETHOD and keep using the same setting for the lifetime of a deployed cluster.Is this meant for a single machine or does it need a whole cluster?It's built for multi-node, data-center-scale deployments over InfiniBand/RDMA; DeepSeek's own benchmark used 180 storage nodes with dual 200Gbps NICs and NVMe SSDs plus 500+ client nodes, so it's not something you'd run on one laptop.
It's built for multi-node, data-center-scale deployments over InfiniBand/RDMA; DeepSeek's own benchmark used 180 storage nodes with dual 200Gbps NICs and NVMe SSDs plus 500+ client nodes, so it's not something you'd run on one laptop.How fast is 3FS in practice?On a 180-node test cluster it reached about 6.6 TiB/s aggregate read throughput, and a GraySort benchmark sorted 110.5 TiB of data across 8,192 partitions in 30 minutes 14 seconds.
On a 180-node test cluster it reached about 6.6 TiB/s aggregate read throughput, and a GraySort benchmark sorted 110.5 TiB of data across 8,192 partitions in 30 minutes 14 seconds.What is KVCache used for here?3FS acts as a cheaper, higher-capacity alternative to DRAM for caching key/value vectors during LLM inference, with client read throughput peaking around 40 GiB/s in DeepSeek's tests.
3FS acts as a cheaper, higher-capacity alternative to DRAM for caching key/value vectors during LLM inference, with client read throughput peaking around 40 GiB/s in DeepSeek's tests.Why pair RDMA networking with NVMe SSDs?Combining RDMA with NVMe lets 3FS disaggregate compute and storage nodes (CRAQ gives strong consistency) while still delivering near-local-disk throughput across the whole cluster, which is what large-scale training and inference pipelines need.
Combining RDMA with NVMe lets 3FS disaggregate compute and storage nodes (CRAQ gives strong consistency) while still delivering near-local-disk throughput across the whole cluster, which is what large-scale training and inference pipelines need.Which topics is it in?kv-cache, big-data, storage, infrastructure, high-performance, distributed-systems, cpp, c++, networking, rust, machine-learning, server
kv-cache, big-data, storage, infrastructure, high-performance, distributed-systems, cpp, c++, networking, rust, machine-learning, server