3FS README

The project's README from github.com/deepseek-ai/3FS. As one markdown file: README.md.

Fire-Flyer File System

Build License

The Fire-Flyer File System (3FS) is a high-performance distributed file system designed to address the challenges of AI training and inference workloads. It leverages modern SSDs and RDMA networks to provide a shared storage layer that simplifies development of distributed applications. Key features and benefits of 3FS include:

Documentation

Performance

1. Peak throughput

The following figure demonstrates the throughput of read stress test on a large 3FS cluster. This cluster consists of 180 storage nodes, each equipped with 2×200Gbps InfiniBand NICs and sixteen 14TiB NVMe SSDs. Approximately 500+ client nodes were used for the read stress test, with each client node configured with 1x200Gbps InfiniBand NIC. The final aggregate read throughput reached approximately 6.6 TiB/s with background traffic from training jobs.

Large block read throughput under stress test on a 180-node cluster

To benchmark 3FS, please use our fio engine for USRBIO.

2. GraySort

We evaluated smallpond using the GraySort benchmark, which measures sort performance on large-scale datasets. Our implementation adopts a two-phase approach: (1) partitioning data via shuffle using the prefix bits of keys, and (2) in-partition sorting. Both phases read/write data from/to 3FS.

The test cluster comprised 25 storage nodes (2 NUMA domains/node, 1 storage service/NUMA, 2×400Gbps NICs/node) and 50 compute nodes (2 NUMA domains, 192 physical cores, 2.2 TiB RAM, and 1×200 Gbps NIC/node). Sorting 110.5 TiB of data across 8,192 partitions completed in 30 minutes and 14 seconds, achieving an average throughput of 3.66 TiB/min.

3. KVCache

KVCache is a technique used to optimize the LLM inference process. It avoids redundant computations by caching the key and value vectors of previous tokens in the decoder layers. The top figure demonstrates the read throughput of all KVCache clients (1×400Gbps NIC/node), highlighting both peak and average values, with peak throughput reaching up to 40 GiB/s. The bottom figure presents the IOPS of removing ops from garbage collection (GC) during the same time period.

KVCache Read Throughput KVCache GC IOPS

Check out source code

Clone 3FS repository from GitHub:

git clone https://github.com/deepseek-ai/3fs

When deepseek-ai/3fs has been cloned to a local file system, run the following commands to check out the submodules:

cd 3fs
git submodule update --init --recursive
./patches/apply.sh

Install dependencies

Install dependencies:

# for Ubuntu 20.04.
apt install cmake libuv1-dev liblz4-dev liblzma-dev libdouble-conversion-dev libdwarf-dev libunwind-dev \
  libaio-dev libgflags-dev libgoogle-glog-dev libgtest-dev libgmock-dev clang-format-14 clang-14 clang-tidy-14 lld-14 \
  libgoogle-perftools-dev google-perftools libssl-dev libclang-rt-14-dev gcc-10 g++-10 libboost1.71-all-dev build-essential

# for Ubuntu 22.04.
apt install cmake libuv1-dev liblz4-dev liblzma-dev libdouble-conversion-dev libdwarf-dev libunwind-dev \
  libaio-dev libgflags-dev libgoogle-glog-dev libgtest-dev libgmock-dev clang-format-14 clang-14 clang-tidy-14 lld-14 \
  libgoogle-perftools-dev google-perftools libssl-dev gcc-12 g++-12 libboost-all-dev build-essential

# for openEuler 2403sp1
yum install cmake libuv-devel lz4-devel xz-devel double-conversion-devel libdwarf-devel libunwind-devel \
    libaio-devel gflags-devel glog-devel gtest-devel gmock-devel clang-tools-extra clang lld \
    gperftools-devel gperftools openssl-devel gcc gcc-c++ boost-devel

# for OpenCloudOS 9 and TencentOS 4
dnf install epol-release wget git meson cmake perl lld gcc gcc-c++ autoconf lz4 lz4-devel xz xz-devel \
    double-conversion-devel libdwarf-devel libunwind-devel libaio-devel gflags-devel glog-devel \
    libuv-devel gmock-devel gperftools gperftools-devel openssl-devel boost-static boost-devel mono-devel \
    libevent-devel libibverbs-devel numactl-devel python3-devel

Install other build prerequisites:

Build 3FS

Build 3FS in build folder:

# Replace <method> with 'g++10' or 'g++11' based on your environment
cmake -S . -B build \
      -DCMAKE_CXX_COMPILER=clang++-14 -DCMAKE_C_COMPILER=clang-14 \
      -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
      -DSHUFFLE_METHOD=<method>
cmake --build build -j 32

Due to the historical use of std::shuffle, binaries compiled with different compiler versions (e.g., g++10 vs. g++11 +) may be incompatible (issue). To resolve this, you must explicitly specify -DSHUFFLE_METHOD during compilation to lock in a consistent shuffle algorithm:

Build 3FS use Docker

Run a test cluster

Follow instructions in setup guide to run a test cluster.

Report Issues

Please visit https://github.com/deepseek-ai/3fs/issues to report issues.

3FS (deepseek-ai/3FS)

Searching Database…

10.3K ·GitHub
3FS
这是什么？3FS（Fire-Flyer File System）是 DeepSeek 开发的高性能分布式文件系统，用于大规模处理 AI 训练和推理的存储负载。它通过 RDMA 网络把数千块 SSD 和数百个存储节点整合成一个共享存储层。
我是像挂载普通文件系统一样使用它，还是要用专门的 API？它对外提供标准、通用的文件接口，底层依赖无状态的元数据服务和 FoundationDB 这类支持事务的键值存储，因此大多数应用直接使用普通文件操作即可；如果需要极致吞吐量，还可以使用单独的 USRBIO API。
我怎么把测试集群跑起来？克隆仓库时同时拉取子模块，安装所需的编译前置依赖，然后编译 3FS（可选择使用项目提供的 TencentOS-4 或 OpenCloudOS-9 Docker 构建镜像），最后按照项目中的 Setup Guide 搭建测试集群。
编译之前需要先装什么？除了 C++/CMake 编译环境，还需要 libfuse 3.16.1+、FoundationDB 7.1+，以及 Rust 工具链（最低 1.75.0，推荐 1.85.0+）。
编译时有哪些坑需要注意？由于 std::shuffle 的行为差异，用不同 GCC 版本（g++10 与 g++11+）编译出的二进制文件可能互不兼容，因此必须显式指定 -DSHUFFLEMETHOD，并且在已部署集群的整个生命周期内始终使用同一设置。
这是给单机用的，还是需要整套集群？它是为多节点、数据中心规模的部署设计的，运行在 InfiniBand/RDMA 网络之上。DeepSeek 自己的基准测试使用了 180 个存储节点（每个配双 200Gbps 网卡和 NVMe SSD）以及 500 多个客户端节点，所以它不是能在一台笔记本上跑起来的东西。
3FS 实际性能如何？在一个 180 节点的测试集群上，它的总读取吞吐量达到约 6.6 TiB/s。在 GraySort 基准测试中，它在 30 分 14 秒内对 8192 个分区中的 110.5 TiB 数据完成了排序。
KVCache 在这里起什么作用？在 LLM 推理过程中，3FS 可以作为 DRAM 的一种更便宜、容量更大的替代方案，用于缓存键/值向量。在 DeepSeek 的测试中，客户端读取吞吐量峰值约为 40 GiB/s。
为什么要把 RDMA 网络和 NVMe SSD 搭配使用？将 RDMA 与 NVMe 结合后，3FS 可以把计算节点和存储节点解耦（CRAQ 提供强一致性），同时在整个集群范围内仍能提供接近本地磁盘的吞吐性能，这正是大规模训练和推理流水线所需要的。
What are similar projects?ceph, scylladb, hadoop, redpanda, nebula, kafka, tikv, seaweedfs
Which topics is it in?kv-cache, big-data, storage, infrastructure, high-performance, distributed-systems, cpp, c++, networking, rust, machine-learning, server
Creator
deepseek-ai
Size
6 MB
Stars
10.3K
Updated
CommitMay 07
Age
1y
Commits · 90 Days