layerfs README

项目的 README,来自 github.com/Ephemeral-AI-Lab/layerfs。合并为一个 markdown 文件:README.md。

LayerFS

LayerFS

Ephemeral Workspaces. Durable Shared History.

LayerFS is the substrate under your agents. Each one mounts a complete, writable Workspace over shared committed history, does its work, commits what it changed, and goes away.

Get started · Website · Docs · Benchmarks · Video · LayerFS × DSH · Community

MIT license 1.1.0 Linux, macOS through DSH, and Windows through WSL2 on current main Discord

[!NOTE] LayerFS 2.0.0 development happens on the layerfs_v2_0_0 branch. Check there for the latest 2.0.0 work.

https://github.com/user-attachments/assets/70c25887-86a1-4d81-bd78-571cf1c09a18

Status: LayerFS 1.1.0, released as source: the JavaScript API split into @layerfs/agent, @layerfs/backend and @layerfs/runtime under api/, plus the layerfs command. It breaks code written for 1.0.0; see the release record. The results below are 1.0.0's, measured on main at e5c7aaae05. The Store runs in its Disposable profile only, with no crash or power-loss durability: use LayerFS for agent work and its history, not as the only copy of data you cannot rebuild.

Since 1.1.0, LayerFS also supports native Windows dsh Live through WSL2 and its own Linux Docker Engine; see the Windows setup.

Try it with LayerFS × DSH

LayerFS × DSH: the LayerFS logo and the DeepSeek Harness logo

LayerFS runs today through LayerFS × DSH, the dsh-layerfs plugin for DeepSeek Harness. Agents in a dsh session fork Branches, test them in Sandboxes, merge and publish, and the Work and History tabs show every agent's Branch, live.

Try LayerFS × DSH: install guide

Core ideas

fs calls go straight into the Store as Commits; each Bash call mounts a Workspace, runs, commits and unmounts
01 · Ephemeral Computing
Compute exists only while a command runs.
Four agents, each on its own computer, read and commit to one shared Store
02 · Decentralized Agents
Agents on different computers share one serverless Store.
Main with two Branches, three agents on each; a commit to a file that already changed is refused with a Conflict
03 · Multiagent Coordination
One project, many Branches. One Branch, many agents.
Agents fork instantly into new Branches, recursively; each Branch is scored and the best one is kept
04 · Recursive Filesystem
Fork recursively any Commit, at any depth.
Four agents commit the same packages; the first stores every chunk and each later Commit adds only what is new
05 · Ultra Storage Optimization
157 snapshots of a real project in 100 MB, 19.9× smaller than Btrfs + zstd.
Without ignored files the tests fail until packages are installed; with LayerFS the mount already has them
06 · Instant Bootstrap
Dependencies come with every mount. No time lost reinstalling.

And underneath them

Supporting idea In one line
07 · Auto Checkpoints Every step an agent takes is a checkpoint to return to.
08 · Append-only Objects Saved objects are never rewritten. The Store only grows.
09 · Conflict Rules (.lww) Name the files that should never conflict. The newest edit wins.
10 · Big File, Small Edit An edit costs the size of the edit, not the size of the file.

See it run

The LayerFS demo: agents on several Branches run tool calls in Sandboxes and fs, each call becomes a Commit, and the history tree grows

Agents fork Branches, work in Workspaces inside Sandboxes, commit, merge and publish to main. The history grows as they go.

Results

Every suite ran once on main at e5c7aaae05 through the JavaScript SDK. Each row links to its method, environment and receipts.

Suite What it measures LayerFS 1.0.0
fs-bench Cloudflare Computer's own twelve-scenario filesystem benchmark, unchanged, same image 5.20 s vs 8.88 s, 1.71× faster; faster in 10 of 12 scenarios
dsh-dedup A real project's history, 157 snapshots (4.94 GB of trees), kept as mountable Commits 100.1 MB vs 1,994.6 MB, 19.9× smaller than Btrfs + zstd; 156/156 read back
pjdfstest POSIX conformance, 8,798 assertions, against kernel overlayfs 0 unexpected failures; keeps 76% of overlayfs's passes, every other failure a deliberate format limit
fsx Random reads, writes, mapped I/O and truncates, every byte checked 14/14 pass, 3.06 M ops, 0 miscompares
fsstress Four concurrent processes, then Commit, remount and compare 3,755/3,755 entries identical; 5.21 GB captured by one Commit in 4.9 s
smallfile 100,000 files of 4 KiB, four threads, against libfuse passthrough Faster in 9 of 9 operations, 1.27×–10.71×
mdtest Metadata create, rename and remove, against libfuse passthrough 0.86–1.28× with 1 process, 1.21–4.22× with 4
fio Data I/O bandwidth, against libfuse passthrough Random read/write mixes 1.49–2.23×; slower on sequential write (0.31×), sequential read (0.81×) and random write (0.40×)

One sample per case, seed 1, no best-of. Host: arm64 Mac, Docker Desktop linux/arm64, 8 CPUs, 8.3 GB; host load is recorded per suite. These are release measurements, not a guarantee for your workload. Earlier fs-bench-pro results: v0.1.4 · v0.1.5 · v0.1.6.

How it works

agent tool call
 ├─ file tool (read, write, edit, grep …) ── SDK, in process ──────► Store
 └─ Bash ── mount ─► exec ─► commit ─► unmount                       ▲
             └─ Workspace: FUSE view in a Sandbox's daemon ───────────┘
                (committed root + its own overlay)

Prior art

LayerFS builds on well-known ideas: layered and union filesystems, copy-on-write snapshots, content-addressed storage and throwaway execution environments. What it adds is putting them together for agents: every tool call gets a full writable tree, and every result becomes a Commit on a Branch in one shared Store.

Prior work What it shares with LayerFS Where LayerFS differs
overlayfs, UnionFS, container image layers A read-only lower tree with a writable upper layer, which is also where the "layer" name comes from The upper layer is not just thrown away or kept as a directory: Commit turns it into an immutable snapshot on a Branch, with history, merge and Conflict. overlayfs is in the kernel and more complete; LayerFS's FUSE view deliberately omits FIFOs, sockets, device nodes, set-id bits and separate atime/ctime (pjdfstest)
Btrfs and ZFS snapshots, reflinks Cheap copy-on-write snapshots that share unchanged data Sharing is by content, not by extent, so a file that is rewritten in full still stores only its new chunks. On a small-file history of 157 snapshots that gives 19.9× less space than Btrfs + zstd. Btrfs and ZFS also stay cheap for in-place edits of large files, and they are durable, mature kernel filesystems; LayerFS is not (why Btrfs used more space here)
Git worktrees One repository, many working copies and branches A Workspace holds the whole tree, including .git, ignored files, dependencies and build output, so a new mount needs no reinstall. Commits happen per tool call, not when someone runs git commit
try Run a command against an overlay of your system, then inspect, apply or discard what it changed try works on one machine with one overlay at a time. LayerFS keeps every result as a Commit, and lets many agents branch from any of them and share one Store
Apiary and virtual layered file systems (Potter and Nieh, USENIX ATC 2010, LISA 2011) Ephemeral containers that start instantly over shared layered filesystems Apiary isolates desktop applications. LayerFS's target is agent tool calls whose results are kept, branched and merged as history

Use something else if you need crash or power-loss durability (the Store runs in its Disposable profile only), full POSIX node types, or top sequential I/O bandwidth (fio). For a few long-lived checkouts on one machine, Btrfs snapshots or Git worktrees may be all you need.

Learn more

Docs LayerFS × DSH guide · layerfs.ai/docs · Core ideas · JavaScript SDK guide
Concepts Core concepts · Documentation index
Benchmarks layerfs.ai/benchmarks
Release 1.1.0 release record · Previous: 1.0.0 release and its plan; 0.1.7 release record and its manual

Community

Questions, ideas and show-and-tell are welcome in either place.

Discord

Join the LayerFS Discord

discord.gg/8mmAMCDc2
WeChat · LayerFS的好朋友

QR code for the LayerFS WeChat group
Scan in WeChat to join the group.

License

LayerFS is licensed under the MIT License.

layerfs (Ephemeral-AI-Lab/layerfs)

正在搜索数据库…

2.8K ·GitHub
这是什么?LayerFS 是为 AI 智能体打造的文件系统:每个智能体都能获得一个完整、可写的工作区,所有工作区都挂载在同一份已提交的历史记录之上,且无需复制基础数据。因此智能体可以并行工作、为自己的修改设置检查点,之后还能分支、回退或合并。
这是我要挂载的文件系统,还是我在代码里调用的库?两者兼有:它是一个基于 FUSE 的文件系统守护进程,底层由单个 SQLite 存储支撑;同时还提供 JavaScript SDK 和 dsh 插件,让智能体能够挂载工作区,并以编程方式调用提交、分支和合并。
我该如何在智能体中开始使用 LayerFS?目前需要通过 DeepSeek Harness 的 dsh-layerfs 插件或 JavaScript SDK 来使用:dsh 会话中的智能体会派生分支、在沙箱中测试,然后合并并发布到 main 分支。
运行它需要什么?需要支持 FUSE 挂载的 Linux 环境,因为工作区挂载面向的是 Linux;基准测试显示,在 Mac 主机上只能在 Docker Desktop 的 linux/arm64 容器内使用。
存放无法重建的数据安全吗?不安全。目前存储只能在一次性配置(Disposable)下运行,不保证崩溃或断电后的持久性;1.0.0 是 0.1.x 预览版之后的第一个正式版本,因此它适合存放智能体的工作成果及其历史,而不适合存放不可替代的重要数据。
LayerFS 实际能节省多少存储空间?在一个真实项目的 157 个快照(共 4.94 GB 的目录树)上,LayerFS 将完整历史保存在约 100 MB 内,比使用 zstd 压缩的 Btrfs 小 19.9 倍,并且能正确读取全部 156 个快照。
.lww 冲突规则是什么?默认情况下,两个修改同一文件的提交会被拒绝,并报告为冲突;而 .lww 规则允许你指定某些文件,让这些文件始终以最新的编辑为准。
DSH 是什么?LayerFS 为什么经常提到它?DSH(DeepSeek Harness)是一个编程智能体的运行框架;dsh-layerfs 插件让 dsh 会话中的智能体派生分支、在沙箱中测试修改,并合并发布到 main 分支,这也是目前使用 LayerFS 的主要方式。
LayerFS 与 overlayfs 相比表现如何?在 fs-bench 测试套件中,LayerFS 在 Cloudflare 的十二个场景基准测试里比内核 overlayfs 快 1.71 倍;在 POSIX 一致性测试(pjdfstest)中,它没有出现意外失败,同时通过的测试数达到 overlayfs 的 76%。
大小
857.1 MB
星标
2.8K
时长
3个月
提交 · 90 天