learned_index_lab README

项目的 README,来自 github.com/gilnett/learned_index_lab。合并为一个 markdown 文件:README.md。

Learned Index Lab: Modern Search Paradigms in Rust

CI Crates.io Docs.rs License: Apache-2.0 Buy Me a Coffee GitHub Sponsors

A clean, zero-dependency Rust implementation and comparative benchmark suite of Piecewise Geometric Model (PGM) Learned Index Structures.

This repository evaluates mathematical function approximations of Empirical Cumulative Distribution Functions (CDFs) against standard pointer trees (BTreeMap), contiguous binary search (slice::binary_search), and cache-conscious breadth-first search (EytzingerSearch).

[!NOTE] Academic Attribution: Learned Indexes were originated by Kraska et al. (MIT/Google, SIGMOD 2018). The PGM-index was invented and proven by Paolo Ferragina and Giorgio Vinciguerra (University of Pisa, PVLDB 2020). The Eytzinger layout was formalized by Paul-Virak Khuong and Pat Morin (ACM JEA 2017). This repository is an independent, clean-room implementation in pure safe Rust.


Architectural Comparison

Structure Layout Lookup Cache Locality Index Metadata (1M keys) Total Process Memory
Flat Slice Contiguous slice Binary search ($O(\log N)$) Medium (stride jumps across RAM) 0.00 KB 7.63 MB
Eytzinger 1-based array Branchless traversal High (root levels L1-pinned) 7.63 MB 15.26 MB
PgmIndex Linear segments ALU projection + local scan High (when segments <= L2 cache) 3.1 KB - 1.50 MB 7.63 MB - 9.13 MB
MultilevelPgmIndex Recursive PGM tree Top-down arithmetic ($O(h \log \epsilon)$) High (bounded descent) 2.0 KB - 1.32 MB 7.63 MB - 8.95 MB
BTreeMap Heap nodes Pointer hops ($O(\log N)$) Low (scattered heap nodes) ~45.78 MB (embedded) 45.78 MB

Benchmark Summary (v0.4.0 Consolidated Results)

Measured on x86_64 hardware in --release mode. Consolidated average across all 6 workload distributions (30 independent runs per target structure with a 32MB L1-L3 CPU cache eviction barrier between phases):

Cible / Target Build Time Hit Latency Miss Latency Lower Bound Index Metadata Moyenne Totale Latence Speedup vs BTree
BTreeMap 52.08 ms 178.8 ns 181.3 ns 180.7 ns 45.78 MB 180.3 ns 1.00x
Flat Slice 0.00 ms 140.9 ns 113.4 ns 122.4 ns 0.00 KB 125.6 ns 1.27x
Eytzinger 5.37 ms 122.0 ns 116.2 ns 121.1 ns 7.63 MB 119.7 ns 1.47x
PGM-Index 4.78 ms 115.5 ns 107.2 ns 146.7 ns 330.6 KB 123.1 ns 1.55x
Multilevel PGM 4.61 ms 149.7 ns 141.7 ns 206.1 ns 274.5 KB 165.8 ns 1.19x

[!NOTE] Trade-Off Summary Across Real-World & Synthetic Workloads:

  • PGM-Index is fastest on smooth/continuous distributions (up to 2.09x vs BTree), reducing metadata by up to 14,700x (~5.7x total resident RAM).
  • BTreeMap wins decisively on irregular, heavily fragmented, or extreme Pareto power-law distributions (e.g. SOSD Books: 2.87x faster than PGM), where non-linearity degrades linear segment count.
  • Eytzinger delivers the lowest and most consistent overall average latency (119.7 ns) without mathematical approximation error.

Detailed breakdown per distribution (Synthetic and SOSD real-world benchmarks) is available in docs/benchmarks/statistical_benchmarks.md.


Reproducibility & Verification

The v0.4.0 test suite and benchmark harness support strict single-threaded isolation, SOSD real-world suites, and modular target selection:

# Clone the repository
git clone https://github.com/gilnett/learned_index_lab.git
cd learned_index_lab

# Run the unit test suite with thread isolation
cargo test --release -- --test-threads=1

# Execute the full 10-run release benchmark across all 5 targets
cargo run --release

# Run isolated benchmarks on specific targets (including Multilevel PGM)
cargo run --release -- --target multilevel,pgm --runs 10
cargo run --release -- --target pgm,multilevel --suite sosd

# Run the batch ingestion and rebuild benchmark (SSTable merge vs BTree in-place)
cargo run --release -- --batch --batch-size 100000

Documentation Index


References

Technical Articles & Deep Dives

Seminal Research Papers

Reference Codebases & Implementations

Industry Storage Engines & Analytical Systems


License

Licensed under the Apache License, Version 2.0 (LICENSE or http://www.apache.org/licenses/LICENSE-2.0).

learned_index_lab (gilnett/learned_index_lab)

正在搜索数据库…

这是什么?这是一个纯 Rust 实现、零依赖的 PGM 索引(Piecewise Geometric Model learned index,分段几何模型学习型索引),并附带一套基准测试,将它与标准的 BTreeMap、扁平切片二分查找以及对缓存友好的 Eytzinger 布局进行对比。
这是可以在我的代码里直接使用的库,还是只是一份基准测试报告?两者都是:仓库里既有一个可以实际使用的 PGM 索引实现,也有一套完整的对比基准测试框架,用来将它与其他三种搜索结构进行比较。
我该如何自己运行这些基准测试?克隆仓库,然后以 release 模式构建并运行,因为所有公布的数据都是在目标硬件上使用 --release 测得的。
使用它需要安装任何外部 crate 吗?不需要,它明确设计为零依赖的纯 Rust 实现。
这是原作者提供的官方 PGM 索引实现吗?不是。它是一个独立的、清洁室(clean-room)式重新实现,使用安全 Rust 编写,与意大利比萨大学团队提供的官方 C++ 参考实现相互独立。
与 BTreeMap 相比,PGM 索引能节省多少内存?对于一百万个键,PGM 索引大约占用 21.1 KB,而 BTreeMap 大约需要 45.78 MB,减少幅度超过 99.9%。
它的查找速度比普通的 BTreeMap 快多少?在 v0.3.0 的综合基准测试中,PGM 索引的查找速度约为 BTreeMap 的 1.93 倍,Eytzinger 布局达到 2.02 倍,而扁平二分查找仅为 1.20 倍。
这个基准测试实际测试了哪些键分布?三种工作负载类型:Dense Time-Series(密集时间序列)、Uniform Sparse(均匀稀疏)和 Heavy-Tailed Lognormal(重尾对数正态),每种各独立运行 30 次,各阶段之间设有 32MB 的缓存清除屏障。
最初是谁提出了 PGM 索引的想法?学习型索引由 Kraska 等人提出(MIT/Google,SIGMOD 2018),而 PGM 索引本身由比萨大学的 Paolo Ferragina 和 Giorgio Vinciguerra 发明(PVLDB 2020)。
有哪些类似的项目?bbtree, goose, ANE, ir, spinel, explaindb, kividb, conc
作者
gilnett
大小
0.2 MB
星标
166
时长
1个月
提交 · 90 天