pg_vectorsearch README

项目的 README,来自 github.com/timescale/pg_vectorsearch。合并为一个 markdown 文件:README.md。

pg_vectorsearch

Build Coverage Coverity

A PostgreSQL extension for high-performance approximate nearest neighbor (ANN) search.

Status: alpha. Meant for testing, not production. Upgrading between prereleases may not be possible; plan to drop and recreate the extension and its indexes. See the changelog for known issues.

PRISM Index

PRISM

PRISM (Partitioned Routing Index for Similarity Matching) is a PostgreSQL-native index access method that is conceptually similar to pgvector's IVFFlat, but it is built around state-of-the-art quantization, routing, and partitioning techniques. It is implemented in C for high performance and seamless integration with PostgreSQL.

The result: an index that is significantly faster than pgvector's IVFFlat and HNSW, for both querying and index building. In fact, compared to graph-based HNSW, PRISM is faster at equivalent recall on a 100-million-vector dataset, on an instance with a quarter of the memory.

To achieve this performance, PRISM uses a hierarchical centroid tree to partition vectors into clusters, and relies on the PostgreSQL buffer cache to keep that tree warm in memory. Vectors are RaBitQ-quantized and stored in posting lists with a page layout optimized for SIMD distance computation. The RaBitQ encoding carries error bounds that let a search rule out vectors that cannot possibly be nearest neighbors, so far fewer of them need reranking against the full-precision vectors stored in the table.

Query Flow:
1. Traverse centroid tree to find nearest clusters (buffer cache)
2. Scan posting lists for candidates (buffer cache, async I/O)
3. RaBitQ two-stage filtering: estimate → error bound → rerank
4. Full-precision reranking from heap
5. Return k nearest neighbors

For detailed architecture, see docs/architecture.md.

Usage

Install the extension first; see Building.

There are two ways to use a PRISM index. Pick the one that matches where your vectors live:

Both paths build the same index and use the same operators (<->, <#>, <=>) and tuning knobs.

Native types

CREATE EXTENSION pg_vectorsearch;

-- PLAIN keeps the vector inline. EXTERNAL (the default) toasts a value
-- over ~2 kB, and rerank then pays an extra fetch the planner does not cost.
CREATE TABLE items (
    id serial PRIMARY KEY,
    embedding vec32(3) STORAGE PLAIN
);

-- Load first; the index is sized from the row count at CREATE INDEX.
-- 500 rows is enough for the planner to use the index.
INSERT INTO items (embedding)
SELECT ARRAY[i::real, (i % 7)::real, (i % 5)::real]::vec32
FROM generate_series(1, 1000) i;

-- For inner product or cosine, see Operator classes below.
CREATE INDEX ON items USING prism (embedding vec32_l2_ops);

-- 0 = auto (~0.95 recall). Raise for recall, lower for speed.
SET prism.nprobe = 40;

SELECT * FROM items
ORDER BY embedding <-> '[3,1,2]'
LIMIT 5;

Use vec16 instead of vec32 to halve the space vectors take in the table, with the matching vec16_* operator classes.

Existing pgvector tables

If you already have a table with pgvector's vector or halfvec columns, you can build a PRISM index on it directly. Nothing about the table or your queries needs to change. vec32 and vec16 are binary-compatible with pgvector's vector and halfvec but have different names, so the two extensions can be installed side by side, even in the same schema.

-- The table you already have, included so the example runs on its own.
CREATE EXTENSION vector;

CREATE TABLE documents (
    id        bigserial PRIMARY KEY,
    body      text,
    embedding vector(384)
);
CREATE INDEX documents_embedding_hnsw
    ON documents USING hnsw (embedding vector_l2_ops);

-- Install pg_vectorsearch next to pgvector and build the index on the
-- existing column. The column is not rewritten or copied.
CREATE EXTENSION pg_vectorsearch;

CREATE INDEX documents_embedding_prism
    ON documents USING prism (embedding vec32_l2_ops);

-- Your queries keep working as before. PRISM is normally faster, so
-- the planner should prefer it over the HNSW index while both exist.
-- Verify with an EXPLAIN:
EXPLAIN (COSTS OFF)
SELECT id, body
FROM documents
ORDER BY embedding <-> (SELECT embedding FROM documents WHERE id = 42)
LIMIT 10;
--  Limit
--    InitPlan 1
--      ->  Index Scan using documents_pkey on documents documents_1
--            Index Cond: (id = 42)
--    ->  Index Scan using documents_embedding_prism on documents
--          Order By: (embedding <-> (InitPlan 1).col1)

-- Once you are happy with it, drop the HNSW index.
DROP INDEX documents_embedding_hnsw;

Use the PRISM operator class for the same distance metric as your HNSW index.

Operator classes

Pick the operator class for your distance metric and column type. L2 is the default.

Distance Operator vec32 / vector vec16 / halfvec
L2 <-> vec32_l2_ops vec16_l2_ops
Inner product <#> vec32_ip_ops vec16_ip_ops
Cosine <=> vec32_cosine_ops vec16_cosine_ops

When replacing an HNSW index, pgvector's vector_<metric>_ops maps to vec32_<metric>_ops and halfvec_<metric>_ops to vec16_<metric>_ops.

Dimensions

Any dimension up to 1968 is supported. Above that the dimension must be a power of two times an odd factor of at most 8, so 2048, 3072, 4096, 6144 and 8192 work and 8192 is the ceiling. Past 1968 each posting list keeps its full-precision encode reference on a short chain of reference pages ahead of its entries (three pages at 4096), and the index reads those pages on every probe; the exact rerank also fetches wider heap rows, so a vec16 column halves what it reads. Inner product routes on uncompressed centroids, which fit a page up to 2036 dimensions as vec32 and 4072 as vec16; use L2 or cosine beyond that. Splitting a wide list needs maintenance_work_mem of a few hundred megabytes, which prism_rebalance reports when it is short.

Tuning

Defaults are tuned from large-scale benchmarks; most deployments only ever adjust prism.nprobe. See the tuning guide for every index parameter and GUC, their tradeoffs, and when changing them makes sense.

Maintenance

An insert appends to whichever posting list its vector routes to, so lists grow as rows arrive and never split on their own. Besides REINDEX, the prism_rebalance procedure splits oversized lists in place. See the maintenance guide for how it works and its current limitations.

Building

Requirements:

# Release build
meson setup builddir --buildtype=release

# Build and install to PostgreSQL
meson install -C builddir

See the development guide for build options.

Documentation

References

Research

License

The PostgreSQL License

pg_vectorsearch (timescale/pg_vectorsearch)

正在搜索数据库…

这是什么?pgvectorsearch 是一个 PostgreSQL 扩展,新增了 PRISM 索引,这是一种面向向量数据的高性能近似最近邻(ANN)搜索方法,设计目标是在大规模数据下超越 pgvector 的 IVFFlat 和 HNSW 索引。
这是一个独立的数据库,还是我可以添加到 PostgreSQL 中的扩展?它是一个用 C 语言编写的 PostgreSQL 扩展,新增了一种索引访问方法(PRISM)以及原生的向量列类型,因此它运行在你现有的 PostgreSQL 18+ 服务器内部,而不是独立的系统。
我该如何安装并开始使用?使用 Meson 针对你的 PostgreSQL 实例进行构建和安装,然后可以选择将向量存储在 pgvectorsearch 自带的 vec32/vec16 列类型中,或者直接在现有的 pgvector 'vector'/'halfvec' 列上构建 PRISM 索引,无需修改表或查询。
编译它需要什么?你需要带有开发头文件的 PostgreSQL 18+、支持 GNU C11 的编译器(GCC 11+ 或 Clang 14+),以及 Meson 构建系统;CBLAS 是可选依赖,用于加快 RaBitQ 编码速度。
现在可以在生产环境中安全使用吗?不可以。项目明确标注为 alpha 版本,仅供测试使用;预发布版本之间升级可能无法正常工作,因此你需要做好删除并重新创建扩展及其索引的准备。
PRISM 索引到底是什么?PRISM(Partitioned Routing Index for Similarity Matching,用于相似度匹配的分区路由索引)是一种 IVF 风格的索引,它使用分层质心树将向量路由到各个聚类中,并让该树常驻于 PostgreSQL 的缓冲区缓存中,以实现快速查找。
什么是 RaBitQ 量化?它为什么重要?RaBitQ 是一种具有理论误差界的二值量化方案。它使 PRISM 能够在使用全精度数据进行重排序之前,排除那些不可能是最近邻的向量,从而大幅减少代价高昂的比较次数。
PRISM 与 pgvector 的 HNSW 索引相比如何?README 中称,在召回率相当的情况下,PRISM 在包含一亿个向量的数据集上比 HNSW 更快,且占用的内存约为其四分之一;这是因为它采用分区倒排列表,而不是 HNSW 的图结构。
随着时间推移不断插入数据,索引会发生什么变化?新向量会直接追加到它们被路由到的倒排列表中,因此列表会不断增长且不会自动拆分;你需要定期运行 prismrebalance 过程(或 REINDEX)来拆分过大的列表。
有哪些类似的项目?pgvector, weaviate, faiss, lancedb, timescaledb, milvus, kividb, qdrant
作者
timescale
大小
4 MB
星标
443
时长
9个月
提交 · 90 天