FlagEmbedding README

The project's README from github.com/FlagOpen/FlagEmbedding. As one markdown file: README.md.

⚡️BGE: One-Stop Retrieval Toolkit For Search and RAG

bge_logo

Build Build License Build Build

News | Installation | Quick Start | Community | Projects | Model List | Contributor | Citation | License

English | 中文

News

More
  • 6/7/2024: Release a new benchmark MLVU, the first comprehensive benchmark specifically designed for long video understanding. MLVU features an extensive range of video durations, a diverse collection of video sources, and a set of evaluation tasks uniquely tailored for long-form video understanding. :fire:
  • 5/21/2024: Release a new benchmark AIR-Bench together with Jina AI, Zilliz, HuggingFace, and other partners. AIR-Bench focuses on a fair out-of-distribution evaluation for Neural IR & RAG. It generates the synthetic data for benchmarking w.r.t. diverse domains and languages. It is dynamic and will be updated on regular basis. Leaderboard :fire:
  • 4/30/2024: Release Llama-3-8B-Instruct-80K-QLoRA, extending the context length of Llama-3-8B-Instruct from 8K to 80K via QLoRA training on a few synthesized long-context data. The model achieves remarkable performance on various long-context benchmarks. Code :fire:
  • 3/18/2024: Release new rerankers, built upon powerful M3 and LLM (GEMMA and MiniCPM, not so large actually :smiley:) backbones, supporitng multi-lingual processing and larger inputs, massive improvements of ranking performances on BEIR, C-MTEB/Retrieval, MIRACL, LlamaIndex Evaluation :fire:
  • 3/18/2024: Release Visualized-BGE, equipping BGE with visual capabilities. Visualized-BGE can be utilized to generate embeddings for hybrid image-text data. :fire:
  • 1/30/2024: Release BGE-M3, a new member to BGE model series! M3 stands for Multi-linguality (100+ languages), Multi-granularities (input length up to 8192), Multi-Functionality (unification of dense, lexical, multi-vec/colbert retrieval). It is the first embedding model which supports all three retrieval methods, achieving new SOTA on multi-lingual (MIRACL) and cross-lingual (MKQA) benchmarks. Technical Report and Code. :fire:
  • 1/9/2024: Release Activation-Beacon, an effective, efficient, compatible, and low-cost (training) method to extend the context length of LLM. Technical Report
  • 12/24/2023: Release LLaRA, a LLaMA-7B based dense retriever, leading to state-of-the-art performances on MS MARCO and BEIR. Model and code will be open-sourced. Please stay tuned. Technical Report and Code
  • 11/23/2023: Release LM-Cocktail, a method to maintain general capabilities during fine-tuning by merging multiple language models. Technical Report
  • 10/12/2023: Release LLM-Embedder, a unified embedding model to support diverse retrieval augmentation needs for LLMs. Technical Report
  • 09/15/2023: The technical report of BGE has been released
  • 09/15/2023: The massive training data of BGE has been released
  • 09/12/2023: New models:
    • New reranker model: release cross-encoder models BAAI/bge-reranker-base and BAAI/bge-reranker-large, which are more powerful than embedding model. We recommend to use/fine-tune them to re-rank top-k documents returned by embedding models.
    • update embedding model: release bge-*-v1.5 embedding model to alleviate the issue of the similarity distribution, and enhance its retrieval ability without instruction.
  • 09/07/2023: Update fine-tune code: Add script to mine hard negatives and support adding instruction during fine-tuning.
  • 08/09/2023: BGE Models are integrated into Langchain, you can use it like this; C-MTEB leaderboard is available.
  • 08/05/2023: Release base-scale and small-scale models, best performance among the models of the same size 🤗
  • 08/02/2023: Release bge-large-*(short for BAAI General Embedding) Models, rank 1st on MTEB and C-MTEB benchmark! :tada: :tada:
  • 08/01/2023: We release the Chinese Massive Text Embedding Benchmark (C-MTEB), consisting of 31 test dataset.

BGE (BAAI General Embedding) focuses on retrieval-augmented LLMs, consisting of the following projects currently:

projects

Installation

Using pip:

If you do not want to finetune the models, you can install the package without the finetune dependency:

pip install -U FlagEmbedding

If you want to finetune the models, you can install the package with the finetune dependency:

pip install -U FlagEmbedding[finetune]

Install from sources:

Clone the repository and install

git clone https://github.com/FlagOpen/FlagEmbedding.git
cd FlagEmbedding
# If you do not need to finetune the models, you can install the package without the finetune dependency:
pip install  .
# If you want to finetune the models, install the package with the finetune dependency:
# pip install  .[finetune]

For development in editable mode:

# If you do not need to finetune the models, you can install the package without the finetune dependency:
pip install -e .
# If you want to finetune the models, install the package with the finetune dependency:
# pip install -e .[finetune]

Quick Start

First, load one of the BGE embedding model:

from FlagEmbedding import FlagAutoModel

model = FlagAutoModel.from_finetuned('BAAI/bge-base-en-v1.5',
                                      query_instruction_for_retrieval="Represent this sentence for searching relevant passages:",
                                      use_fp16=True)

Then, feed some sentences to the model and get their embeddings:

sentences_1 = ["I love NLP", "I love machine learning"]
sentences_2 = ["I love BGE", "I love text retrieval"]
embeddings_1 = model.encode(sentences_1)
embeddings_2 = model.encode(sentences_2)

Once we get the embeddings, we can compute similarity by inner product:

similarity = embeddings_1 @ embeddings_2.T
print(similarity)

For more details, you can refer to embedder inference, reranker inference, embedder finetune, reranker fintune, evaluation.

If you're unfamiliar with any of related concepts, please check out the tutorial. If it's not there, let us know.

For more interesting topics related to BGE, take a look at research.

Community

We are actively maintaining the community of BGE and FlagEmbedding. Let us know if you have any suggessions or ideas!

Currently we are updating the tutorials, we aim to create a comprehensive and detailed tutorial for beginners on text retrieval and RAG. Stay tuned!

The following contents are releasing in the upcoming weeks:

The whole tutorial roadmap

Model List

bge is short for BAAI general embedding.

Model Language Description query instruction for retrieval
BAAI/bge-en-icl English A LLM-based embedding model with in-context learning capabilities, which can fully leverage the model's potential based on a few shot examples Provide instructions and few-shot examples freely based on the given task.
BAAI/bge-multilingual-gemma2 Multilingual A LLM-based multilingual embedding model, trained on a diverse range of languages and tasks. Provide instructions based on the given task.
BAAI/bge-m3 Multilingual Multi-Functionality(dense retrieval, sparse retrieval, multi-vector(colbert)), Multi-Linguality, and Multi-Granularity(8192 tokens)
LM-Cocktail English fine-tuned models (Llama and BGE) which can be used to reproduce the results of LM-Cocktail
BAAI/llm-embedder English a unified embedding model to support diverse retrieval augmentation needs for LLMs See README
BAAI/bge-reranker-v2-m3 Multilingual a lightweight cross-encoder model, possesses strong multilingual capabilities, easy to deploy, with fast inference.
BAAI/bge-reranker-v2-gemma Multilingual a cross-encoder model which is suitable for multilingual contexts, performs well in both English proficiency and multilingual capabilities.
BAAI/bge-reranker-v2-minicpm-layerwise Multilingual a cross-encoder model which is suitable for multilingual contexts, performs well in both English and Chinese proficiency, allows freedom to select layers for output, facilitating accelerated inference.
BAAI/bge-reranker-v2.5-gemma2-lightweight Multilingual a cross-encoder model which is suitable for multilingual contexts, performs well in both English and Chinese proficiency, allows freedom to select layers, compress ratio and compress layers for output, facilitating accelerated inference.
BAAI/bge-reranker-large Chinese and English a cross-encoder model which is more accurate but less efficient
BAAI/bge-reranker-base Chinese and English a cross-encoder model which is more accurate but less efficient
BAAI/bge-large-en-v1.5 English version 1.5 with more reasonable similarity distribution Represent this sentence for searching relevant passages:
BAAI/bge-base-en-v1.5 English version 1.5 with more reasonable similarity distribution Represent this sentence for searching relevant passages:
BAAI/bge-small-en-v1.5 English version 1.5 with more reasonable similarity distribution Represent this sentence for searching relevant passages:
BAAI/bge-large-zh-v1.5 Chinese version 1.5 with more reasonable similarity distribution 为这个句子生成表示以用于检索相关文章：
BAAI/bge-base-zh-v1.5 Chinese version 1.5 with more reasonable similarity distribution 为这个句子生成表示以用于检索相关文章：
BAAI/bge-small-zh-v1.5 Chinese version 1.5 with more reasonable similarity distribution 为这个句子生成表示以用于检索相关文章：
BAAI/bge-large-en English Embedding Model which map text into vector Represent this sentence for searching relevant passages:
BAAI/bge-base-en English a base-scale model but with similar ability to bge-large-en Represent this sentence for searching relevant passages:
BAAI/bge-small-en English a small-scale model but with competitive performance Represent this sentence for searching relevant passages:
BAAI/bge-large-zh Chines

FlagEmbedding (FlagOpen/FlagEmbedding)

Searching Database…

12.3K ·GitHub
FlagEmbedding
这是什么？FlagEmbedding（BGE）是 BAAI 推出的一站式工具包，面向检索以及检索增强的大语言模型，提供嵌入模型、重排序模型、微调脚本和评测工具，用于搜索和 RAG（检索增强生成）。
这是模型合集、Python 库，还是两者都是？两者都是：它是一个可以通过 pip 安装的 Python 包，封装了推理和微调代码；同时还有一系列预训练的嵌入模型和重排序模型，托管在 Hugging Face 上（BAAI 组织下）。
我该怎么开始使用？可以用 pip 安装（可选是否带上 finetune 依赖），也可以克隆代码仓库后以可编辑模式从源码安装；然后加载一个 BGE 嵌入模型，把句子输入进去，就能得到嵌入向量。
除了 Python，使用它还需要别的东西吗？只需要 Python 和 pip；模型从 Hugging Face（BAAI 仓库）下载。只有当你打算微调自己的嵌入模型或重排序模型时，才需要安装可选的 'finetune' 依赖。
列表里到底该选哪个模型？模型变体很多，而且彼此有重叠（v1、v1.5、M3、ICL、基于 gemma2 的版本），它们支持的语言和取舍各不相同（精度与速度之间的权衡），因此选对模型需要仔细查看模型表，并不一目了然。
除了英语和中文，它还支持其他语言吗？支持。BGE-M3 和 bge-multilingual-gemma2 等模型明确是多语言的，支持 100 多种语言；另外还有专门针对英语和中文的 v1.5 模型。
BGE-M3 相比普通嵌入模型多了什么？BGE-M3 在一个模型中统一了三种检索方式：稠密检索、稀疏（词汇）检索，以及类似 ColBERT 的多向量检索；它支持 100 多种语言，并且可以处理最长 8192 个 token 的输入。
重排序模型和嵌入模型各自有什么作用？重排序模型是一种交叉编码器（cross-encoder），它会对嵌入模型已经检索出的前 k 个文档重新打分。相比只看嵌入相似度，它通常结果更准确，但速度更慢。
BGE 能和 LangChain 一起用吗？可以。README 中提到，BGE 模型已经集成到 LangChain 中，因此可以在该框架里直接作为嵌入提供方使用。
什么是 Visualized-BGE？Visualized-BGE 为 BGE 增加了视觉能力，使其能够为图文混合数据生成嵌入，从而支持多模态搜索。
What are similar projects?txtai, LLMSearchRecommender, fastembed-rs, sentence-transformers, SkillCorpus, memvid, PageIndex, MSA
Which topics is it in?information-retrieval, retrieval-augmented-generation, vector-search, semantic-search, embeddings, fine-tuning, transformer, multilingual, nlp, rag, huggingface, benchmark
Creator
FlagOpen
Size
52.2 MB
Stars
12.3K
Updated
CommitAug 24Releasev1.4.2Aug 24
Age
3y
Commits · 90 Days