LLMSurvey README

The project's README from github.com/RUCAIBox/LLMSurvey. As one markdown file: README.md.

LLMSurvey

A collection of papers and resources related to Large Language Models.

The organization of papers refers to our survey "A Survey of Large Language Models". Paper page

Please let us know if you find out a mistake or have any suggestions by e-mail: [email protected]

(we suggest ccing another email [email protected] meanwhile, in case of any unsuccessful delivery issue.)

If you find our survey useful for your research, please cite the following paper:

@article{LLMSurvey,
    title={A Survey of Large Language Models},
    author={Zhao, Wayne Xin and Zhou, Kun and Li, Junyi and Tang, Tianyi and Wang, Xiaolei and Hou, Yupeng and Min, Yingqian and Zhang, Beichen and Zhang, Junjie and Dong, Zican and Du, Yifan and Yang, Chen and Chen, Yushuo and Chen, Zhipeng and Jiang, Jinhao and Ren, Ruiyang and Li, Yifan and Tang, Xinyu and Liu, Zikang and Liu, Peiyu and Nie, Jian-Yun and Wen, Ji-Rong},
    year={2023},
    journal={arXiv preprint arXiv:2303.18223},
    url={http://arxiv.org/abs/2303.18223}
}

🚀(New) We have released the Chinese book of our survey!

The Chinese book focuses on providing explanations for beginners in the field of LLMs, aiming to present a comprehensive framework and roadmap for LLMs. This book is suitable for senior undergraduate students and junior graduate students with a foundation in deep learning and can serve as an introductory technical book. You can download the Chinese book at https://llmbook-zh.github.io/.

Here is our Chinese book sales page.

chinese_version

🚀(New) The content about long CoT reasoning

In our latest version, we add new content of the recent popular reasoning paradigm by allocating more time to thinking before responding to a problem. We focus on long CoT reasoning which is the mainstream approach taken by recent LLMs, such as DeepSeek-R1 and OpenAI's o-series models. We first discuss the reasoning patterns and advantages of the long CoT paradigm. Then we present the construction approaches of long CoT data, including data distillation, search-based data synthesis, and multi-agent collaboration. Moreover, we introduce the commonly-used two training methods: long CoT instruction tuning and scaling reinforcement learning training. Finally, we conduct a in-depth discussion about recent test-time scaling efforts for LLMs.

Cover

Here are the trends of the cumulative numbers of arXiv papers that contain the keyphrases “language model” (since June 2018) and “large language model” (since October 2019), respectively.

arxiv_llms

The statistics are calculated using exact match by querying the keyphrases in title or abstract by months. We set different x-axis ranges for the two keyphrases, because “language models” have been explored at an earlier time. We label the points corresponding to important landmarks in the research progress of LLMs. A sharp increase occurs after the release of ChatGPT: the average number of published arXiv papers that contain “large language model” in title or abstract goes from 0.40 per day to 8.58 per day.

Technical Evolution of GPT-series Models

A brief illustration for the technical evolution of GPT-series models. We plot this figure mainly based on the papers, blog articles and official APIs from OpenAI. Here, solid lines denote that there exists an explicit evidence (e.g., the official statement that a new model is developed based on a base model) on the evolution path between two models, while dashed lines denote a relatively weaker evolution relation.

gpt-series

Evolutionary Graph of LLaMA Family

An evolutionary graph of the research work conducted on LLaMA. Due to the huge number, we cannot include all the LLaMA variants in this figure, even much excellent work.

LLaMA_family

To support incremental update, we share the source file of this figure, and welcome the readers to include the desired models by submitting the pull requests on our GitHub page. If you're instrested, please request by application.

Prompts

We collect some useful tips for designing prompts that are collected from online notes and experiences from our authors, where we also show the related ingredients and principles (introduced in Section 8.1).

prompt examples

Please click here to view more detailed information.

Welcome everyone to provide us with more relevant tips in the form of issues. After selection, we will regularly update them on GitHub and indicate the source.

Experiments

Instruction Tuning Experiments

We will explore the effect of different types of instructions in fine-tuning LLMs (i.e., 7B LLaMA26), as well as examine the usefulness of several instruction improvement strategies.

instruction_tuning_table

Please click here to view more detailed information.

Ability Evaluaition Experiments

We conduct a fine-grained evaluation on the abilities discussed in Section 7.1 and Section 7.2. For each kind of ability, we select representative tasks and datasets for conducting evaluation experiments to examine the corresponding performance of LLMs.

ability_main

Please click here to view more detailed information.

We also call for support of computing power for conducting more comprehensive experiments.

Table of Contents

Timeline of LLMs

LLMs_timeline

List of LLMs

<
Category model Release Time Size(B) Link
Publicly
Accessbile		 T5 2019/10 11 Paper
mT5 2021/03 13 Paper
PanGu-α 2021/05 13 Paper
CPM-2 2021/05 198 Paper
T0 2021/10 11 Paper
GPT-NeoX-20B 2022/02 20 Paper
CodeGen 2022/03 16 Paper
Tk-Instruct 2022/04 11 Paper
UL2 2022/02 20 Paper
OPT 2022/05 175 Paper
YaLM 2022/06 100 GitHub
NLLB 2022/07 55 Paper
BLOOM 2022/07 176 Paper
GLM 2022/08 130 Paper
Flan-T5 2022/10 11 Paper
mT0 2022/11 13 Paper
Galatica 2022/11 120 Paper
BLOOMZ 2022/11 176 Paper
OPT-IML 2022/12 175 Paper
Pythia 2023/01 12 Paper
LLaMA 2023/02 65 Paper
Vicuna 2023/03 13 Blog
ChatGLM 2023/03 6 GitHub
CodeGeeX 2023/03 13 Paper
Alpaca 2023/03 7 Blog
Koala 2023/04 13 Blog
Mistral 2023/09 7 Blog
Closed
Source		 GShard 2020/01 600 Paper
GPT-3 2020/05 175 Paper
LaMDA 2021/05 137 Paper
HyperCLOVA 2021/06 82 Paper
Codex 2021/07 12 Paper
ERNIE 3.0 2021/07 10 Paper
Jurassic-1 2021/08 178 Paper
FLAN 2021/10 137 Paper
MT-NLG 2021/10 530 Paper
Yuan 1.0 2021/10 245 Paper
Anthropic 2021/12 52 Paper
WebGPT 2021/12 175 Paper
Gopher 2021/12 280 Paper
ERNIE 3.0 Titan 2021/12 260 Paper

LLMSurvey (RUCAIBox/LLMSurvey)

Searching Database…

12.2K ·GitHub
LLMSurvey
这是什么？这是综述论文《A Survey of Large Language Models》的官方 GitHub 页面，精选了大语言模型的构建、适配和使用相关的论文与资源。
这是能运行的软件，还是阅读资料？是阅读资料，不是可运行的工具：这个仓库按照综述的结构整理论文和清单（大语言模型时间线、模型目录、语料库、各种库、提示词技巧等）。
刚接触大语言模型的人该从哪里入手？先读综述本身——它给出了从预训练到评估的完整框架和路线图——或者选择那本中文书，它是为零基础读者写的入门技术书。
看懂它需要深度学习基础吗？需要一些：那本中文书面向高年级本科生和低年级研究生，前提是已经有深度学习基础。
它能跟得上这个领域的变化速度吗？团队一直在持续更新：最新版本增加了关于长 CoT 推理的章节，这正是 DeepSeek-R1 和 OpenAI o 系列等近期模型背后的范式。
里面的实验我自己能试试吗？他们介绍了在 7B 的 LLaMA 模型上做指令微调的实验，以及细粒度的能力评估，并提到正在寻求算力来做更全面的实验。
它的模型目录收录了哪些大语言模型？包括一条时间线，以及按开放和闭源分类的模型清单——开放的有 T5、BLOOM、GLM、LLaMA、ChatGLM、Vicuna、Mistral；闭源的有 GPT-3、LaMDA、Gopher 和 MT-NLG（530B）等——每个都标有发布日期、参数规模和链接。
新增的长 CoT 推理章节讲了什么？讲了长 CoT 范式的推理模式和优势，这类数据是怎么构造的（蒸馏、基于搜索的合成、多智能体协作），两种训练方法（长 CoT 指令微调和规模化强化学习），以及近期的测试时扩展（test-time scaling）工作。
README 里的 arXiv 论文数量图表是干什么用的？用来说明这个领域的爆发式增长：ChatGPT 发布之后，标题或摘要中提到“large language model”的 arXiv 论文，日均数量从每天 0.40 篇涨到了 8.58 篇。
除了论文清单，它还有什么实用的内容？有——一套提示词设计技巧，附带构成要素和原则（欢迎通过 issues 投稿补充），另外还有指令微调和能力评估实验的详细介绍。
What are similar projects?Awesome-Multimodal-Large-Language-Models, Foundations-of-LLMs, LLMsPracticalGuide, llm_interview_note, Prompt-Engineering-Guide, LLMs-from-scratch, Awesome-LLM, open-llms
Which topics is it in?in-context-learning, chain-of-thought, instruction-tuning, rlhf, reasoning, llms, papers, natural-language-processing, academic, chatgpt, large-language-models, prompt-engineering
Creator
RUCAIBox
Size
44.2 MB
Stars
12.2K
Updated
CommitMar 11
Age
3y
Commits · 90 Days