gpt-3 README

The project's README from github.com/openai/gpt-3. As one markdown file: README.md.

GPT-3: Language Models are Few-Shot Learners

arXiv link

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions – something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

Contents

How to cite

@article{brown2020language,
    title={Language Models are Few-Shot Learners},
    author={Tom B. Brown and Benjamin Mann and Nick Ryder and Melanie Subbiah and Jared Kaplan and Prafulla Dhariwal and Arvind Neelakantan and Pranav Shyam and Girish Sastry and Amanda Askell and Sandhini Agarwal and Ariel Herbert-Voss and Gretchen Krueger and Tom Henighan and Rewon Child and Aditya Ramesh and Daniel M. Ziegler and Jeffrey Wu and Clemens Winter and Christopher Hesse and Mark Chen and Eric Sigler and Mateusz Litwin and Scott Gray and Benjamin Chess and Jack Clark and Christopher Berner and Sam McCandlish and Alec Radford and Ilya Sutskever and Dario Amodei},
    year={2020},
    eprint={2005.14165},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

gpt-3 (openai/gpt-3)

Searching Database…

15.7K ·GitHub
Gpt-3
archived
这是什么?这是 GPT-3 论文《Language Models are Few-Shot Learners》的官方配套仓库。GPT-3 是一个拥有 1750 亿参数的自回归语言模型——比之前任何非稀疏模型多 10 倍——并在众多自然语言处理任务上测试了少样本学习能力。
这里放的是 GPT-3 模型本身,还是别的东西?模型权重和模型代码都不在这里——这个仓库提供的是论文的相关材料:示例输出、合成的单词乱序和算术数据集、训练语料混合的语言统计数据、n-gram 重叠分析,以及一份模型卡。
这里有什么需要安装或运行的东西吗?没有。这里没有构建脚本、依赖项或可运行的代码——你只需要阅读 model-card.md 这类文件,或者下载 175bsamples.jsonl 和那些数据集。
175bsamples.jsonl 里的原始样本是什么样的?是在 p=.85 和 t=1 设置下生成的无条件、未过滤的 2048 个 token 的文本。README 提醒说,由于 GPT-3 是用杂乱的互联网数据训练的,这些样本可能包含冒犯性的内容和语言。
论文标题里的 "few-shot" 是什么意思?GPT-3 在使用时不做任何梯度更新或微调:任务说明和少量示例以纯文本形式直接写进提示词里。正是这种与具体任务无关的设置,让这一个模型能与在数千个样本上微调过的系统相媲美。
在论文的评测中,GPT-3 的表现怎么样?在翻译、问答、完形填空任务,以及乱序单词还原和三位数算术这类临场推理上表现很强——有时能与之前微调过的最先进水平相当。人类评估者也很难分辨它写的新闻文章和真人写的新闻。
还有人维护这个仓库吗?没有——GitHub 已将它标记为已归档,所以它就定格为一个 2020 年的研究资料。arXiv 论文链接、示例数据、数据集统计和模型卡都仍然可以查看。
What are similar projects?gpt-2, MSA, Attention-Residuals, train-llm-from-scratch, math, LoRA, minGPT, nanoGPT
Which topics is it in?gpt-3, pretraining, text-generation, arxiv, paper, dataset, gpt, fine-tuning, transformer, nlp, large-language-models, academic-research
Creator
openai
Size
2.2 MB
Stars
15.7K
Updated
CommitSep 18
Age
6y
Commits · 90 Days