gpt-3 README

The project's README from github.com/openai/gpt-3. As one markdown file: README.md.

GPT-3: Language Models are Few-Shot Learners

arXiv link

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions – something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

Contents

How to cite

@article{brown2020language,
    title={Language Models are Few-Shot Learners},
    author={Tom B. Brown and Benjamin Mann and Nick Ryder and Melanie Subbiah and Jared Kaplan and Prafulla Dhariwal and Arvind Neelakantan and Pranav Shyam and Girish Sastry and Amanda Askell and Sandhini Agarwal and Ariel Herbert-Voss and Gretchen Krueger and Tom Henighan and Rewon Child and Aditya Ramesh and Daniel M. Ziegler and Jeffrey Wu and Clemens Winter and Christopher Hesse and Mark Chen and Eric Sigler and Mateusz Litwin and Scott Gray and Benjamin Chess and Jack Clark and Christopher Berner and Sam McCandlish and Alec Radford and Ilya Sutskever and Dario Amodei},
    year={2020},
    eprint={2005.14165},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

gpt-3 (openai/gpt-3)

gpt-3
Searching Database…

15.7K ·GitHub
Gpt-3
archived
What is this?The official companion repo for the GPT-3 paper 'Language Models are Few-Shot Learners'. GPT-3 is an autoregressive language model with 175 billion parameters — 10x more than any previous non-sparse model — tested on few-shot learning across many NLP tasks.
Is this the GPT-3 model itself, or something else?Neither the weights nor model code are here — the repo ships paper artifacts: sample outputs, synthetic word-scramble and arithmetic datasets, training-mix language statistics, n-gram overlap analysis, and a model card.
Is there anything to install or run here?No. There are no build scripts, dependencies, or runnable code — you simply read files like model-card.md or download 175bsamples.jsonl and the datasets.
What are the raw samples in 175bsamples.jsonl like?Unconditional, unfiltered 2048-token generations produced with p=.85 and t=1. The README warns they may contain offensive content and language because GPT-3 was trained on arbitrary web data.
What does 'few-shot' mean in the paper's title?GPT-3 is applied without any gradient updates or fine-tuning: the task and a handful of demonstrations are written into the prompt as plain text. That task-agnostic setup let one model rival systems fine-tuned on thousands of examples.
How well did GPT-3 do in the paper's evaluations?Strongly on translation, question-answering, cloze tasks, and on-the-fly reasoning such as unscrambling words and 3-digit arithmetic — sometimes competitive with prior fine-tuned state of the art. Human evaluators also struggled to tell its news articles from real ones.
Is anyone still maintaining the repo?No — GitHub lists it as archived, so it stands as a frozen 2020 research artifact. The arXiv paper link, sample data, dataset statistics, and model card all remain readable.
What are similar projects?gpt-2, MSA, Attention-Residuals, train-llm-from-scratch, math, LoRA, minGPT, nanoGPT
Which topics is it in?gpt-3, pretraining, text-generation, arxiv, paper, dataset, gpt, fine-tuning, academic-research, transformer, large-language-models, nlp
Creator
openai
Size
2.2 MB
Stars
15.7K
Updated
CommitSep 18
Age
6y
Commits · 90 Days