BERT
***** New March 11th, 2020: Smaller BERT Models *****
This is a release of 24 smaller BERT models (English only, uncased, trained with WordPiece masking) referenced in Well-Read Students Learn Better: On the Importance of Pre-training Compact Models.
We have shown that the standard BERT recipe (including model architecture and training objective) is effective on a wide range of model sizes, beyond BERT-Base and BERT-Large. The smaller BERT models are intended for environments with restricted computational resources. They can be fine-tuned in the same manner as the original BERT models. However, they are most effective in the context of knowledge distillation, where the fine-tuning labels are produced by a larger and more accurate teacher.
Our goal is to enable research in institutions with fewer computational resources and encourage the community to seek directions of innovation alternative to increasing model capacity.
You can download all 24 from here, or individually from the table below:
Note that the BERT-Base model in this release is included for completeness only; it was re-trained under the same regime as the original model.
Here are the corresponding GLUE scores on the test set:
| Model |
Score |
CoLA |
SST-2 |
MRPC |
STS-B |
QQP |
MNLI-m |
MNLI-mm |
QNLI(v2) |
RTE |
WNLI |
AX |
| BERT-Tiny |
64.2 |
0.0 |
83.2 |
81.1/71.1 |
74.3/73.6 |
62.2/83.4 |
70.2 |
70.3 |
81.5 |
57.2 |
62.3 |
21.0 |
| BERT-Mini |
65.8 |
0.0 |
85.9 |
81.1/71.8 |
75.4/73.3 |
66.4/86.2 |
74.8 |
74.3 |
84.1 |
57.9 |
62.3 |
26.1 |
| BERT-Small |
71.2 |
27.8 |
89.7 |
83.4/76.2 |
78.8/77.0 |
68.1/87.0 |
77.6 |
77.0 |
86.4 |
61.8 |
62.3 |
28.6 |
| BERT-Medium |
73.5 |
38.0 |
89.6 |
86.6/81.6 |
80.4/78.4 |
69.6/87.9 |
80.0 |
79.1 |
87.7 |
62.2 |
62.3 |
30.5 |
For each task, we selected the best fine-tuning hyperparameters from the lists below, and trained for 4 epochs:
- batch sizes: 8, 16, 32, 64, 128
- learning rates: 3e-4, 1e-4, 5e-5, 3e-5
If you use these models, please cite the following paper:
@article{turc2019,
title={Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
author={Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
journal={arXiv preprint arXiv:1908.08962v2 },
year={2019}
}
***** New May 31st, 2019: Whole Word Masking Models *****
This is a release of several new models which were the result of an improvement
the pre-processing code.
In the original pre-processing code, we randomly select WordPiece tokens to
mask. For example:
Input Text: the man jumped up , put his basket on phil ##am ##mon ' s head
Original Masked Input: [MASK] man [MASK] up , put his [MASK] on phil [MASK] ##mon ' s head
The new technique is called Whole Word Masking. In this case, we always mask
all of the the tokens corresponding to a word at once. The overall masking
rate remains the same.
Whole Word Masked Input: the man [MASK] up , put his basket on [MASK] [MASK] [MASK] ' s head
The training is identical -- we still predict each masked WordPiece token
independently. The improvement comes from the fact that the original prediction
task was too 'easy' for words that had been split into multiple WordPieces.
This can be enabled during data generation by passing the flag
--do_whole_word_mask=True to create_pretraining_data.py.
Pre-trained models with Whole Word Masking are linked below. The data and
training were otherwise identical, and the models have identical structure and
vocab to the original models. We only include BERT-Large models. When using
these models, please make it clear in the paper that you are using the Whole
Word Masking variant of BERT-Large.
| Model |
SQUAD 1.1 F1/EM |
Multi NLI Accuracy |
| BERT-Large, Uncased (Original) |
91.0/84.3 |
86.05 |
| BERT-Large, Uncased (Whole Word Masking) |
92.8/86.7 |
87.07 |
| BERT-Large, Cased (Original) |
91.5/84.8 |
86.09 |
| BERT-Large, Cased (Whole Word Masking) |
92.9/86.7 |
86.46 |
***** New February 7th, 2019: TfHub Module *****
BERT has been uploaded to TensorFlow Hub. See
run_classifier_with_tfhub.py for an example of how to use the TF Hub module,
or run an example in the browser on
Colab.
***** New November 23rd, 2018: Un-normalized multilingual model + Thai +
Mongolian *****
We uploaded a new multilingual model which does not perform any normalization
on the input (no lower casing, accent stripping, or Unicode normalization), and
additionally inclues Thai and Mongolian.
It is recommended to use this version for developing multilingual models,
especially on languages with non-Latin alphabets.
This does not require any code changes, and can be downloaded here:
***** New November 15th, 2018: SOTA SQuAD 2.0 System *****
We released code changes to reproduce our 83% F1 SQuAD 2.0 system, which is
currently 1st place on the leaderboard by 3%. See the SQuAD 2.0 section of the
README for details.
***** New November 5th, 2018: Third-party PyTorch and Chainer versions of
BERT available *****
NLP researchers from HuggingFace made a
PyTorch version of BERT available
which is compatible with our pre-trained checkpoints and is able to reproduce
our results. Sosuke Kobayashi also made a
Chainer version of BERT available
(Thanks!) We were not involved in the creation or maintenance of the PyTorch
implementation so please direct any questions towards the authors of that
repository.
***** New November 3rd, 2018: Multilingual and Chinese models available
*****
We have made two new BERT models available:
BERT-Base, Multilingual
(Not recommended, use Multilingual Cased instead): 102 languages,
12-layer, 768-hidden, 12-heads, 110M parametersBERT-Base, Chinese:
Chinese Simplified and Traditional, 12-layer, 768-hidden, 12-heads, 110M
parameters
We use character-based tokenization for Chinese, and WordPiece tokenization for
all other languages. Both models should work out-of-the-box without any code
changes. We did update the implementation of BasicTokenizer in
tokenization.py to support Chinese character tokenization, so please update if
you forked it. However, we did not change the tokenization API.
For more, see the
Multilingual README.
***** End new information *****
Introduction
BERT, or Bidirectional Encoder Representations from
Transformers, is a new method of pre-training language representations which
obtains state-of-the-art results on a wide array of Natural Language Processing
(NLP) tasks.
Our academic paper which describes BERT in detail and provides full results on a
number of tasks can be found here:
https://arxiv.org/abs/1810.04805.
To give a few numbers, here are the results on the
SQuAD v1.1 question answering
task:
| SQuAD v1.1 Leaderboard (Oct 8th 2018) |
Test EM |
Test F1 |
| 1st Place Ensemble - BERT |
87.4 |
93.2 |
| 2nd Place Ensemble - nlnet |
86.0 |
91.7 |
| 1st Place Single Model - BERT |
85.1 |
91.8 |
| 2nd Place Single Model - nlnet |
83.5 |
90.1 |
And several natural language inference tasks:
| System |
MultiNLI |
Question NLI |
SWAG |
| BERT |
86.7 |
91.1 |
86.3 |
| OpenAI GPT (Prev. SOTA) |
82.2 |
88.1 |
75.0 |
Plus many other tasks.
Moreover, these results were all obtained with almost no task-specific neural
network architecture design.
If you already know what BERT is and you just want to get started, you can
download the pre-trained models and
run a state-of-the-art fine-tuning in only a few
minutes.
What is BERT?
BERT is a method of pre-training language representations, meaning that we train
a general-purpose "language understanding" model on a large text corpus (like
Wikipedia), and then use that model for downstream NLP tasks that we care about
(like question answering). BERT outperforms previous methods because it is the
first unsupervised, deeply bidirectional system for pre-training NLP.
Unsupervised means that BERT was trained using only a plain text corpus, which
is important because an enormous amount of plain text data is publicly available
on the web in many languages.
Pre-trained representations can also either be context-free or contextual,
and contextual representations can further be unidirectional or
bidirectional. Context-free models such as
word2vec or
GloVe generate a single "word
embedding" representation for each word in the vocabulary, so bank would have
the same representation in bank deposit and river bank. Contextual models
instead generate a representation of each word that is based on the other words
in the sentence.
BERT was built upon recent work in pre-training contextual representations —
including Semi-supervised Sequence Learning,
Generative Pre-Training,
ELMo, and
ULMFit
— but crucially these models are all unidirectional or shallowly
bidirectional. This means that each word is only contextualized using the words
to its left (or right). For example, in the sentence I made a bank deposit the
unidirectional representation of bank is only based on I made a but not
deposit. Some previous work does combine the representations from separate
left-context and right-context models, but only in a "shallow" manner. BERT
represents "bank" using both its left and right context — I made a ... deposit
— starting from the very bottom of a deep neural network, so it is deeply
bidirectional.
BERT uses a simple approach for this: We mask out 15% of the words in the input,
run the entire sequence through a deep bidirectional
Transformer encoder, and then predict only
the masked words. For example:
Input: the man went to the [MASK1] . he bought a [MASK2] of milk.
Labels: [MASK1] = store; [MASK2] = gallon
In order to learn relationships between sentences, we also train on a simple
task which can be generated from any monolingual corpus: Given two sentences A
and B, is B the actual next sentence that comes after A, or just a random
sentence from the corpus?
Sentence A: the man went to the store .
Sentence B: he bought a gallon of milk .
Label: IsNextSentence
Sentence A: the man went to the store .
Sentence B: penguins are flightless .
Label: NotNextSentence
We then train a large model (12-layer to 24-layer Transformer) on a large corpus
(Wikipedia + BookCorpus) for a long time (1M
update steps), and that's BERT.
Using BERT has two stages: Pre-training and fine-tuning.
Pre-training is fairly expensive (four days on 4 to 16 Cloud TPUs), but is a
one-time procedure for each language (current models are English-only, but
multilingual models will be released in the near future). We are releasing a
number of pre-trained models from the paper which were pre-trained at Google.
Most NLP researchers will never need to pre-train their own model from scratch.
Fine-tuning is inexpensive. All of the results in the paper can be
replicated in at most 1 hour on a single Cloud TPU, or a few hours on a GPU,
starting from the exact same pre-trained model. SQuAD, for example, can be
trained in around 30 minutes on a single Cloud TPU to achieve a Dev F1 score of
91.0%, which is the single system state-of-the-art.
The other important aspect of BERT is that it can be adapted to many types of
NLP tasks very easily. In the paper, we demonstrate state-of-the-art results on
sentence-level (e.g., SST-2), sentence-pair-level (e.g., MultiNLI), word-level
(e.g., NER), and span-level (e.g., SQuAD) tasks with almost no task-specific
modifications.
What has been released in this repository?
We are releasing the following:
- TensorFlow code for the BERT model architecture (which is mostly a standard
Transformer architecture).
- Pre-trained checkpoints for both the lowercase and cased version of
BERT-Base and BERT-Large from the paper.
- TensorFlow code for push-button replication of the most important
fine-tuning experiments from the paper, including SQuAD, MultiNLI, and MRPC.
All of the code in this repository works out-of-the-box with CPU, GPU, and Cloud
TPU.
Pre-trained models
We are releasing the BERT-Base and BERT-Large models from the paper.
Uncased means that the text has been lowercased before WordPiece tokenization,
e.g., John Smith becomes john smith. The Uncased model also strips out any
accent markers. Cased means that the true case and accent markers are
preserved. Typically, the Uncased model is better unless you know that case
information is important for your task (e.g., Named Entity Recognition or
Part-of-Speech tagging).
These models are all released under the same license as the source code (Apache
2.0).
For information about the Multilingual and Chinese model, see the
Multilingual README.
When using a cased model, make sure to pass --do_lower=False to the training
scripts. (Or pass do_lower_case=False directly to FullTokenizer if you're
using your own script.)
The links to the models are here (right-click, 'Save link as...' on the name):
BERT-Large, Uncased (Whole Word Masking):
24-layer, 1024-hidden, 16-heads, 340M parametersBERT-Large, Cased (Whole Word Masking):
24-layer, 1024-hidden, 16-heads, 340M parametersBERT-Base, Uncased:
12-layer, 768-hidden, 12-heads, 110M parametersBERT-Large, Uncased:
24-layer, 1024-hidden, 16-heads, 340M parametersBERT-Base, Cased:
12-layer, 768-hidden, 12-heads , 110M parametersBERT-Large, Cased:
24-layer, 1024-hidden, 16-heads, 340M parametersBERT-Base, Multilingual Cased (New, recommended):
104 languages, 12-layer, 768-hidden, 12-heads, 110M parametersBERT-Base, Multilingual Uncased (Orig, not recommended)
(Not recommended, use Multilingual Cased instead): 102 languages,
12-layer, 768-hidden, 12-heads, 110M parametersBERT-Base, Chinese:
Chinese Simplified and Traditional, 12-layer, 768-hidden, 12-heads, 110M
parameters
Each .zip file contains three items:
- A TensorFlow checkpoint (
bert_model.ckpt) containing the pre-trained
weights (which is actually 3 files).
- A vocab file (
vocab.txt) to map WordPiece to word id.
- A config file (
bert_config.json) which speci
这是 Google 官方用 TensorFlow 实现的 BERT,附带预训练检查点。BERT 是一种预训练语言表示的方法,曾在从 SQuAD 阅读理解问答到自然语言推理等众多 NLP 任务上创下当时的最优成绩。