BERT masks 15% of tokens during pre-training, but only uses [MASK] for 80% of them. What happens to the other 20%?

Question

BERT  masks 15% of tokens during pre-training, but only uses [MASK] for 80% of them. What happens to the other 20%?

Accepted Answer

10% are replaced with a random word from the vocabulary

Answer

This mix prevents the model from learning to only focus on [MASK] positions

Answer

All 20% are deleted from the sentence

Answer

10% are left as the original word unchanged

Crowdly