4

3

1

2

 model.eval()

model.load_pretrained()

model.freeze()

model.train()

HeNormal (he_normal)

Zeros (initializers.Zeros)

RandomNormal with std=1.0

Xavier Uniform (glorot_uniform)

0

Depends on the dataset

20

Mean = 0 and std = 1 → improves training speed

No effect on convergence

Mean ≠ 0 and std ≠ 1

Mean = 1 and std = 0

To double the number of neurons during training

To make the model deterministic during training

To prevent underfitting by skipping layers

To randomly deactivate 50% of neurons during training to reduce overfitting

 Transformer

A way of determining the importance of each word in a sentence for the translation of another sentence

A way of identifying the topic of a sentence

 A way of predicting the next word in a sentence

A way of determining the similarity between two sentences

 (None, 1, 1, 4096)

(None, 224, 224, 3)

(None, 7, 7, 512)

(None, 14, 14, 256)

Trains a new BERT model from scratch

Converts text into numerical token representations

Applies data augmentation to text

 Converts numerical tokens back into text