Training a Transformer idle
step 0loss —examples seen 00 steps/s

One training step

repeat until the loss stops falling
1

Sample a batch

Random digit sequences and their correct answers.

src, tgt
2

Forward pass

Encoder reads the source; decoder predicts every next token at once (teacher forcing).

logits = model(src, tgt_in)
3

Loss

Cross-entropy between predicted probabilities and the true tokens.

loss = CE(logits, tgt_out)
4

Backward pass

Chain rule from the loss back through every layer gives ∂loss/∂w.

loss.backward()
5

Adam update

Each weight moves a small step against its gradient.

opt.step()

Press ▶ Train to run steps continuously, or Step ×1 to watch one step in slow motion.

Learning curves

train loss*—
held-out loss—
token accuracy—
exact match—
train lossheld-out losstoken accuracyexact match (greedy)chance level

* The train loss is measured against label-smoothed targets, so it bottoms out above zero (≈ 0.5 at ε = 0.1). The held-out loss uses the true tokens and can fall towards 0.

This batch

tgt_in is the answer shifted right behind a start token <s>; the model must output tgt_out, one position ahead.

Learning-rate schedule

Model on an unseen example

greedy decoding, refreshed as it trains
input
correct
model says

softmax over digits at each output position

Token embeddings

10 digit vectors projected to 2-D (PCA)

The embedding table starts random. Training arranges the digits into whatever geometry the task needs.

Attention on the unseen example

last layer · rows = queries, columns = keys

Backward pass · gradient flow

‖gradient‖ (log scale)update ÷ weight size (log scale)

The backward pass sends a gradient to every module. Adam turns it into an update that is a tiny fraction of the weight itself (here roughly 10⁻⁴ to 10⁻²). Change the learning rate while training to see that fraction jump.

step 0