Language Modeling
from Scratch
From data to behavior: the full language-modeling pipeline
What four ingredients make an LLM work?
If you built one from scratch, what all would you need?
???
Click for answer
Data
Curated text that teaches the model what patterns exist.
???
Click for answer
Architecture
Transformer blocks that turn context into next-token scores.
???
Click for answer
Learning Signal
Next-token loss plus gradient updates that change the weights.
???
Click for answer
Compute Power
Massive hardware that makes large-scale training and inference actually feasible.
Ask: “If we built an LLM from scratch, what would we actually need?” Reveal the four ingredients, then use the learning signal to bridge into objective, loss, and updates.
Workshop roadmap
Why LLMs behave the way they do
- → Inference vs. training
- → Next-token prediction
- → Loss → gradients → updates
- → Why this objective works
- → Data, scale, and alignment
How a Transformer processes text and computes the next word
- → Tokenization & embeddings
- → Self-attention
- → Feed-forward layers
- → Full forward pass end-to-end
“You’ve already seen inference: context goes in, a next-token distribution comes out. But that behavior only exists because of how the weights were trained. Today is the shaping story. Tomorrow we zoom into the computation.”
The inference (use-time) loop
At use time, the model runs a fixed computation. The input text is tokenized once. After that, the model repeats the same loop: score next tokens, choose one, decode it, append it, and continue.
input
(one-time)
tokens
tokens
(fixed)
(scores for each token)
next token
(token → text)
“The model never produces words directly — it produces token IDs. Decoding just turns those IDs into readable text. The loop itself runs entirely at the token level.”
The training (learning) loop
Training starts with the same forward pass as inference, then adds ground truth, loss, backpropagation, and a weight update.
tokens
Blocks
(weights \(\theta\))
(raw scores)
next token
\(P(\text{token})\)
(from data)
teacher forcing — the correct next token is always known during training
\(\mathcal{L} = -\log P(\text{correct token})\)
"How surprised was the model by the right answer?"
gradients
layer \(N \to N-1 \to \dots \to 1\)
error
nudge every weight to reduce the loss
Walk through each step. Emphasize: "The forward pass is identical to inference. Training only adds the teacher signal and backward pass. Backpropagation is just the chain rule applied mechanically — for each weight, how much did it contribute to the error? Then nudge it to contribute less."
Teacher forcing: shift the sequence by one token
One training sequence becomes many prediction problems: each prefix is used to predict the token that comes next.
Given a token sequence \(x_1, x_2, \ldots, x_T\), teacher forcing shifts it by one position:
Inputs (prefixes)
Targets (next tokens)
At position \(t\), the model sees \(x_{\le t}\) and is scored on the true next token \(x_{t+1}\).
The equivalent loss function
Same objective, written as a loss: negative log-likelihood, or cross-entropy.
Lower loss = the model assigns higher probability to tokens that actually occurred in the dataset
What the loss never names
After one full update, the core constraint is visible: training directly rewards only better next-token prediction. Anything richer has to be useful for prediction, not separately requested.
"The loss does not explicitly name truth, reasoning, helpfulness, or safety. It only rewards better next-token prediction. So if any richer behavior appears, it has to be instrumentally useful for prediction."
Predict the next token
We’ll trace one concrete training example from input context to weight update.
Set up the concrete example. This is one single training step — we'll trace it all the way through loss computation to parameter update.
The model outputs a probability distribution
The prefix runs through the Transformer and becomes a probability distribution over the vocabulary.
What the model should have predicted
We know the correct next token from the training data. The target distribution puts \(100\%\) probability on the correct answer.
Cross-entropy loss: how wrong is the model?
The loss measures the gap between prediction and target. For next-token prediction, we use cross-entropy loss.
Backpropagation computes gradients
The loss signal flows backward through the network. For each parameter, we compute how changing it would affect the loss.
compute gradients
update params
"This is the part that actually makes learning happen. The gradient is a vector pointing in the direction of steepest increase of the loss — so we go the opposite direction to decrease it."
The model improves!
After the update, the model assigns more probability to the correct token. Repeat this across many examples, and the model learns.
Why a local objective can create reasoning-like behavior
Local objective, global pressure: the model is graded one token at a time, but getting that token right often requires a coherent picture of the whole prefix.
Track who is who and how references connect across the passage.
Infer what caused what, and which outcomes are still possible.
Rule out continuations that violate timing, access, or prior facts.
Maintain the shape of a plan, proof, argument, or story.
Next-token prediction in a Pakistani drama
The answer is implied, not stated. To predict the next token here, the model has to combine timing, access, and entity tracking.
One next token can depend on the whole story
The label is still local, but the evidence is distributed across many lines. That is why next-token training can reward richer internal state.
What training still asks for
Given a prefix, output a distribution over the next token. During training, we reward higher probability on the true continuation.
To predict correctly, it must track
- Time window: when the poison could have been taken (10:20–10:30)
- Access: who could have reached the pantry during that window (departures + CCTV)
- Consistency: reconcile all lines into one coherent timeline
Immediately after the drama slide: the supervision is still only the next token, but getting that token right rewards a coherent internal case file. The “reasoning” is the latent state that makes the correct continuation likely, not an explicit label.
You Are the Language Model
Try each blank first. Then reveal what kind of knowledge it depends on. “Just predict the next token” quietly demands many different capabilities.
if n <= 1: return n
return fibonacci(n-1) + ??? type? code structure
Let participants try the blanks first. The point is not memorization; even this simple objective calls on world knowledge, common sense, code structure, logic, arithmetic, and pragmatics.
Pretraining at scale repeats the same objective billions of times
Same loop, huge exposure. This is where the model first picks up language fluency, broad statistical regularities, and background knowledge useful for continuation.
Capability follows the data mixture
Once the model has language foundations, the next lever is the distribution you continue training on. Different mixtures reward different strengths.
Code-heavy mixture
Example: 80% code + technical text, 20% generic data
- Strength: better symbolic precision, decomposition, and code completion.
- Tradeoff: less stylistic range in narrative or poetic writing.
Literary-heavy mixture
Example: 80% literary/prose text, 20% generic data
- Strength: richer tone control, narrative fluency, and stylistic variation.
- Tradeoff: less reliability on coding and math-heavy symbolic tasks.
This is the specialization dial. Foundations come from broad pretraining; later data mixture changes what the model becomes especially good at.
It's still just a text completer
Pretraining made the model a powerful completer, not automatically a helpful assistant. So what continuation does it find likely here?
What continuation does the base model expect?
Ask the audience what the model finds likely, not what it “should” do. Then reveal the continuation and use it to set up prompt steering versus actual alignment.
Can we fix this without changing the model?
The weights are frozen. All you can change is the prompt. Can formatting alone steer next-token completion into useful behavior?
Prompt shaping can steer behavior
Same model, same weights. Only context formatting changes what continuation becomes likely.
Chat — get a helpful reply
→ "The capital of France is Paris."
Math — get the answer
→ "475"
Code — get the implementation
→ completes the function body
Run this as a pure prompt-formatting experiment. Stress that no retraining is happening yet; we're only steering local continuation probabilities.
Common LLM failures, live
Even chat-tuned models can still fail on low-level tasks. Try a legacy chat model on letter counting, reversal, exact copying, sorting, and decimal comparison.
What these probes test
- Character control: counting letters or reversing a word requires finer tracking than many token predictors manage well.
- Exactness: copying a word N times or sorting a short list reveals how quickly outputs drift from strict constraints.
- Symbolic comparison: decimals and ranking tasks expose the gap between pattern-matching and exact reasoning.
Clear context whenever you want a fresh start.
Why post-training exists
Prompt tricks can steer a base model, but they are fragile. Post-training updates the weights so assistant-style behavior becomes more consistent, robust, and policy-constrained.
Supervised fine-tuning (SFT)
- Train on chat demonstrations: user turns to assistant replies.
- Teach stable instruction-following and response structure.
- Make assistant-style continuation the default.
Preference training + RLHF
- Use human rankings to separate better from worse candidate replies.
- Push behavior toward helpfulness, policy compliance, and safer refusals.
- Improve robustness beyond specific prompts or formatting tricks.
Bridge directly from the activity: prompting can steer behavior, but post-training changes the default behavior itself. Keep repeating: still autoregressive, now trained to prefer better continuations.