GPT-2 small, written from scratch before I let myself use one

Context

I finished an M.Sc. in 2024 on classical machine learning: linear models, random forests, gene expression data. By the following spring the work I wanted to do was all large language models, and I could already see how easy it would be to build on them without understanding them. Call an API, copy a notebook, tune a prompt until it behaves. I did not want to start that way. I wanted to know what was happening under the hood before I built anything on top of it, so that the choices I made later would be mine and not somebody else's that I had copied.

So from late May to the start of July 2025 I worked through Sebastian Raschka's book Build a Large Language Model (From Scratch) and wrote GPT-2 small, the 124-million-parameter model, myself. The book sets the route, and anyone who knows it will recognise the short story I trained on and the prompt I kept typing. The code is mine, written a piece at a time, with a note to myself beside each piece saying why it is there. Everything ran on my laptop, on an RTX 3070 Ti with 8 GB of memory.

"From scratch" needs a definition. I wrote the attention mechanism, layer normalisation, the GELU activation, the feed-forward network, the transformer block, the model, the training loops and the text sampling by hand. From PyTorch I took linear layers, embedding tables, dropout, the AdamW optimiser and autograd. For tokenizing I wrote my own tokenizer first, then switched to OpenAI's byte-pair encoder through tiktoken, for a reason the next section covers.

Text into tensors

My first tokenizer split text on whitespace and punctuation and gave every distinct word a number. It worked until I gave it a word that was not in the training text, and then it fell over. The second version added an unknown-word token, which stopped the crash and threw the word away. That was the lesson of the exercise. GPT-2's byte-pair encoding breaks a word it has not seen into pieces it has, so nothing is ever unknown, and its 50,257-token vocabulary is what I used from then on.

Training data for a language model is the text set against itself. I slid a window along the token stream, and the target for each window was the same window moved one token to the right, so every position learns to predict the token after it. Each token ID then looks up a 768-number vector, and a second table adds a learned vector for the position, because attention on its own has no idea what order the words came in. Before writing those tables I spent some time with pretrained word2vec vectors, checking that king minus man plus woman really does land near queen, to convince myself that a row of numbers can carry meaning at all.

Attention, four times

I built attention four times, each version fixing what the last one lacked. The first had no trainable weights at all: dot products between token vectors, a softmax, a weighted sum. It was enough to see that attention is a way for each token to pull in a blend of the others. The second added the trainable query, key and value projections, which is where the model gets to learn what to look for. The third made it causal. A model that predicts the next word must not see it, so every score above the diagonal is set to minus infinity before the softmax and comes out as exactly zero. I stored that mask as a buffer on the module, not a plain tensor, so that it follows the model onto the GPU.

The fourth was multi-head attention, and it was the hardest code in the project. My first version was the obvious one: twelve small attention modules in a list, run one after another, outputs glued together. It was correct and slow. The real version does one large projection and then reshapes it, from batch, tokens, 768 into batch, tokens, 12 heads, 64 numbers a head, then swaps two axes so that the heads sit beside the batch and one matrix multiplication serves all twelve at once. Nothing in it is conceptually hard. Keeping four-dimensional shapes straight in my head was, and I got there by writing out the shape at every step and printing it to check.

The model

The rest of the block went faster. I wrote layer normalisation myself, including the detail that GPT-2 divides the variance by n and not n minus 1, and the GELU activation from its tanh approximation. To see why the block needs shortcut connections I built a five-layer network with and without them and looked at the gradients. Without shortcuts the first layer's gradient averaged 0.0002. With them it was 0.22. That single comparison explained more to me than any diagram had.

Figure 1 is the whole model: embeddings, twelve identical blocks, a final normalisation and an output layer that scores every token in the vocabulary. When I counted the parameters I got 163.0 million, not 124. The difference is one matrix. GPT-2 uses the same 50,257 by 768 table to turn tokens into vectors on the way in and vectors into token scores on the way out, and my model kept two separate copies. Count the table once and the total is 124.4 million. That was the first time the number in the model's name meant something to me.

Training on one short story

I trained it on "The Verdict", a short story by Edith Wharton: 20,479 characters, 5,145 tokens. With a context of 256 tokens that made nine training batches and one validation batch. Before any training the loss was 10.99, about what guessing evenly across 50,257 tokens gives you. When I had first written the loss function, an untrained model's perplexity on two three-word phrases came out at 48,726: as unsure as if it were picking from nearly the whole vocabulary. Ten epochs with AdamW took 0.36 minutes.

Figure 2 is the most useful thing I saw all summer. Training loss fell from 9.82 to 0.81. Validation loss reached 6.12 in the seventh epoch and then rose to 6.32. I knew that shape from textbooks. I had never watched a model of mine draw it, and the text samples showed what the curve meant. After the first epoch, the prompt "Every effort moves you" was completed with a row of commas. By the fifth it produced English-shaped nonsense: "I felt, and I felt--I had the fact a little of the honour of the picture". By the tenth it answered with whole sentences lifted word for word from the story. A 124-million-parameter model shown five thousand tokens does not learn English. It learns the five thousand tokens.

That memorised output is also where sampling earned its place. Picking the most likely token every time recited the story. I wrote temperature scaling and top-k sampling by hand and tested them first on a toy nine-word vocabulary, where a thousand draws picked "forward" 582 times and "towards" 343 times, as the probabilities said they should. On the trained model, a temperature of 1.4 over the top 25 tokens broke the recital and gave new, much less coherent text. Sampling could add variety. It could not add knowledge that the training had not put there.

Loading OpenAI's weights

The fix for a model that has seen too little text is more text, and OpenAI had already spent that compute. They released GPT-2's weights in 2019. If my architecture really was GPT-2, their weights should fit it.

This was the second hard part, and it was hard in a different way from attention: nothing to understand, a great deal to get exactly right. The weights ship as a TensorFlow checkpoint, so I read them with TensorFlow into nested dictionaries and copied them across tensor by tensor. OpenAI stores the query, key and value projections as one fused matrix, and I had written three, so each block's matrix is cut into thirds. Their linear layers keep weights the other way round from PyTorch's, so every matrix is transposed on the way in. Their attention layers have bias terms that mine did not, so I switched biases on, and I put the context length back to 1,024 from the 256 I had trained with. Every copy goes through a small function that refuses to assign a tensor whose shape does not match, so a mistake stops the load loudly and cannot turn into a quietly wrong model.

When the last tensor was in, I gave it a prompt, "The purpose of life is", and it replied "as important as the pursuit. What does the need for happiness make of living". That is not wisdom, but it is fluent English from code I had written line by line, and it is the result on this page I care about most. Close to two hundred tensors fitted with no shape left over, which means the architecture in Figure 1 is GPT-2's, not something close to it.

Two fine-tunes

With real weights in place I fine-tuned twice. The first turned the model into a spam filter. I took a public set of 5,572 text messages, of which only 747 were spam, and balanced it to 747 of each, split 70, 10 and 20 percent into training, validation and test. I replaced the 50,257-way output layer with a two-way one, froze everything except the last transformer block and the final normalisation, and read the prediction from the last token only, because under a causal mask it is the only position that has seen the whole message. I also tried simply asking the untouched model whether a message was spam. It repeated my prompt back to me.

Before fine-tuning it scored 48.75% on test batches, a coin toss. Five epochs took 0.58 minutes. On the full sets afterwards it scored 97.60% on training, 97.32% on validation and 96.00% on the 300 test messages. Figure 3 shows the two lines staying together, the opposite of Figure 2: a pretrained model and a task with enough examples.

The second fine-tune taught a model to follow instructions. I moved up to GPT-2 medium, 355 million parameters, for this. Its first download failed with a network error, and the failing address showed me something I had missed: I had been building web addresses with a file-path join, which on Windows puts backslashes in them. The data was 1,100 instruction and response pairs in the Alpaca format, split into 935 for training, 55 for validation and 110 for testing. The fiddly part was batching. Responses differ in length, so each batch is padded to its own longest example, the targets are the inputs moved along by one, and every padding token after the first is given a label the loss function ignores, so the model is never marked on predicting padding. It took me three drafts.

One epoch took 6.17 minutes and brought validation loss from 3.76 to 0.66. Asked to turn "The chef cooks the meal every day" into the passive voice, the base model repeated the sentence and then the instruction. The tuned one wrote "The meal is prepared every day by the chef." and stopped.

To mark all 110 test answers I ran Llama 3.1 8B on my laptop through Ollama, at temperature zero, and had it score each answer against the reference from 0 to 100. The scores average 63.8, and they run from 0 to 100. It marked "The car is as fast as a bullet" at 90 for a simile exercise. It gave a 0 to the answer that Pride and Prejudice was written by George Bernard Shaw. One epoch was enough to teach a small model the form of a good answer, and nowhere near enough to make its facts reliable.

Looking back

I stopped where the book's main path stops, and there are things I did not add. My training loop has a fixed learning rate, with no warm-up, no cosine decay and no gradient clipping, all of which a serious training run would have. I had a much larger text ready, the collected Sherlock Holmes stories, and never trained on it, so the only pretraining result I have is the overfit one. The instruction model had a single epoch. I did not try LoRA, which would have let me tune the larger model by training a small fraction of its weights, and generation recomputes attention over the whole sequence for every new token because I never wrote a key-value cache. Each of those is on my list because I now know what it is for.

That is what the five weeks bought. Since then I have built retrieval pipelines, agents and a customer-facing chatbot on top of models far larger than this one, all through APIs. When a context window fills up, when a temperature setting changes the answers, when a model states something false with total confidence, I know which part of the machine is responsible, because I have written that part and watched it do exactly that.