# Build a Large Language Model (From Scratch): write your own GPT in 368 pages to understand the model you deploy for clients

> Sebastian Raschka's book will not turn you into a researcher. It will help you answer the questions clients keep asking about latency, fine-tuning and how to measure results.

Original: https://fdetimes.net/en/books-courses/build-llm-from-scratch-raschka-review/

Picture a client asking you: why does compute cost grow faster than the prompt when the prompt gets longer? If you have only ever called a model through an API, your answer will be vague. If you have written the attention mechanism yourself, you can sketch it on a whiteboard in two minutes.

Sebastian Raschka's *Build a Large Language Model (From Scratch)* promises exactly that. The book runs to 368 pages and was published by Manning in September 2024 (ISBN 9781633437166). According to Manning, you write a base model yourself without relying on any existing LLM library, then turn it into a classifier and an instruction-following chatbot.

If you deploy LLMs for clients, this is not a theory book. It helps you understand what you are deploying well enough to diagnose failures, estimate costs and choose the right way to customise a model.

## A book for typing, not for reading

The official repo, rasbt/LLMs-from-scratch, follows the order of someone who has to make a model actually run. Chapter 2 handles text data, chapter 3 covers attention, chapter 4 assembles a GPT model, chapter 5 is pretraining, and chapters 6 and 7 cover two kinds of fine-tuning. Appendix E deals with LoRA.

The code is written to run on an ordinary laptop in reasonable time, and uses a GPU automatically if the machine has one. Manning asks only for intermediate Python and some machine learning knowledge. Raschka also offers a free YouTube course that accompanies the book, useful when you get stuck on a chapter.

## Why does a prompt twice as long make attention cost four times as much?

The attention chapter makes you compute a score for every pair of tokens yourself. Do a small calculation: a sequence of n tokens produces an n×n matrix. At 1,000 tokens, the matrix has 1 million cells. At 2,000 tokens, it has 4 million.

The chapter 4 bonus material in the repo includes a KV cache. Raschka explains that the cache stores the key and value vectors already computed, so they can be reused when generating new tokens.

A rough estimate shows the difference: a 1,000-token prompt, with the model generating 200 more tokens. Without a cache, each generation step reprocesses the whole sequence, from 1,001 to 1,200 tokens, which adds up to 220,100 token passes.

With a cache, the model processes the 1,000 tokens once, then takes 200 steps of one new token each: 1,200 passes in total. That figure is not the total compute cost. Each new token still has to attend to every stored key, so the saving comes from not recomputing keys and values for earlier tokens; attention itself does not go away.

Raschka is also frank about the price: the code is more complex, and memory grows with sequence length. So when a client complains about latency, you know what to ask: how long is the prompt, how long is the output, is the cache on, and how much memory does the machine have left.

**Key point:** You do not need to train a model for your client, but you do need to know where it breaks.

## Does the client need labels or generated text?

Chapters 6 and 7 separate two different jobs. Fine-tuning for classification turns the model into a labeller. Instruction fine-tuning teaches the model to respond to requests. A client who wants to sort support tickets automatically into five groups has a chapter 6 problem, measured by accuracy on a test set.

If the client instead wants an assistant that drafts replies to tickets, you are in chapter 7, and measurement is much harder. The freer the output, the less there is a single right answer to compare against. So you need a repeatable way of scoring before you can claim the fine-tuned model beats the original.

At the client's site, before discussing fine-tuning, write down whether their problem is labelling or text generation. The answer decides both the approach and how you measure it.

## Two traps: skimming and rushing to fine-tune

An independent reader wrote on DEV Community that the book demands a lot of effort, and that finishing it may take longer than you expect. Do not read it just to know. The n×n calculation only really sinks in when you change the sequence length yourself and watch the matrix size change on screen.

The second trap appears when you take the book to a client: jumping straight into instruction fine-tuning. You can lose a whole week on chapter 7 when all the client needed was a ticket sorter measured by accuracy.

It also helps to know what the book leaves out. Another review calls it a guide from first principles, line by line of code. But the reviewer notes that some topics, such as top-p sampling and RLHF, are only mentioned in passing or left out altogether.

## In what order should you read it?

Read chapters 2 to 4 very closely and retype the code. Chapter 5 on pretraining can be read faster. Then choose chapter 6 or 7 depending on the kind of problem your clients usually bring, and finish with LoRA in appendix E.

The payoff comes late but it is real. The DEV Community reviewer says that after finishing, they found articles on attention or the KV cache far easier to follow. For an FDE, that is the skill of keeping up with new technical material every week.

For your career, do not let the learning sit idle in a notebook. Put a model fine-tuned on public data on GitHub, with a page that sets out how you measured it and the results. On a CV, a line such as "implemented a GPT-style model from scratch, fine-tuned it for classification and measured accuracy on a test set" is far more convincing than "experience with LLMs".

The next time a client asks why the model is slow or why fine-tuning did not improve results, you will not have to guess. You will have written that very code yourself.

**Try this week:**

- Clone the rasbt/LLMs-from-scratch repo, run the attention notebook on your laptop, change the input sequence length and print the size of the attention matrix.
- Take a public classification dataset, fine-tune following chapter 6 and record the test-set accuracy in the README.
- Before starting chapter 7, write a table of 20 questions with scoring criteria for a text-generation problem.

## Sources

- [Build a Large Language Model (From Scratch) – Manning Publications](https://www.manning.com/books/build-a-large-language-model-from-scratch)

- [Sebastian Raschka – Manning Publications](https://www.manning.com/authors/sebastian-raschka)

- [rasbt/LLMs-from-scratch – GitHub](https://github.com/rasbt/LLMs-from-scratch)

- [Books – Sebastian Raschka](https://sebastianraschka.com/books/)

- [Understanding and Coding the KV Cache in LLMs from Scratch – Sebastian Raschka](https://magazine.sebastianraschka.com/p/coding-the-kv-cache-in-llms)

- [Book Review: Build a Large Language Model (From Scratch) by Sebastian Raschka – DEV Community](https://dev.to/uponthesky/book-review-build-a-large-language-model-from-scratch-by-sebastian-raschka-1286)

- [Building an LLM from scratch – dpranantha](https://dpranantha.github.io/2025-04-10-building-llm-from-scratch)