Context selection is an algorithm: the lesson worth keeping from Prompt Engineering for LLMs

A prompt running in production is rarely a paragraph written once and left alone.

Each new request makes the system rebuild the prompt from scratch, picking a few pieces of context from a much larger pool. Prompt Engineering for LLMs by John Berryman and Albert Ziegler teaches exactly that skill, and it is the same skill that separates a demo from a system that works on a customer’s real data.

The book is published by O’Reilly Media, with the full subtitle The Art and Science of Building Large Language Model–Based Applications. The Helion bookshop lists the ebook as released on 4 November 2024 and notes that the print date may differ; another bookseller’s page gives 2025.

The book covers few-shot prompting, chain-of-thought and RAG, but according to its description the core task is gathering, sorting and presenting pieces of context to build an effective prompt.

For an FDE, the most useful way to read it is to stop treating a prompt as a clever instruction. Treat it as the output of a program, with inputs, constraints and an order of execution. The authors do not put it in those words, but the way the book breaks a prompt into parts that are gathered, filtered and then assembled leads straight there.

The authors come from search, and it shows

Helion describes Berryman as one of the first engineers on GitHub Copilot, specialising in language models and information retrieval systems. His Arcturus Labs site also mentions his contribution to Copilot’s early days and his work on search and recommendation systems used by millions of people every day.

That background explains why the book treats a prompt like a ranking problem: many candidates, limited space, choices to make. The mindset pays off whenever you have to find the right contract, the right ticket or the right passage of documentation and then place them in front of the model in a deliberate order.

Arcturus Labs now sells a service that embeds an AI specialist directly inside the client’s team. The site does not use the term FDE, but the working model is very close to the job.

Idea one: separate the static from the dynamic

According to Strick van Linschoten’s notes on chapter 5, the book divides prompt content into two kinds: static and dynamic. The dynamic part changes with each request.

Imagine building an assistant that answers complaints for an insurance client. The tone of voice and the rule “answer only on the basis of the policy terms” are static. The terms of the policy under dispute and the complainant’s ticket history are dynamic.

Separating the two from the start lets you test them independently: manage the static part like versioned configuration, the dynamic part like query results. The table of contents on Helion also places RAG, along with lexical retrieval and neural retrieval, in chapter 5 on prompt content, meaning retrieval is one way of obtaining the dynamic part.

Idea two: position in the prompt carries weight

Gigazine’s summary of chapter 6 states that the closer information sits to the end of the prompt, the more strongly it influences the model. The middle is called the “Valley of Meh”, where the model struggles to process information.

Back to the insurance example. An easy mistake in a first version is to put the rules only at the top of the prompt and then pour five passages of policy terms and the ticket history in after them.

The most important instruction is then pushed far away from the point where the model starts writing by the whole block of context. Following from that observation, the cheapest fix is to restate the task and constraints at the end, just before the answer.

Beyond position, you also have to choose a format. Again according to Gigazine’s summary, structured documents such as JSON, XML or YAML are very effective, especially when the model has been trained to expect that kind of structure.

On a real project, extracting fields from invoices suits a structured format. Choose prose, and you will probably end up writing extra code just to patch outputs that drift from the expected format.

Idea three: context assembly is a budgeted algorithm

This is the part where the book reads most like a programming book. Strick van Linschoten’s notes on chapter 6 say the chapter teaches how to structure a prompt, assemble its elements and choose which context goes in.

According to Gigazine’s summary, the process has several steps, one of which is to choose the best element taking token length and dependencies into account, then repeat until no elements remain.

The mistake this loop prevents is dumping every retrieved chunk into the prompt. The prompt swells, cost and latency rise, and the pieces that really matter easily slip into the middle, exactly where the model handles them worst.

Apply it to the insurance example. The policy terms have the highest value. The latest ticket has high value but depends on customer information already being present. The general FAQ has low value. If the client demands fast responses, you will want to lower the token budget, and the FAQ is the first thing cut.

Below is a sketch of that loop, short enough to copy into a project and refine:

def assemble_prompt(task, elements, token_budget):
    # elements: [{"id", "text", "value", "tokens", "depends_on", "order"}]
    remaining = sorted(elements, key=lambda e: e["value"], reverse=True)
    chosen, chosen_ids, used = [], set(), 0
    while remaining:                      # stop when no elements remain
        e = remaining.pop(0)              # highest-value element left
        if used + e["tokens"] > token_budget:
            continue                      # skip if it doesn't fit the budget
        if not set(e["depends_on"]) <= chosen_ids:
            continue                      # skip if its dependencies are missing
        chosen.append(e)
        chosen_ids.add(e["id"])
        used += e["tokens"]
    chosen.sort(key=lambda e: e["order"]) # order, then assemble
    context = "\n\n".join(e["text"] for e in chosen)
    return f"{task}\n\n{context}\n\nRestating the task: {task}"

This version is deliberately simple: an element skipped for missing dependencies is never reconsidered. But it already moves the decision about what to cut into code, where you can test it, tune token_budget to latency requirements and explain it to the client, rather than leaving that decision in the memory of whoever wrote the prompt.

In what order should you read it?

The suggestions below rest only on the chapter titles on Helion and one reader’s notes on chapters 5 and 6, not on the full table of contents or the full text.

If you have called LLM APIs but not yet put a system into production, read chapter 5 on prompt content first to grasp the static/dynamic split and the kinds of retrieval, then move on to chapter 6, Assembling the Prompt.

If you already run RAG for clients, you can go straight to chapter 6. That is where a pile of retrieved context becomes a prompt with a clear order and concrete limits.

For your career, this skill belongs on a CV in technical language. A line such as “designed a prompt assembler that selects context by token budget and latency requirements” is far more convincing than “proficient in prompt engineering”.

When reading job descriptions, look for roles that mention context, retrieval or latency alongside LLMs. There, interviewers will want to hear what information you decided to cut, and why.

Prompt tricks go stale with each generation of models. The discipline of choosing what goes in and where to put it stays useful, and that is what is worth keeping from this book.