# A contract summary with no false sentence can still miss a clause

> Long-context models lose what sits mid-document, so split the contract with map-reduce or refine and check the result against its own clause list.

Original: https://fdetimes.net/en/guides/summarise-long-contracts-missed-clauses/

Picture this. A client's legal team sends you a 300-page framework agreement and needs a summary before Friday's meeting.

You put the whole file into a long-context model, get back two tidy pages, read them through and find no sentence that is wrong. Three weeks later, the client's lawyer asks why the summary says nothing about the automatic renewal clause on page 180.

The model invented nothing here. It simply left something out, and in a contract an omission is as dangerous as an error. This guide shows how to build a summarisation pipeline that can answer the question every client will eventually ask: "Did we miss any clause?"

An FDE working with legal, procurement or insurance clients runs into this very early. Making a summary sound good is easy. The hard part is proving it is complete.

## Why does a single call drop clauses?

The "Lost in the Middle" study, published in TACL volume 12 in 2024, found that LLMs usually perform best when the information they need sits at the beginning or end of the context. When it sits in the middle of a long context, performance drops markedly. The authors observed this even in models designed specifically for long contexts.

In a 300-page contract, "the middle" is almost the entire body. The obligations a client needs to understand, such as liability, indemnity, termination and renewal, are usually scattered through that body rather than packed into the first or last few pages.

So the fix is not a model with a longer context. You need to split the document so that no clause has to sit in the middle of an overly long context. There are two common ways to split.

## Map-reduce or refine: choose by the client's question

The documentation for an old LangChain release (0.0.149) describes the `map_reduce` chain like this: summarise each part separately, then merge the partial summaries. The `refine` chain works sequentially: it keeps one summary and updates it each time it reads another part. Both can return intermediate steps for you to review.

Current LangGraph documentation describes the same idea under the name orchestrator-worker. The orchestrator splits the work, the workers run in parallel through the Send API, and a synthesizer combines the workers' outputs into a final result. Whichever library you use, the structure is still map-reduce.

A simple rule applies. If the client needs a complete list of clauses to review, choose map-reduce. If the client needs a narrative that captures the logic of the whole contract, choose refine, but save every intermediate version.

## Example: split by clause, not by token count

The first common mistake is cutting the contract into a chunk every N tokens. That can split a clause in two and lose its number. Cut along the structure lawyers use instead: Article, Clause, Section.

```python
import re

HEADING = re.compile(r"^(Article|Clause|Section)\s+(\d+(?:\.\d+)*)", re.M)

def split_by_clause(text):
    marks, parent = [], None
    for m in HEADING.finditer(text):
        kind, num = m.group(1), m.group(2)
        if kind == "Clause" and parent:
            cid = f"{parent}.{num}"  # Clause 2 of Article 14 -> "14.2"
        else:
            parent = num
            cid = num
        marks.append((m.start(), cid))
    chunks = []
    for i, (start, cid) in enumerate(marks):
        end = marks[i + 1][0] if i + 1 < len(marks) else len(text)
        chunks.append({"clause_id": cid, "text": text[start:end]})
    return chunks

MAP_PROMPT = """Contract definitions: {definitions}
Summarise the clause below strictly according to the definitions above. Return JSON:
{{"clause_id": ..., "type": ..., "obligation": ..., "quote": "<the most important original sentence>"}}
Do not skip the clause even if it looks procedural.

{clause}"""

def map_prompt(chunk, definitions):
    return MAP_PROMPT.format(definitions=definitions, clause=chunk["text"])
```

Two small details are worth noting. A Clause is attached to the Article that contains it, so Clause 2 of Article 14 gets the id "14.2" instead of colliding with every other "Clause 2". And in `MAP_PROMPT`, the braces of the sample JSON are doubled as `{{ }}` so that `.format()` does not mistake them for placeholders.

The crux is the `split_by_clause` function. It gives you a clause list produced by code, independent of the model. This is the answer key for grading. The map step receives the list of definitions and returns JSON with a `clause_id`; the reduce step combines those JSON objects into the summary for the client.

Now for the most important calculation. Suppose the regex finds 140 clauses. Count how many `clause_id` values appear in the output of the map step, then count again in the final summary. If the map step has all 140 but the final summary mentions only 118, then 22 clauses were dropped at the reduce step, and you know exactly which 22.

## Faithfulness cannot answer the question about omissions

A multi-domain benchmark published in 2025 places the omission of key information in the same group of known problems with LLM summarisation as hallucination and repetition.

The benchmark scores summaries on three separate dimensions: faithfulness, completeness and conciseness. The authors also note that many earlier benchmarks measured only faithfulness, meaning consistency with the facts in the source text.

That applies directly to contracts. The summary in the opening scenario has perfect faithfulness, because no sentence is wrong. But its completeness is low, and completeness is the dimension that measures lost clauses.

**Key point:** A summary without a single wrong word can still be a dangerous summary.

Your eval therefore needs two separate parts. Check faithfulness by comparing each `quote` against the source text. Check completeness with a coverage matrix: each row is a `clause_id` from the regex, and the columns are "present in map output", "present in final summary" and "clause type".

## When a clause drops out, find where it fell

The coverage matrix tells you which clause was lost. The intermediate steps tell you at which step it was lost. With map-reduce, if a clause is missing from the map output, the fault lies in the map prompt or in the splitting. If it is in the map output but not in the final summary, the reduce step compressed too hard.

With refine, remember that the chain reads parts in order, so a later part can overwrite what was recorded earlier. Imagine Article 47 is present in version 12 but gone from version 13.

That means the thirteenth update pushed Article 47 out of the summary. The fix is to add an instruction to the refine prompt to preserve the list of clauses already recorded, then run it again.

After the fix, do not rerun the whole pipeline and hope it works this time. Run only the dropped clauses, add them to the summary under a clearly labelled section, and let the coverage matrix confirm that all 140 out of 140 are there.

## Mistakes that cost you the client's trust

The first mistake is presenting a summary without its coverage figure. Lawyers do not trust a summary because it reads well. They trust it when they see the line "checked 140 of 140 clauses" alongside a detailed table.

The second mistake is letting the model build the clause list itself. That means the very system that may have missed something is grading whether it missed something. The reference list must come from the document's structure, extracted with a regex or a parser.

The third mistake is ignoring the schedules and definitions. A definition at the start of the contract can change the meaning of a clause on page 200. So put the list of definitions into the prompt for every map step, so that each clause is summarised with its defined meaning, and into the reduce prompt as well.

## Showing this skill on a CV

On an FDE application, a line such as "built a long-document summarisation pipeline with clause-level coverage checks" is worth far more than "experienced with LLMs". When reading job descriptions, look for phrases such as document intelligence, evals or legal tech. Those are the places where this skill is used every day.

In interviews, if you are asked how to evaluate a summary, answer along the two dimensions of faithfulness and completeness, then describe the coverage matrix. Separating the two cleanly shows the interviewer that you have had to answer a real client's question: "Did we miss anything?"

Clients will not remember how good your summary was. They will remember the time a clause was missed and you did not catch it.

**Try this week:**

- Take a publicly available sample contract, write a regex to split out the clause headings and count how many clauses there are. This is your reference list.
- Summarise that contract two ways: in a single call, and with map-reduce by clause. Compare how many clauses each summary misses.
- Add a step to your pipeline that turns on intermediate steps and saves the intermediate summaries, so the next time a client asks, you can show at which step a clause was dropped.

## Sources

- [Lost in the Middle: How Language Models Use Long Contexts](https://ar5iv.labs.arxiv.org/html/2307.03172)

- [Lost in the Middle: How Language Models Use Long Contexts (TACL 2024)](https://aclanthology.org/2024.tacl-1.9/)

- [Summarization (LangChain 0.0.149 documentation notebook)](https://lagnchain.readthedocs.io/en/latest/modules/chains/index_examples/summarize.html)

- [LangGraph workflows and agents - Docs by LangChain](https://docs.langchain.com/oss/python/langgraph/workflows-agents)

- [Towards Multi-dimensional Evaluation of LLM Summarization across Domains and Languages](https://huggingface.co/papers/2506.00549.md)
