# Hands-on: cleaning customer email threads before they go into RAG

> A 12-email thread can hold 78 copies of its content, and your retriever rereads the old messages every time someone hits Reply.

Original: https://fdetimes.net/en/guides/clean-email-threads-for-rag/

Picture a 12-email thread between you and a customer. An analysis on DEV Community points out that when you pull this thread through the Gmail API, every reply comes back with the entire quoted chain beneath it. The first email therefore appears 12 times, the second 11 times. Add it up and 12 real emails become 1 + 2 + … + 12 = 78 copies of content.

Of those 78 copies, only 12 are new; the rest are repeats. A user on the LangChain forum reported that when chunking whole emails, 60-80% of the content was quoted history. That is a field observation rather than a systematic measurement, but it matches the arithmetic above.

So when a customer asks "when did we last agree the deadline?", the retriever is likely to pull back the old agreement, because that sentence is repeated most often. This article builds a small pipeline that deals with the problem at ingest. If a customer wants question answering over their mailbox, this is the stage to build first.

## What will you build, and what do you need?

The input is the raw emails of one thread. The output is a list of records, one per email: the text the sender just wrote, with no quoted text and no signature, plus metadata. The field list follows a suggestion on the LangChain forum: sender, recipients, date, thread_id, message_id, in_reply_to, has_attachments.

You need Python and two open-source libraries. Zapier's Email Reply Parser is a Python port of GitHub's parser, built to extract the latest reply in a thread. Mailgun's Talon was written specifically to separate quoted text and signatures from emails.

```bash
pip install email_reply_parser
pip install talon
# Talon's machine learning mode needs its own extra
pip install "talon[ml]"
```

Check: both libraries import in a REPL. If you don't need ML mode yet, skip the third line.

## Step 1: get the sender and date before touching the content

Headers must be read first. Once you cut the quoted text, you also lose lines like "On Monday, A wrote:", which are the only place in the body that says who said what, whereas the headers stay intact. The snippet below is a **simplified** sketch: it uses Python's built-in `email` module and assumes each email has a single plain-text part.

One detail is easy to miss: the `Date` header is a string such as "Mon, 3 Feb 2025 09:15:00 +0700". Sorting these strings alphabetically does not give chronological order, so convert to `datetime` as you read.

```python
import email
from email.utils import parsedate_to_datetime

def read_headers(raw: str, thread_id: str) -> dict:
    msg = email.message_from_string(raw)
    raw_date = msg["Date"]
    return {
        "sender": msg["From"],
        "recipients": msg["To"],
        # the raw Date string doesn't sort chronologically, so parse it
        "date": parsedate_to_datetime(raw_date) if raw_date else None,
        "thread_id": thread_id,
        "message_id": msg["Message-ID"],
        "in_reply_to": msg["In-Reply-To"],
        # rough approximation: multipart doesn't always mean an attachment
        "has_attachments": msg.is_multipart(),
        "body": msg.get_payload(),
    }
```

Check: print the records for the second and third emails. `date` must be a `datetime` object, and the third email's `in_reply_to` must match the `message_id` of the email it replies to. Don't underestimate this field.

The DEV Community piece stresses that an email thread is a conversation tree, not a straight line, so flattening a thread loses the information about who replied to whom. Keep `in_reply_to` and you can rebuild the tree when you need it.

## Step 2: keep only what the sender just wrote

This step removes the 66 repeated copies in the opening example. `EmailReplyParser.parse_reply` returns exactly the newest reply in an email.

```python
from email_reply_parser import EmailReplyParser

def newest_text(body: str) -> str:
    return EmailReplyParser.parse_reply(body)
```

Check: compare the length of `body` with the result. For the last email in the 12-message thread, the result should be far shorter and contain no sentence from earlier emails. The library leans towards plain text, so if the customer sends a lot of HTML email, don't assume it still works well: collect a set of their real HTML samples and read the output by eye before trusting it.

**Key point:** Clean emails before embedding; don't expect the retriever to filter out the noise by itself.

## Step 3: strip signatures, because no two signatures look alike

With the quoted text gone, the end of each passage still carries a signature. The DEV Community article describes a real organisation with more than 30 signature styles, not counting compliance disclaimers written in several languages. Left in place, all of that gets embedded as content.

Talon strips signatures in two modes. The rule-based mode needs nothing extra, and Talon's README describes it as "fast and works great 90% of the time". The remaining cases use ML mode, built on scikit-learn's SVM classifiers and requiring the `talon[ml]` extra.

The function returns a pair of the body and the extracted signature; only the body is kept.

```python
from talon.signature.bruteforce import extract_signature

def strip_signature(text: str, sender: str) -> str:
    # rule-based mode: returns (body, signature or None)
    body, _signature = extract_signature(text)
    # ML mode (needs talon[ml]) also takes the sender's address;
    # switch to it only when tests show the rule-based mode misses
    return body
```

Check: take 20 emails from 20 different senders and read the end of each result by eye. Long disclaimers often don't look like signatures, so build a separate pattern list for this particular customer's disclaimers.

## Step 4: one chunk per email, and count against an independent number

A chunking guide on theneuralbase recommends treating each email as a natural boundary and attaching the thread's metadata. Chunking by a fixed token count cuts across emails and mixes two senders in the same passage.

The check this source suggests is that the number of chunks must equal the number of emails. But if you recount the list you just built, the check can never fail. So compare against a number from somewhere else, such as the email count the API reports for the thread, and also check that no `message_id` is duplicated.

```python
def build_chunks(raws: list[str], thread_id: str,
                 expected_count: int) -> list[dict]:
    # expected_count: the thread's email count as reported by the API, not counted from raws
    chunks = []
    for raw in raws:
        rec = read_headers(raw, thread_id)
        text = newest_text(rec.pop("body"))
        rec["text"] = strip_signature(text, rec["sender"])
        chunks.append(rec)
    ids = {c["message_id"] for c in chunks}
    assert len(ids) == len(chunks), "duplicate message_id found"
    assert len(chunks) == expected_count, "chunk count differs from the API's email count"
    return chunks
```

Check: for the sample thread, chunking whole emails also yields 12 chunks, so the number 12 on its own proves nothing. What you need to see is that each chunk now holds only one email's content, meaning the 66 repeated copies are gone. Search all chunks for a distinctive sentence from the first email: it must appear exactly once.

The open-source project mailrag also lists removing reply chains among its core preprocessing steps, alongside deduplicating chunks with identical text and filtering out newsletters. Those two steps should run straight after this chunking stage.

## Step 5: filter on metadata first, search semantically second

Once you have clean metadata, don't make the embeddings guess dates. The LangChain forum puts the principle simply: filter by metadata first, retrieve semantically second. The question "when did we last agree the deadline?" should run in this order: filter by `thread_id`, sort by `date` (already a `datetime` from step 1) in descending order, and only then rank semantically within the filtered set.

Check: ask the same question twice, once with and once without the date filter. If the unfiltered run returns an older email, you have a ready-made before-and-after example worth showing the customer.

## What are the most common mistakes?

The first is testing only on your own email. Email Reply Parser's README admits that every email client formats replies differently, which makes parsing hard. Ask the customer which clients they use, then add real samples from each to your test set.

The second is stripping quoted text before reading the headers, leaving you with clean text and no idea who wrote it. The third is treating a thread as a straight chain. When two people reply to the same email, only `in_reply_to` tells you where the two branches split.

## On a real project, what will the customer complain about?

Customers rarely ask you to "clean the email". More likely they will complain that the bot answers with outdated information, and that is exactly the symptom the count at the top of this article explains.

The advice is to open with that same count: take one of their real threads, count the repeats and show them, and only then propose spending time on the ingest stage.

When you put this experience on your CV, state the measured result rather than listing tool names. For example: "removed 66 of 78 duplicate content copies in a 12-email thread by stripping quotes at ingest, and added date filtering before retrieval". That sentence shows an FDE interviewer that you understand customer data.

If a job description mentions RAG over enterprise data, this is an example worth bringing to the interview.

No model, however good, can rescue a mailbox where every old email is repeated a dozen times. To get the right answer, clean the data first.

**Try this week:**

- Take a real thread of at least 10 emails (with permission to use it) and count the characters before and after running parse_reply to see how much is quoted text.
- Write tests for emails from at least three different clients the customer uses, and rerun the suite every time you change parser.
- In your existing ingest pipeline, compare each thread's chunk count with the email count the API reports for it, and check that no message_id is duplicated.

## Sources

- [mailgun/talon – GitHub](https://github.com/mailgun/talon)

- [zapier/email-reply-parser – GitHub](https://github.com/zapier/email-reply-parser)

- [Why email breaks every RAG pipeline – DEV Community](https://dev.to/sam_kennard_8d1969bb/why-email-breaks-every-rag-pipeline-4i9g)

- [Beginner questions & Troubleshooting: RAG Best practices for Mail Automation – LangChain Forum](https://forum.langchain.com/t/beginner-questions-troubleshooting-rag-best-practices-for-mail-automation/3818)

- [Email threads by message with thread context – Chunking strategies (theneuralbase)](https://theneuralbase.com/chunking-strategies/learn/intermediate/email-threads-by-message-with-thread-context/)

- [fmasi/mailrag – GitHub](https://github.com/fmasi/mailrag)