Hands-on: cleaning customer email threads before they go into RAG

Picture a 12-email thread between you and a customer. An analysis on DEV Community points out that when you pull this thread through the Gmail API, every reply comes back with the entire quoted chain beneath it. The first email therefore appears 12 times, the second 11 times. Add it up and 12 real emails become 1 + 2 + … + 12 = 78 copies of content.

Of those 78 copies, only 12 are new; the rest are repeats. A user on the LangChain forum reported that when chunking whole emails, 60-80% of the content was quoted history. That is a field observation rather than a systematic measurement, but it matches the arithmetic above.

So when a customer asks “when did we last agree the deadline?”, the retriever is likely to pull back the old agreement, because that sentence is repeated most often. This article builds a small pipeline that deals with the problem at ingest. If a customer wants question answering over their mailbox, this is the stage to build first.

What will you build, and what do you need?

The input is the raw emails of one thread. The output is a list of records, one per email: the text the sender just wrote, with no quoted text and no signature, plus metadata. The field list follows a suggestion on the LangChain forum: sender, recipients, date, thread_id, message_id, in_reply_to, has_attachments.

You need Python and two open-source libraries. Zapier’s Email Reply Parser is a Python port of GitHub’s parser, built to extract the latest reply in a thread. Mailgun’s Talon was written specifically to separate quoted text and signatures from emails.

pip install email_reply_parser
pip install talon
# Talon's machine learning mode needs its own extra
pip install "talon[ml]"

Check: both libraries import in a REPL. If you don’t need ML mode yet, skip the third line.

Step 1: get the sender and date before touching the content

Headers must be read first. Once you cut the quoted text, you also lose lines like “On Monday, A wrote:”, which are the only place in the body that says who said what, whereas the headers stay intact. The snippet below is a simplified sketch: it uses Python’s built-in email module and assumes each email has a single plain-text part.

One detail is easy to miss: the Date header is a string such as “Mon, 3 Feb 2025 09:15:00 +0700”. Sorting these strings alphabetically does not give chronological order, so convert to datetime as you read.

import email
from email.utils import parsedate_to_datetime

def read_headers(raw: str, thread_id: str) -> dict:
    msg = email.message_from_string(raw)
    raw_date = msg["Date"]
    return {
        "sender": msg["From"],
        "recipients": msg["To"],
        # the raw Date string doesn't sort chronologically, so parse it
        "date": parsedate_to_datetime(raw_date) if raw_date else None,
        "thread_id": thread_id,
        "message_id": msg["Message-ID"],
        "in_reply_to": msg["In-Reply-To"],
        # rough approximation: multipart doesn't always mean an attachment
        "has_attachments": msg.is_multipart(),
        "body": msg.get_payload(),
    }

Check: print the records for the second and third emails. date must be a datetime object, and the third email’s in_reply_to must match the message_id of the email it replies to. Don’t underestimate this field.

The DEV Community piece stresses that an email thread is a conversation tree, not a straight line, so flattening a thread loses the information about who replied to whom. Keep in_reply_to and you can rebuild the tree when you need it.

Step 2: keep only what the sender just wrote

This step removes the 66 repeated copies in the opening example. EmailReplyParser.parse_reply returns exactly the newest reply in an email.

from email_reply_parser import EmailReplyParser

def newest_text(body: str) -> str:
    return EmailReplyParser.parse_reply(body)

Check: compare the length of body with the result. For the last email in the 12-message thread, the result should be far shorter and contain no sentence from earlier emails. The library leans towards plain text, so if the customer sends a lot of HTML email, don’t assume it still works well: collect a set of their real HTML samples and read the output by eye before trusting it.

Step 3: strip signatures, because no two signatures look alike

With the quoted text gone, the end of each passage still carries a signature. The DEV Community article describes a real organisation with more than 30 signature styles, not counting compliance disclaimers written in several languages. Left in place, all of that gets embedded as content.

Talon strips signatures in two modes. The rule-based mode needs nothing extra, and Talon’s README describes it as “fast and works great 90% of the time”. The remaining cases use ML mode, built on scikit-learn’s SVM classifiers and requiring the talon[ml] extra.

The function returns a pair of the body and the extracted signature; only the body is kept.

from talon.signature.bruteforce import extract_signature

def strip_signature(text: str, sender: str) -> str:
    # rule-based mode: returns (body, signature or None)
    body, _signature = extract_signature(text)
    # ML mode (needs talon[ml]) also takes the sender's address;
    # switch to it only when tests show the rule-based mode misses
    return body

Check: take 20 emails from 20 different senders and read the end of each result by eye. Long disclaimers often don’t look like signatures, so build a separate pattern list for this particular customer’s disclaimers.

Step 4: one chunk per email, and count against an independent number

A chunking guide on theneuralbase recommends treating each email as a natural boundary and attaching the thread’s metadata. Chunking by a fixed token count cuts across emails and mixes two senders in the same passage.

The check this source suggests is that the number of chunks must equal the number of emails. But if you recount the list you just built, the check can never fail. So compare against a number from somewhere else, such as the email count the API reports for the thread, and also check that no message_id is duplicated.

def build_chunks(raws: list[str], thread_id: str,
                 expected_count: int) -> list[dict]:
    # expected_count: the thread's email count as reported by the API, not counted from raws
    chunks = []
    for raw in raws:
        rec = read_headers(raw, thread_id)
        text = newest_text(rec.pop("body"))
        rec["text"] = strip_signature(text, rec["sender"])
        chunks.append(rec)
    ids = {c["message_id"] for c in chunks}
    assert len(ids) == len(chunks), "duplicate message_id found"
    assert len(chunks) == expected_count, "chunk count differs from the API's email count"
    return chunks

Check: for the sample thread, chunking whole emails also yields 12 chunks, so the number 12 on its own proves nothing. What you need to see is that each chunk now holds only one email’s content, meaning the 66 repeated copies are gone. Search all chunks for a distinctive sentence from the first email: it must appear exactly once.

The open-source project mailrag also lists removing reply chains among its core preprocessing steps, alongside deduplicating chunks with identical text and filtering out newsletters. Those two steps should run straight after this chunking stage.

Step 5: filter on metadata first, search semantically second

Once you have clean metadata, don’t make the embeddings guess dates. The LangChain forum puts the principle simply: filter by metadata first, retrieve semantically second. The question “when did we last agree the deadline?” should run in this order: filter by thread_id, sort by date (already a datetime from step 1) in descending order, and only then rank semantically within the filtered set.

Check: ask the same question twice, once with and once without the date filter. If the unfiltered run returns an older email, you have a ready-made before-and-after example worth showing the customer.

What are the most common mistakes?

The first is testing only on your own email. Email Reply Parser’s README admits that every email client formats replies differently, which makes parsing hard. Ask the customer which clients they use, then add real samples from each to your test set.

The second is stripping quoted text before reading the headers, leaving you with clean text and no idea who wrote it. The third is treating a thread as a straight chain. When two people reply to the same email, only in_reply_to tells you where the two branches split.

On a real project, what will the customer complain about?

Customers rarely ask you to “clean the email”. More likely they will complain that the bot answers with outdated information, and that is exactly the symptom the count at the top of this article explains.

The advice is to open with that same count: take one of their real threads, count the repeats and show them, and only then propose spending time on the ingest stage.

When you put this experience on your CV, state the measured result rather than listing tool names. For example: “removed 66 of 78 duplicate content copies in a 12-email thread by stripping quotes at ingest, and added date filtering before retrieval”. That sentence shows an FDE interviewer that you understand customer data.

If a job description mentions RAG over enterprise data, this is an example worth bringing to the interview.

No model, however good, can rescue a mailbox where every old email is repeated a dozen times. To get the right answer, clean the data first.