Client documents change every week: updating and deleting in a RAG index without rebuilding from scratch

Picture a Monday morning. The client’s operations team reports that the old version of the claims procedure was taken out of the document store on Friday, yet the chatbot is still quoting it to staff. The new version has been in the index for ages. The old one was simply never deleted.

This is one of the most common incidents when RAG runs on real data. In a demo, the document set stands still. In production, the client edits, renames and removes documents every week. Sooner or later, the question “how do we keep the index in line with the source?” lands on the FDE’s desk.

The simplest approach is to drop the whole collection and re-embed everything. That works while the document set is small. But it hides the real problem: your system does not know which chunk came from which document. Once the corpus is too large to rebuild every week, that debt comes due.

Every chunk must know where it came from

Every sync mechanism rests on one principle: each chunk carries the ID of its source document, and something keeps a record of what has been indexed. In LangChain, that something is the record manager, a set of timestamped records showing which documents have been updated. The source_id_key parameter links each chunk to its source document.

This ID must be stable across runs. LlamaIndex is explicit about it: the refresh feature only works automatically when documents have a fixed id_. With SimpleDirectoryReader, the filename_as_id flag uses the file name as the ID. If you generate random IDs at load time, the system treats every run as a completely new set of documents.

Once the source ID is in the metadata, the vector database itself can delete by document. Qdrant deletes points by payload filter rather than by a list of IDs. Pinecone lets you delete records in a namespace by metadata value. A single field such as source is enough to remove every chunk of a document with one command.

The wrong cleanup mode loses data

LangChain’s Indexing API has several cleanup modes. The three below are three different answers to the question “which chunks are stale?”

Mode	What it cleans up, when to use it	Caveats
incremental	Old chunks of source IDs that appear in this run	If chunks of the same document are spread across several batches, the API does redundant work, so keep them together
full	Every document the loader does not return in this run	The loader must return the entire dataset, not part of it
scoped_full	For when the loader cannot return all the data at once, or batch size is hard to choose	Available from langchain-core 0.3.25

The crucial detail is in the middle column. Incremental only cleans up what it “sees”. A document the client has deleted from the source does not appear in the run, so incremental never touches it. That is exactly why the old claims procedure was still in the index.

Full does delete removed documents, but it is also the most dangerous mode. LangChain’s documentation notes that in full mode the loader must return the entire dataset. If the loader returns only part of it, say because the client’s API paginates and you forgot to fetch the next page, full treats everything else as garbage and deletes it.

A worked example, end to end

The numbers below are hypothetical, for illustration. Suppose the client has 1,200 policy documents. This week they edit 30 files, add 10 and remove 5. You want the index to reflect exactly those three changes, and to leave the 1,165 unchanged documents alone.

import json
import pathlib

from langchain.indexes import SQLRecordManager, index

record_manager = SQLRecordManager(
    namespace="acme_policies/qdrant",
    db_url="sqlite:///record_manager.db",
)
record_manager.create_schema()

docs = load_all_policies()  # your loader; metadata["source"] = the client's stable document ID

# Number of source documents in the previous run, kept in a small file
state = pathlib.Path("last_run.json")
last_count = json.loads(state.read_text())["sources"] if state.exists() else 0
current_sources = {d.metadata["source"] for d in docs}

if len(current_sources) < 0.9 * last_count:
    raise RuntimeError("Loader returned too few documents, stopping before cleanup")

result = index(
    docs,
    record_manager,
    vector_store,
    cleanup="full",
    source_id_key="source",
)
print(result)

state.write_text(json.dumps({"sources": len(current_sources)}))

Take each group of changes in turn. For the 30 edited files, the record manager sees different content, so it writes new chunks and cleans up the old ones. For the 10 new files, chunks are added. The 5 removed files are not in docs, so full mode deletes their chunks. Everything else matches the existing records and is skipped without re-embedding.

Staying with the same hypothetical, now run the same code with a broken loader that returns only the 40 changed files. Without the 0.9 check, full mode would treat the 1,170 documents in the index that the loader did not return as removed, and delete them.

The 90% figure appears in no documentation. It is a safety catch you set yourself, and it should be tuned to each client’s actual rate of change. Note that the count is only saved after index() finishes, so a failed run does not corrupt the baseline for the following week.

If you ever switch to incremental, keep the chunks of each document together in docs. LangChain warns that when the chunks of one source ID fall into different batches, the API does redundant work.

What if you don’t use LangChain?

LlamaIndex has refresh(). It only updates documents that have the same id_ but changed content, re-embeds only those, and in the same call inserts documents not yet in the index. Refresh does not handle deletion. Removed documents must be deleted separately by ID, and the delete removes every node of that document.

There is a small trap here: delete_from_docstore defaults to False, because nodes may be shared by several indexes using the same docstore. If your project does not share a docstore, turn this flag on explicitly, or the docstore will swell over time.

If you write directly to Qdrant, you need to understand upsert correctly. When you upsert a point whose ID already exists, Qdrant deletes the old point and re-inserts it with only the vectors you pass in.

Suppose the collection has both dense and sparse vectors, and you only want to re-embed the dense part. In that case use the update vectors operation, which leaves the vectors you don’t mention untouched.

With Pinecone, the issue is timing. Pinecone is eventually consistent, so immediately after a delete or update, queries may still return old chunks for a short while. If the sync job checks its results straight after writing, it will report false failures. Wait a while before running the verification step.

The mistakes that make a chatbot quote removed documents

The most common mistake is unstable IDs. If file names include dates, or the client often renames files, filename_as_id turns every rename into a new document, and the old version stays in the index.

During discovery, ask the client whether the source system has a fixed document ID that does not change with the name, and use that ID if it does.

The second mistake is using incremental and believing everything is clean. The third is the reverse: using full with a loader that has never been checked for returning the complete dataset. Both run smoothly until the week the client deletes their first document, or the week the client’s API fails halfway through.

This skill also deserves its own line on a CV. Don’t just write “built a RAG system”. Describe how you kept the index in line with the source when data changed every week, and how you prevented accidental deletions. When a job description mentions phrases such as ingestion pipeline or data freshness, that is your cue to tell exactly this story in the interview.

A RAG system is judged at the demo. But the client decides whether to trust it in the first week the chatbot answers with a document they have already removed.