Why the AI assistant still says a cancelled order is on its way
Before blaming the model, check which replica it read; the CAP theorem exposes a consistency choice nobody made on purpose.
In brief
- When an AI assistant sees data that lags the source system, it is usually because the system chose availability over consistency. The model is not at fault.
- Partition tolerance is mandatory. The only real choice is between C and A when the network fails.
- Strong consistency costs more: DynamoDB charges double for strongly consistent reads, and GSIs do not support them.
The customer’s head of operations sends you a screenshot. She has just cancelled an order in the order management system, then asked the AI assistant your team deployed about it. The assistant told her the order is “out for delivery”. Her question is simple: “Does this bot actually read real data?”
Many engineers assume the model made it up. Before you touch the prompt, check another possibility. The model may have read exactly what it was given, and what it was given was a copy that had not yet been updated.
That is why an FDE needs to understand the CAP theorem and consistency levels. The goal is not to pass an interview. It is to give that head of operations an honest answer and then fix the right thing.
What is CAP really asking you?
CAP has three letters, but the popular reading, “pick two of three”, is misleading. Henry Robinson’s CAP FAQ defines Consistency as linearizability: the system behaves as if it were running on a single machine. Robert Greiner’s version is easier to remember: a read always returns the most recent write.
Availability in CAP is also stricter than the everyday meaning. Every request sent to a non-failing node must get a response. The response may take as long as it likes, but an error does not count.
P, partition tolerance, is not something you choose. Greiner says plainly that networks are unreliable, so a distributed system has to tolerate partitions. The CAP FAQ goes further: if you describe your distributed database as “CA”, you have misunderstood something.
That leaves one real question. When the network between nodes is cut, should the system refuse to answer so that it stays correct (CP), or answer with the data it has, even if that data may be stale (AP)?
A healthy network does not end the story
One point is often missed. The CAP FAQ stresses that the theorem says nothing about whether partitions are common or rare. It describes the choice a system must make at the worst moment. It does not tell you how often that trade-off shows up in a day.
So keep two things apart. CP versus AP describes how the system behaves when the network is partitioned. Strongly versus eventually consistent reads, such as DynamoDB’s ConsistentRead flag covered below, is a setting on each read, and it applies even when the network is perfectly healthy.
In practice, this second kind of consistency is what customers notice day to day. Strong consistency guarantees that a read sees a change immediately, at the cost of lower availability and higher latency. It suits work like money transfers. Weak consistency makes no promise that the next read will see the change. In exchange, availability is high and latency is low.
Eventual consistency is a form of weak consistency. Updates will show up eventually, but until they do, several conflicting versions of the data can exist. Greiner names the mechanism: choosing AP means returning local data that may be stale. That is how an AI assistant ends up behind the source system.
Example: tracing a cancelled order
Picture this architecture. The customer’s order system writes to a DynamoDB global table in Region A. The AI assistant runs in Region B, close to its users, and reads from the Region B replica through a get_order_status tool.
AWS documentation states that with global tables in the default mode, changes are usually replicated to other replicas within one second, and they are only eventually consistent across Regions.
Suppose a staff member cancels the order at 10:00:00.000 and types a question to the assistant at 10:00:00.400. If the replica has not received the change, the tool reads “out for delivery”. The model then gives a faithful answer based on wrong data.
The same trap exists within a single Region. DynamoDB reads are eventually consistent by default, so a result may not reflect a write that has just completed. Read again a little later and the new version appears. If you want a strong read, you have to ask for it:
import boto3
table = boto3.resource("dynamodb").Table("orders")
def get_order_status(order_id: str, fresh: bool = False) -> dict:
resp = table.get_item(
Key={"order_id": order_id},
ConsistentRead=fresh, # True = strongly consistent, default is False
)
return resp.get("Item", {})
The documentation is explicit about the cost. An eventually consistent read costs half as much as a strongly consistent one, and Global Secondary Indexes do not support strongly consistent reads. If your tool looks up orders through a GSI keyed on the customer’s phone number, the fresh=True flag above will not save you. You have to look the order up again by its primary key.
For the cross-Region problem, AWS offers MRSC mode, which synchronously replicates writes to other Regions before returning. The cost then moves to the write side: the customer’s system waits longer so that the assistant reads more accurately.
Five steps to apply at a customer site
First, map the data path from where data is written to where the assistant reads it. Each hop (a replica, a cache, a search index, a nightly sync pipeline) is a place where data can fall behind. Record the observed lag for each one.
Second, classify the questions the assistant will receive. Classify them by business question, not by data table:
| Question type | Example | Consistency to use |
|---|---|---|
| Financial or legal consequences | “Has this order been refunded?” | Strong, read from the source |
| Operational status that just changed | “Has the order I just cancelled been cancelled?” | Strong, look up by primary key |
| Reference lookup | “What is the returns policy?” | Eventual is enough |
| Aggregate report | “How many orders were cancelled this week?” | Eventual, with an explicit timestamp |
Third, ask the customer instead of deciding yourself. Greiner says the choice between CP and AP depends on business requirements: choose consistency when you need atomic reads and writes, and availability when the business can tolerate replication lag. A useful question is: “If the assistant answers with data that is a few seconds old, what is the worst thing that could happen?”
Fourth, have the tool return the data’s timestamp too, and instruct the model to state it for sensitive questions, for example “according to data updated at 10:00:00”.
Fifth, accept that a CP system may return an error or time out while it waits for a partitioned node. Write a polite response for that case in advance instead of leaving the agent to guess.
Common mistakes
The most common mistake is blaming the model and editing the prompt. No prompt can make a stale replica fresh. Check what the tool returns before you touch the prompt.
The second mistake is turning on strong consistency everywhere “to be safe”. You pay double for reads and add latency, and on a GSI you cannot even turn it on. Most reference questions do not need it.
The third mistake is testing only in a single-Region dev environment, where replication lag is almost invisible. A bug like “the order I just cancelled is still out for delivery” only appears when a real user writes and then asks straight away, which internal demos rarely do.
The last mistake is forgetting that eventual consistency can produce conflicting versions. If the assistant is also allowed to write, say to change a delivery address, you need to know how the system resolves two overlapping writes before you give the agent that permission.
A short exercise
Pick a business you know well, such as the customer support bot for an e-wallet. Write down five questions users often ask. For each one, note three things: where the data lives, whether the assistant should read strong or eventual, and what happens if it reads data that is a few seconds old.
Then pick one question you placed in the strong group and ask yourself how you would redesign the tool if that lookup had to go through a secondary index. When you are done, you will have a table like the one in step two, ready for your first discovery session.
Putting this skill on your CV
When you read an FDE job description and see phrases like “integrate with customer systems of record” or “real-time data”, have a story ready about how you dealt with stale data. It is also a good moment to ask the employer a question in return: does their agent read from the source or from a replica?
On a CV, a line such as “classified the agent’s queries into 4 groups by freshness requirement and moved refund-related queries to strongly consistent reads” says more than any “proficient in distributed systems”.
The next time someone asks “does the bot read real data?”, don’t answer “yes” or “no”. Tell them when the data it reads was written, and ask whether they have agreed to that lag.
Was this article useful?
Thanks for the feedback!