When the client's architect says your design will fall over, turn the argument into a joint load test

The architecture review has been going for 40 minutes. You have just presented the design for a document search service to be deployed on the client’s side, and their lead architect folds his arms: “Every request goes synchronously through one database. It will fall over at peak.”

You believe the design is sound. He believes it will fall over. You are arguing about something that hasn’t happened yet, so neither of you can win with words, and every extra minute of arguing costs you a little more of the client’s trust.

Arguing better won’t help. Instead, invite him to help design the very test that could prove him right. This skill is what separates a good FDE from someone who can only write code: turning a disagreement into an experiment whose scoring both sides agreed on beforehand.

Why does a test cool the argument down?

Because it changes the question. “Will it fall over?” is a matter of belief. “At what load does it fall over, and which component breaks first?” is an engineering question, and every engineer wants to know the answer.

At Google, you have to know a system’s limits before you can run it safely. Their SRE book says plainly that engineers use stress tests to find the limits of a web service, and that load tests reveal the breaking point, which is basic knowledge for capacity planning.

Seen that way, the architect’s warning is not a nuisance. It is a test requirement nobody has written down yet.

When you say “You’re right, we don’t know where the breaking point is. Let’s find it together”, he stops being the critic and becomes a co-author. Whatever the result, he agreed to the method from the start.

Thresholds must be signed off before anyone presses run

The most damaging mistake is to run the test first and only then discuss what the results mean. By then each side reads the charts in its own favour. AWS Well-Architected lists exactly this as an anti-pattern: not defining clear, measurable performance requirements such as response time, throughput and scalability targets.

Grafana k6 has a built-in mechanism for this: thresholds, the pass/fail criteria you declare for each metric. The thresholds live in the test file itself, so what both sides agreed is kept in version control, not just in a line of the meeting minutes.

The second thing to agree is where to run it. AWS warns that a scaled-down test environment can mispredict behaviour in production. If you run on a tiny staging box and the system holds up, the architect is entirely justified in rejecting the result. So let him choose the environment.

A worked example from start to finish

Take the scenario above with some assumed numbers. After 20 minutes of work, the two sides agree that normal load is 50 concurrent users, and that the peak the architect is worried about is four times that. The pass threshold at normal load is p95 under 800 ms and an error rate under 1%.

The architect adds a question you hadn’t considered: if the system collapses at peak, how far must load drop before it recovers on its own? Google’s SRE book raises the same question: when some servers crash under heavy load, how far does load have to fall before the system stabilises again? So the scenario has three phases.

import http from 'k6/http';
import { check } from 'k6';
import { SharedArray } from 'k6/data';

// Real (anonymised) queries supplied by the client, not made up
const queries = new SharedArray('queries', () => JSON.parse(open('./queries.json')));

export const options = {
  scenarios: {
    baseline: { executor: 'constant-vus', vus: 50, duration: '15m' },
    stress: {
      executor: 'ramping-vus', startTime: '15m', startVUs: 50,
      stages: [{ duration: '10m', target: 200 }, { duration: '5m', target: 200 }],
    },
    recovery: { executor: 'constant-vus', startTime: '30m', vus: 25, duration: '10m' },
  },
  thresholds: {
    // Thresholds both sides signed off on: they apply only to normal load and the recovery phase
    'http_req_duration{scenario:baseline}': ['p(95)<800'],
    'http_req_failed{scenario:baseline}': ['rate<0.01'],
    'http_req_duration{scenario:recovery}': ['p(95)<800'],
  },
};

export default function () {
  const q = queries[Math.floor(Math.random() * queries.length)];
  const res = http.post(`${__ENV.BASE_URL}/search`, JSON.stringify({ q }),
    { headers: { 'Content-Type': 'application/json' } });
  check(res, { 'status 200': (r) => r.status === 200 });
}

Note that the stress phase has no thresholds. It isn’t there to pass. It is there to find the breaking point. k6 classifies tests by purpose, and a stress test is the kind you use to see how the system behaves at its limits, when load goes beyond the expected average.

AWS also lists failing to push load beyond expected limits as an anti-pattern.

Suppose the results come back like this: the baseline phase passes, p95 shoots up in the stress phase once users exceed about 140, and the recovery phase passes. Each of you was partly right. The design handles today’s load but has a ceiling, and now you can both see where that ceiling is.

The recovery phase here tests only one level, 25 users, so it tells you only whether the system recovers at that level. To answer the architect’s question in full, replace it with a ramping-vus that steps load down in several stages, say 150, 100, 50 and then 25, and record the first stage at which p95 returns below 800 ms.

Steps to run it yourself at a client

Start early. Ed Glas, in a post on a Microsoft blog, points out that many performance problems are architectural, not the kind you can fix just before going to production. If the architect is worried, test this week rather than in go-live week.

Load enough data before measuring, because, as Ed Glas notes, many performance problems only appear with large data volumes. A database with 10,000 records cannot test a warning about a database that will hold tens of millions.

Test each component first and the whole system after. Google SRE notes that each component has its own breaking point and needs its own load test, while Ed Glas suggests going from single transactions, to mixed transactions, and only then to the full system.

In the example above, keep the scenario as it is but point BASE_URL at an endpoint that only runs the database query. If it breaks before the full system does, you have found the culprit.

Finally, write it down. An ADR (Architectural Decision Record) records an architectural decision along with its reasoning, trade-offs and consequences. An ADR such as “keep the synchronous design, measured ceiling about 140 users, add a queue when real load reaches 100” turns the test result into a conditional commitment, so that three months later nobody has to restart the argument from scratch.

Mistakes that make a joint test backfire

The most common mistake is writing the scenario yourself and inviting the client to come and see the results. A client who didn’t help design the test won’t trust its results. Let the architect choose the peak load and the environment, even if you think his number is a little pessimistic.

The second mistake is using made-up queries. A list of five queries repeated over and over will all be cached, and p95 will look suspiciously good. Ask the client for a sample of real, anonymised queries.

The third mistake is treating the stress phase as an exam you must pass. If you avoid pushing the system to its breaking point, the architect will assume you are afraid of the answer. Finding the breaking point is a shared win, not your failure.

The last mistake is forgetting the recovery phase. A system that slows down at peak and then heals itself is very different from one that collapses and stays down. Ask the architect directly which case worries him more, and let his answer decide how carefully the recovery phase needs to be measured.

Turn it into a line on your CV

On your CV, don’t just write “performed load testing”. Write something like: designed a joint load test with the client’s architecture team, agreed p95 thresholds before the run, identified the database breaking point and recorded the decision in an ADR.

When interviewing for FDE roles, be ready to tell the story of one such episode, because it shows you can resolve technical disagreements with a client using numbers rather than volume.

This week’s exercise: recall the last time you disagreed with someone about performance. Write down the three numbers both sides should have agreed beforehand, plus a three-phase scenario to test them. The next time someone folds their arms and says “it will fall over”, you’ll have your answer ready: “Fine. Let’s find out together where it falls over.”