The demo works at your desk but fails on the customer's network: debugging DNS, proxies, CORS and cookies

In your first week on a customer site, an integration that ran smoothly on your laptop returns a 407, then a 504, then a red “CORS” line in the console. Not a single line of code has changed. The network has.

FDEs run into this all the time, because you are not deploying into a clean cloud that you control. You are deploying into someone else’s network. That network has a corporate proxy, firewalls, reverse proxies and SSO cookies, and a different team manages each one. The people who clear the blockage fastest are usually not the ones who know the most tricks.

They are the ones who can say which layer the fault is in before they start fixing anything.

How many parties handle a single request?

HTTP is a client-server protocol: requests are sent by a user agent, or by a proxy acting for it. MDN describes the many machines between client and server, collectively called proxies, that act as gateways or caches. Before a request can go anywhere, DNS, a hierarchical and decentralised naming system, has to turn the domain name into an address.

When you read a customer’s network diagram, first separate the two kinds of proxy. A forward proxy sits on the client side and hides the client’s identity. Bank staff go through this kind of proxy to reach the internet. A reverse proxy sits on the server side and hides the server’s identity. This is the layer in front of your API, or the customer’s own.

That gives you a four-layer model, with four questions to ask in strict order. Does the domain resolve to the right place from inside the customer network? Can the connection get out, or do the proxy and firewall block it? What HTTP code comes back, and who sent it?

The last question is for the browser: will it let the page read the response and send cookies?

The status code tells you who is answering

The most common mistake is to assume the backend sent every error code. A 407 looks like a 401, but it means authentication has to happen with the proxy. If you see a 407, the corporate proxy wants credentials, and the ticket belongs with the network team, not the application team.

502 and 504 also point to the middle layer. The machine answering you is acting as a gateway or proxy, and it got either a bad response or no timely response from upstream. Cloudflare adds a 520, a code that is not in the standard.

When you see a 504, measure the gateway’s timeout and how long your API takes before you think about optimising queries.

For HTTPS, a client behind an HTTP proxy has to open a tunnel with the CONNECT method. Not every proxy supports CONNECT, and some only allow port 443.

The connection layer has one more trap. Firewalls and middleboxes in general are hard to update, and some are configured to block any traffic carrying unfamiliar extensions. QUIC and HTTP/3 run over UDP, so they may be blocked on a customer network. Your client always needs a fallback to HTTP over TCP.

Symptom	Suspect layer	First check
Domain does not resolve, or points to the wrong IP	Layer 1: DNS	Run nslookup from a machine inside the customer network
CONNECT refused	Layer 2: connection through proxy, firewall	Is the destination port 443, and is the host on the allowlist?
407	Layer 3: HTTP code, returned by the forward proxy	Ask the network team how to authenticate with the proxy
502, 504, 520	Layer 3: HTTP code, returned by a gateway or reverse proxy	Gateway timeout compared with API processing time
CORS error, cookies not sent	Layer 4: browser	Console and Network tab in DevTools: the preflight response, the Cookie header, cookie attributes (curl cannot reproduce this layer)

Example: a widget embedded in a bank’s portal

Take a hypothetical case. The customer is a bank with an internal portal at portal.khachhang.vn. You embed a widget from app.congty-ban.com, and the widget calls the API at api.congty-ban.com using a session cookie. Everything works on your laptop, but on bank staff machines the widget is blank.

Sit at a machine inside the customer network and work through the layers with three commands. Because the widget sends a POST with a JSON body, the preflight simulation must also declare the Content-Type header, exactly as the browser would:

# 1. Từ trong mạng khách, tên miền phân giải ra đâu?
nslookup api.congty-ban.com

# 2. Đi qua proxy doanh nghiệp: xem dòng CONNECT và mã proxy trả về
curl -v -x http://proxy.khachhang.vn:8080 https://api.congty-ban.com/health

# 3. Giả lập preflight giống hệt trình duyệt
curl -v -X OPTIONS https://api.congty-ban.com/v1/data \
  -H "Origin: https://portal.khachhang.vn" \
  -H "Access-Control-Request-Method: POST" \
  -H "Access-Control-Request-Headers: Content-Type"

(The comments read: 1. From inside the customer network, where does the domain resolve? 2. Through the corporate proxy: look for the CONNECT line and the code the proxy returns. 3. Simulate the preflight exactly as the browser sends it.)

If command 2 returns a 407, you know who to talk to. If command 2 works but command 3 gets a 403 from a firewall error page, you have found the culprit.

MDN stresses that a preflight never carries credentials. A security layer that demands a cookie on every request will therefore block the OPTIONS call too. The browser then reports a CORS error, even though your backend has no fault at all.

Suppose the diagnosis ends there, and the Application tab also shows that the session cookie does not declare SameSite. The root cause then lies in two places outside the code. The firewall blocks the preflight because it carries no cookie, and the cookie is treated as Lax, so it is not sent with requests from the iframe.

The fix is to ask the network team to let OPTIONS requests to api.congty-ban.com through without requiring a cookie, and to reset the cookie with SameSite=None; Secure, as described below.

The final step is to confirm on a bank staff machine, in the browser rather than with curl. In the Network tab, the OPTIONS preflight must succeed with the Access-Control-Allow-* headers, the real POST request must carry a Cookie header, and the console must show no red lines.

CORS and cookies: the browser is the last gatekeeper

CORS works through HTTP headers, which a server uses to declare which origins may read its data from a browser. HTTP itself is stateless, and cookies are what create a stateful session. The catch is that when you call fetch or XHR across origins, the browser does not send credentials by default, which means it does not send cookies.

For the widget in the example, the client has to enable credentials explicitly:

fetch('https://api.congty-ban.com/v1/data', {
  method: 'POST',
  credentials: 'include',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify(payload),
});

On the server side, the response must include Access-Control-Allow-Credentials: true before the real request can use the cookie. Name the portal’s origin explicitly rather than leaving the door wide open:

Access-Control-Allow-Origin: https://portal.khachhang.vn
Access-Control-Allow-Credentials: true
Set-Cookie: session=abc123; SameSite=None; Secure; Domain=congty-ban.com

The Set-Cookie line is the part people tend to overlook. A cookie that does not declare SameSite is treated as Lax. In a cross-site iframe such as the bank portal, that is often exactly why the session “disappears”. SameSite=None only works if Secure is set as well.

The Domain attribute decides which servers receive the cookie. Setting congty-ban.com means the cookie reaches that server and its subdomains, such as api and app.

Mistakes that cost you a week

The first is trusting the X-Forwarded-For header. When your API sits behind the customer’s reverse proxy, you will be tempted to take the real IP from this header for rate limiting or access control. MDN is clear that any security-related use must rely only on IPs added by a trusted proxy.

If you naively take the first value, anyone who sends a forged header can get past your limits.

The second is a vague request to “open the firewall”. The bank’s network team needs to know the exact host, port and protocol, and that the OPTIONS method must be allowed through. A request written around the four-layer table above will be approved far faster than “the API is broken, could you take a look?”

The third is fixing the backend because you saw a 504. First establish which machine returned it. The last is assuming the client can use HTTP/3. The customer’s firewall may be configured to block unfamiliar traffic, and QUIC, which runs over UDP, may well be part of that.

For developers moving into FDE roles, this is a skill that is easy to demonstrate. On your CV, do not write a generic line about “troubleshooting network issues”. Describe one incident layer by layer: what the symptom was, how you narrowed it down to DNS, the proxy, the HTTP code or the browser, and how you fixed it.

When a job description mentions “enterprise network”, “on-premise” or “customer environment”, have a story like that ready for the interview.

Next time a widget goes blank on a customer site, do not open the code first. Open a terminal, work through the four layers in order, and take the evidence to the team that has to fix it.