Shipping into a customer's cluster without losing sight of it

The contract is signed. The customer’s security team set just one condition: no document may leave their infrastructure. From now on your software will run on a cluster you do not own, behind a firewall you did not configure, and when it breaks at 2am the first to know may well be the customer.

Reducto, a document-processing software company, hires for exactly this role: Forward Deployed Engineer, Infrastructure Specialist. According to the job description, this person deploys Reducto into customer environments and owns the work end to end, from planning and configuration to testing and rollout.

If you want to be an FDE on the infrastructure side, this is the skill to practise first. Making something run on your own machine is the easy part. The hard part is making it run on someone else’s machine, and still knowing whether it is healthy or sick.

Who holds the data, who holds the compute?

Before typing a single command, you need to know which model the customer is buying, because the answer decides who holds the data, who runs the compute and where you will debug. Reducto offers three deployment models, of which VPC and on-premise are reserved for Enterprise customers. The two that infrastructure FDEs meet most often differ as follows.

	Hybrid VPC	Full VPC / on-prem
Data and storage	Stays in the customer’s cloud account	Inside the customer’s environment
Compute	Fully managed by Reducto, using ephemeral single-tenant workers that keep nothing beyond memory	Runs inside the customer’s environment
What you can see	Job records in Reducto’s database: status, request settings, where results are stored	Only what the customer allows, because the customer controls observability access and egress
Customer controls	The cloud account and the storage that receives results	Network, API access, Kubernetes policy, storage policy, observability, egress

The last row shows how the customer’s control grows, and the full VPC/on-prem column is the hardest case. With hybrid VPC, results are written to the customer’s storage but you still have job records to check status. With full VPC or on-prem, even the right to look at the system is something you have to ask for.

So the question to ask at kickoff is not only where the data lives, but also who grants you access to logs and metrics, and by which route signals are allowed out.

Seeing the system without touching the documents

Reducto’s security policy draws a clear line: telemetry is metadata-only observability, not a data pipeline. It may contain job IDs, counts, statuses, durations, sanitised error classes and other metadata used for debugging.

Responsibilities are split just as sharply. Reducto handles application security and tenant-level access; the customer handles external network exposure, caller authorisation, egress control, Kubernetes hardening, and the routing and retention of telemetry. In other words, where your telemetry goes and how long it lives is the customer’s decision.

Picture a telemetry event you might design for a document-processing job. The example below is hypothetical, not Reducto’s schema:

{
  "event": "job_completed",
  "job_id": "j_8f2c",
  "pages": 42,
  "file_type": "pdf",
  "duration_ms": 18350,
  "worker": "cpu-3",
  "status": "partial",
  "error_code": "TABLE_PARSE_TIMEOUT"
}

No file name, not a single line of text from the document. Yet when the customer reports that “yesterday’s contract file came back incomplete”, you can still filter for partial jobs, see that they all exceed 40 pages and share the same timeout error code. That gives you a hypothesis to test on your own sample data.

The lesson is that error classes must be specific enough. A generic error: "failed" field forces you to ask the customer to send the document, which is exactly what the contract forbids.

One upgrade, step by step

The first install is only the start, because an on-prem release has to be upgraded many times afterwards. Reducto’s on-prem changelog for v1.12.6 states that Helm requires Redis and healthy io/cpu workers, on v1.12.6 or later, before it allows an upgrade.

That sentence can be read two ways: either the workers must run images from v1.12.6, or the blocking mechanism only exists from chart v1.12.6 onwards. So do not hard-code one reading into your script. Ask the vendor, and keep the version check optional.

This blocking condition protects you, but it can also halt an upgrade in the middle of a maintenance window the customer has scheduled. Run the checks yourself before Helm gets the chance to refuse. The script below illustrates the idea; the namespace, labels, deployment names and the assumption that image tags carry a version number are all hypothetical:

#!/usr/bin/env bash
set -euo pipefail
NS="${1:-reducto}"
CHECK_VERSION="${CHECK_VERSION:-0}"   # set =1 once confirmed with the vendor
MIN="1.12.6"

echo "== Redis =="
redis_ready=$(kubectl -n "$NS" get pods -l app=redis \
  -o jsonpath='{.items[*].status.containerStatuses[*].ready}')
[[ -n "$redis_ready" && "$redis_ready" != *false* ]] \
  || { echo "Redis is missing or not ready"; exit 1; }

for role in io cpu; do
  echo "== worker $role =="
  d="worker-$role"
  ready=$(kubectl -n "$NS" get deploy "$d" -o jsonpath='{.status.readyReplicas}')
  want=$(kubectl -n "$NS" get deploy "$d" -o jsonpath='{.spec.replicas}')
  [ "${ready:-0}" = "$want" ] \
    || { echo "$d not ready: ${ready:-0}/$want"; exit 1; }

  if [ "$CHECK_VERSION" = "1" ]; then
    image=$(kubectl -n "$NS" get deploy "$d" \
      -o jsonpath='{.spec.template.spec.containers[0].image}')
    ver="${image##*:}"; ver="${ver#v}"
    lowest=$(printf '%s\n%s\n' "$MIN" "$ver" | sort -V | head -n1)
    [ "$lowest" = "$MIN" ] \
      || { echo "$d is on $ver, lower than $MIN"; exit 1; }
  fi
done

echo "Preflight OK, safe to run helm upgrade"

By default, the script stops when Redis has no pods or has a container that is not ready, or when a worker is short of replicas. Run it with CHECK_VERSION=1 and it also compares the worker image tag version against v1.12.6. It does not replace Helm’s own check, and if the customer uses tags such as latest or digests, you will have to read the version another way.

Part of the script’s value is that the customer can run it themselves. If the customer does not give you a kubeconfig, a diagnostic tool packed into a single command that prints readable results stands in for the access you do not have.

When the cluster has no route to the internet

Honeycomb splits this into two models: the vendor operates the platform itself, or the customer manages the entire infrastructure. The second model suits air-gapped environments, and there every signal you assumed would “phone home” has to take another route.

Replicated solves this by collecting telemetry from air-gapped installs and storing it in a Kubernetes Secret inside the customer’s environment. That data only reaches the vendor when someone generates a support bundle in the environment, and the local store has a size limit.

The practical consequence: if the customer generates a support bundle only once every few months, older signals may already have been pushed out of the store. For each stream, such as usage, logs or patches, ask the customer at the outset how it leaves, who carries it out and how often.

Try it: five tasks before rollout day

The first task is to pin down the model with the customer through three questions: where the data lives, where compute runs, and whether the cluster has a route to the internet. Record the answers in the kickoff document, signed off by the security team.

Next, draw up a table of who on the customer’s side owns each item: external network exposure, egress, Kubernetes policy, observability access, telemetry routing and retention. Every row needs a named person.

Then agree the list of telemetry fields in writing, so that the security team can review each field. After that, write a preflight for every upgrade operation and have the customer try it on staging. Finally, write an incident runbook that assumes you have no access: what commands the customer runs, and what output they send back to you.

The mistakes that delay a project by a month

The most common mistake is treating the customer’s environment like your own dev environment. You may run into internal registries, proxies, network policies and change-approval processes that the vendor’s documentation never mentions.

The next is leaving telemetry for later. Because the customer controls telemetry routing, adding a log field when the first incident hits may require another round of security review. The opposite mistake is over-logging: a single line that prints document content to stdout is enough to break the metadata-only promise on which the whole contract rests.

The last is treating an upgrade as a command. It is really a scheduled working session, with prerequisites and a rollback path. If you have not tested rollback on staging, you do not have a rollback path.

How to prove this skill when job hunting

When reading FDE job ads, watch for phrases such as “customer environments”, “on-prem”, “VPC” or “air-gapped”.

On your CV, do not write “experienced with Kubernetes”. Write a line such as: deployed and upgraded a system on a customer-operated cluster, with diagnostic tooling that the customer’s operators could run themselves.

If you have no real project yet, a repo with a Helm chart, a preflight script and a metadata-only telemetry schema is evidence an interviewer can read in five minutes.

The customer’s cluster will never be yours. You can still do your job well if you design to see it from day one.