# From data engineer to FDE: when the data belongs to the customer

> At your old company, you decided the schema. At the customer's site, the schema was there before you arrived, and it does not care what you prefer.

Original: https://fdetimes.net/en/guides/data-engineer-to-fde-customer-data/

Picture your first week at a customer site. You need to load supplier invoice data, and you receive three things at once: an export from SAP, an Excel file the finance team maintains by hand, and a folder of scanned PDFs. Nobody owns the schema, there is no data contract, and the demo for the leadership team is already on the calendar.

For a data engineer used to internal pipelines, this is a genuine shock. At your old company you controlled the sources and the formats, and if the data was wrong you filed a ticket with the upstream team. Here there is no upstream to blame, because the upstream is the customer.

The good news is that most of your skills still apply. Only the order of priorities changes, and FDE job postings describe that change with surprising clarity.

## The first job is finding the data, not writing the pipeline

Valdera's FDE posting says every customer is different: the data sits in different systems, takes different shapes and follows different rules. The systems to integrate are the customer's ERP, procurement and finance platforms, such as SAP, Oracle, Coupa and Ariba. That is not a data lake your team manages.

Rowspace goes further. Its FDEs work after the contract is signed, sitting with the customer and building integrations and pipelines directly on the customer's data, in the customer's environment.

They want people who are not put off by PDFs, Excel models and inconsistent formats. Vori, hiring a Forward Deployed Data Engineer, lists inputs ranging from CSV and EDI to PDF, and even scanned paper.

So the first question is not "Airflow or dbt?". You need to know where the data lives, who is editing it, and which business rules nobody has written down. The supplier codes in the Excel file, for example, may be old codes from before the SAP migration.

## Bad data must not bring down the whole run

Valdera states plainly what FDEs have to handle: messy formats, mismatched schemas, and a need for validation, retries and graceful degradation throughout. Those three terms capture the biggest break from internal habits.

Run the numbers. Suppose the first batch holds 1,000 invoices, of which 60 rows are missing a supplier code or record the amount as "1.250,00" instead of "1250.00".

Under internal habits, an assertion fails, the job stops and 0 invoices are loaded. The demo has nothing to show. The FDE way, 940 rows are loaded and 60 go into quarantine, each with a specific reason to send to the finance team.

The code below is a minimal skeleton for the two tabular sources in that scenario, the SAP export and the finance team's Excel file, once they have been read into rows. Each source has its own mapping, kept in configuration rather than hard-coded. That is also the part you will change most in the first few weeks.

```python
REQUIRED = ["vendor_id", "invoice_no", "amount", "currency"]

# One mapping per source; this is config, not code
SAP_MAPPING = {"vendor_id": "LIFNR", "invoice_no": "BELNR",
               "amount": "WRBTR", "currency": "WAERS"}

# Finance team's Excel: same targets, different column names
EXCEL_MAPPING = {"vendor_id": "Supplier Code", "invoice_no": "Invoice #",
                 "amount": "Total", "currency": "Curr"}

def normalize(row: dict, mapping: dict) -> dict:
    out = {target: row.get(src) for target, src in mapping.items()}
    amt = out.get("amount")
    if isinstance(amt, str):
        amt = amt.strip()
        if "," in amt and amt.rfind(",") > amt.rfind("."):
            amt = amt.replace(".", "").replace(",", ".")  # 1.250,00 -> 1250.00
        out["amount"] = amt.replace(",", "")
    return out

def validate(row: dict) -> list[str]:
    errors = [f"missing {f}" for f in REQUIRED if not row.get(f)]
    try:
        float(row.get("amount") or "x")
    except ValueError:
        errors.append(f"amount not numeric: {row.get('amount')!r}")
    return errors

def ingest(rows, mapping):
    good, quarantine = [], []
    for raw in rows:
        row = normalize(raw, mapping)
        errs = validate(row)
        if errs:
            quarantine.append({"raw": raw, "errors": errs})
        else:
            good.append(row)
    return good, quarantine

# Same function, two sources
sap_good, sap_bad = ingest(sap_rows, SAP_MAPPING)
xls_good, xls_bad = ingest(excel_rows, EXCEL_MAPPING)
```

The folder of scanned PDFs is outside this skeleton. It needs a separate extraction step to turn pages into rows first, and only then can it go through the same `ingest` function with a third mapping. The point worth keeping is the shared entry gate: every source passes through the same validation and quarantine layer.

Retries belong in the system-call layer, for instance when the ERP's API occasionally times out. You wrap that call in backoff and cap the number of attempts. Quarantine belongs in the data layer. Do not mix the two: retrying a malformed row ten times will not make it correct.

**Key point:** At a customer site, you do not command the data; you negotiate with it.

## Prototype first, harden later, reconcile forever

Valdera describes the working rhythm as prototyping quickly, then hardening whatever actually gets used into a solution robust enough for production. Paraform lists building custom integrations and pipelines, on timelines set by the deployment, among an FDE's core responsibilities.

For someone used to designing carefully before running anything, this rhythm will feel sloppy at first. In fact it simply reverses the order: you learn the data's rules by running against real data.

The reason is that demos do not reflect reality. A Stackmatix guide argues that customer data is messy and that real pipelines are full of edge cases no demo reveals. According to the same guide, evaluation happens in production: until something runs on the real system, nothing has been proven.

Hardening is not the end of the work. Vori treats configuring and maintaining data mappings and reconciliation as part of the operations team's job, not a one-off migration project. The customer changes its supplier codes, the finance team adds a column to the Excel file, and your mapping has to keep up.

Rowspace adds one more loop: the best of what you build for one customer should become a feature for all of them. The config-driven `normalize` function above is the seed of exactly such a feature.

## Hands-on: five steps for a new data source

When you meet an unfamiliar data source, follow this order.

**Step 1: Discovery.** Write one page setting out which system holds the data, who owns it, what format it is in, and which business rules users only describe out loud.

**Step 2: Measure before you transform.** Take a real sample and count the bad rows by error type before writing any transform.

**Step 3: Prototype with quarantine.** Build a prototype that separates valid and quarantined rows from the start, so you have something to show the customer within days.

**Step 4: Hand the quarantine list to the business owner.** They can fix at source the errors you could never have guessed.

**Step 5: Harden and package the mapping.** Harden only the paths the customer actually uses, and move the mapping into configuration so it can be reused next time.

## Common mistakes career switchers make

The most common mistake is treating one bad row as a failure of the whole job, the way quality was protected at the old company. The second is guessing business rules instead of asking. A blank "status" column might mean "paid" in that one department's process.

The third is pouring effort into an elegant architecture before touching real data. The fourth is about people. Vori states in its requirements that candidates need empathy for customers and the ability to stay calm while handling incidents in messy, high-pressure environments.

Palantir, the company that created the role, describes FDEs as often working directly on site with customers. There, how you explain the 60 quarantined rows matters as much as the code that separated them.

## Turning past experience into an FDE profile

Palantir includes processing data at very large scale in its description of an FDE's working day. In other words, a data engineering background is a real advantage, not something to hide. Paraform cites a Financial Times report that FDE job postings rose by more than 800% between January and September 2025, so demand for exactly this skill set exists.

But the CV has to speak the role's language. Instead of "built a Spark pipeline processing 2 TB/day", describe a time you integrated a data source you did not control, how you quarantined errors, and whom you sat down with to clarify the business rules.

Take the 1,000-invoice scenario above as a hypothetical example. The old-style CV line would read: "Built an ETL pipeline loading invoice data into the data warehouse." That is accurate, but an FDE recruiter sees no customer, no dirty data and no outcome.

Rewritten in FDE language: "Integrated invoice data from SAP and an Excel file managed by the client's finance team; designed a quarantine layer that loaded 940/1,000 rows on the first run and sent 60 failed rows with reasons to finance for correction at source; packaged the mapping as reusable configuration for a second source."

The second line carries all three signals: data you did not control, graceful degradation with numbers, and something reusable. Write your own version on that template for a real project, with real figures. When reading job postings, look for phrases such as "messy data", "reconciliation", "post-sale" and "on site": they tell you how much of your time will be spent with customers.

Internal pipelines teach you to make data correct. Customer data teaches you to keep the system useful while the data is not yet correct, and that is the skill companies pay to send on site.

**Try this week:**

- Take a real CSV or Excel file you have never seen and write an ingest step that returns two lists: valid records and quarantined records with the reason for each error.
- Read three FDE job postings, underline phrases such as 'messy data', 'reconciliation' and 'graceful degradation', then rewrite one CV line to show you have done exactly that work.
- Write a one-page discovery for a data source at your current company: which system holds it, who owns it, what format it is in, and which business rules are hidden inside it.

## Sources

- [Valdera - Forward Deployed Engineer](https://jobs.lever.co/valdera/d734a9de-3890-407c-b7ef-95576db4e604)

- [Forward Deployed Engineer (Rowspace)](https://talent.emcap.com/companies/rowspace/jobs/70242953-forward-deployed-engineer)

- [Forward Deployed Data Engineer (Vori, San Francisco, CA)](https://zerogtalent.com/space-jobs/vori/forward-deployed-data-engineer)

- [Palantir Technologies - Forward Deployed Software Engineer](https://jobs.lever.co/palantir/98b81271-9e15-40b7-8dce-0c62f24702f9)

- [What Is a Forward Deployed Engineer? GTM Guide for Startups](https://www.stackmatix.com/blog/forward-deployed-engineer)

- [What is a forward deployed engineer? A complete guide](https://www.paraform.com/insights/what-is-a-forward-deployed-engineer)