FDE Times dataFDE jobs open 8840 Change in open roles over 7 days, like-for-like: roles from companies we started tracking this week are left outNew in 7 days 53Companies hiring 185Remote-friendly 25%Median US pay $200kTop hirer Databricks 98
中文VI

The newspaper of the Forward Deployed Engineer

Tools

The best orchestrator is the one a client can run after you leave

Airflow scales but leaves a scheduler, a database and workers to maintain, while Luigi cannot even trigger itself without cron.

ShareLinkedInFacebookX

In brief

  • To run Airflow, the client has to maintain a scheduler, a metadata database (usually PostgreSQL or MySQL), an API server and workers
  • Prefect lets you write workflows in plain Python, and its workers can sit inside the client's own infrastructure
  • Luigi is batch-only, has no built-in triggers and is not designed to scale beyond tens of thousands of jobs
Numbered five-step diagram: trigger, processing type, scale, operational capacity and on-site data, applied to a retailer. Trigger: only a nightly schedule is needed, so cron plus Luigi is enough. Processing type: pure batch. Scale: 40 jobs, far below tens of thousands. Operational capacity: Airflow has four components to watch: metadata DB, scheduler, API server and worker. With Luigi and Prefect, you count and compare yourself. On-site data is a hard constraint. Arrows from questions 1 and 5 lead down to an orange box: when the client needs event-driven triggers and data must stay on-site, Prefect is the candidate to try first because its hybrid worker sits inside the client's infrastructure.
For 40 nightly batch jobs, cron plus Luigi is enough. When the client wants the pipeline to run the moment a file lands, the trigger question and the on-site data question together make Prefect the first candidate to try.

Luigi’s documentation says plainly that it has no built-in triggering. If you want a pipeline to run every night, you have to attach cron or something similar yourself.

Every time an FDE builds a data pipeline for a client, an orchestrator has to be chosen. Many engineers reach for the tool they know best. That misses something important: you stay for a few weeks, but the client’s operations team will live with the system for years.

So the question to ask is not which tool is most powerful but whether the client’s team can keep it running. All three popular options are job schedulers in the sense Wikipedia describes: software that controls background jobs running without anyone watching over them.

Where they differ is in the operating burden each one leaves with the client.

Airflow is powerful, but the client has to carry a whole system

Airflow’s homepage describes it as a community-built platform for authoring, scheduling and monitoring workflows as code. Pipelines are defined in Python, so they can be generated dynamically. Airflow has a modular architecture and uses a message queue to coordinate an arbitrary number of workers, which gives it a path to scale as load grows.

The cost is spelled out in the architecture documentation. An Airflow installation consists of a scheduler (which both triggers scheduled workflows and submits tasks to the executor), a metadata database, usually PostgreSQL or MySQL, that stores the state of tasks, DAGs and variables, an API server, and optional workers. Each of these components needs someone to monitor, upgrade and back it up.

If the client already has a platform team used to running PostgreSQL and hundreds of pipelines, Airflow is a sensible choice. If operations consists of two people doing it alongside other jobs, think carefully. Ask from the start: when the metadata database fills its disk at 2am, who deals with it?

Prefect: plain Python, with workers inside the client’s infrastructure

Prefect describes itself as an open-source orchestration engine that turns Python functions into production-grade data pipelines. Its documentation stresses that workflows are written in plain Python, with no DSL, YAML or special syntax. For a client team that already writes Python scripts, the learning curve is mostly conceptual.

Two further points matter to an FDE. Flows can be triggered on a schedule, by external events or through an API. And in the hybrid model, workers live in the client’s infrastructure and submit runs into that same infrastructure.

This is worth raising when the client is a bank or a hospital, where data must not leave internal systems. But workers are only one part of the picture. Before promising a client that Prefect is lighter than Airflow, list every other component your specific deployment needs to keep running.

Luigi states its own limits

Luigi is a Python package, tested on versions 3.10 to 3.13, for building complex pipelines of batch jobs. Its documentation lists three limits outright: it has no built-in triggering; it is focused on batch, so it is of little use for near real-time pipelines or continuously running processes; and it is not designed to scale beyond tens of thousands of jobs.

A tool that is this candid about its limits makes elimination easier for an FDE. If the client’s problem fits comfortably within those limits, Luigi deserves a place on the shortlist.

The next step is to count what a Luigi-plus-cron setup needs to run in the client’s environment and put that next to Airflow’s component list, rather than assuming it is lighter.

Applying this to a real client

Picture a retail chain with 40 batch jobs that run every night to aggregate sales and inventory. The operations team has two people, and data may not leave the company’s data centre. Walk through the five questions in the diagram in turn.

Triggering: only a nightly schedule is needed, so cron plus Luigi is enough. Processing style: pure batch, still squarely in the area where Luigi does well. Scale: 40 jobs is a long way from tens of thousands.

Operating capacity is where most care is needed. For Airflow the list is already clear: metadata database, scheduler, API server and workers, all looked after by two people. For Luigi and Prefect, you need to count the always-on components of the specific setup yourself before any comparison is possible.

Internal data is a hard constraint, to be kept in view and rechecked whenever requirements change.

Then the client adds that next quarter they want the pipeline to run automatically as soon as a supplier’s file arrives. That is triggering by an external event, which Luigi does not provide out of the box and Prefect supports. Going back to the internal-data question, Prefect’s hybrid workers sit inside the client’s infrastructure, so Prefect becomes the candidate to try first.

Airflow still has to answer the operating-capacity question: can two people look after those four components? The answer comes from sitting down with the client’s team, not from a feature comparison page.

Three common mistakes when choosing an orchestrator

The first is choosing by your own habits. The tool you know best is not necessarily one the client’s team can support at midnight.

The second is skipping the section on limits. Picking Luigi and only then discovering it has no built-in triggers, or putting it into a near real-time pipeline after the documentation warned against it, is an expensive price for one unread page.

The third is a lopsided comparison: carefully counting one tool’s components while ignoring the equivalent parts of another. The fix is simple. Before reaching a conclusion, draw up a table listing every process that must run continuously for each candidate.

What to learn first

If you only have time to learn one tool in depth, start with Airflow, because its architecture documentation names each component clearly: scheduler, executor, metadata database, workers. Once you can picture those parts, it becomes easier to see which pieces other tools leave to the user, such as Luigi handing triggering to cron.

When reading job descriptions for FDE or client-facing data engineering roles, notice whether they say just “Airflow” or “orchestration”. One might guess that the second wording suggests they need someone who can choose tools rather than merely use them, but that is only a guess, so ask directly in the interview.

On your CV, rather than listing tool names, write a sentence along these lines: chose Prefect over Airflow because the client needed event-driven triggering and data had to stay on-premises.

Once you have left the project, the client’s team will not remember which tool you chose. They will remember whether the system woke them up at 2am.

Was this article useful?

Use with your AI assistantAsk Claude ↗Ask ChatGPT ↗
7 sources
Read next on the roadmap · Stage 5: DeploymentWriting one OpenAPI spec that serves both humans and agentsAfter building more than 50 MCP servers from OpenAPI, Speakeasy found models got lost choosing among 200 tools; people and agents need different things from a spec.