Dagster: orchestrating pipelines by the tables they produce, not the steps they run

On 13 July 2026 Nick Schrock, Dagster’s founder, wrote on the company blog that Prefect would acquire Dagster Labs. Two tools that were routinely compared with Airflow now sit under one roof.

That makes it a good moment to ask what idea Dagster brought that an FDE needs to understand, whether or not the tool survives on its own.

The short answer is one word: asset. According to its official documentation, Dagster is a data orchestrator built for data engineers, with lineage and observability built in. What sets it apart is not the feature list. It is the unit the tool uses to look at a pipeline.

For FDEs this is a practical matter. ServiceNow’s Applied AI FDE job ad, posted on 21 October 2026, describes the role as building LLM applications that span backend logic, data orchestration and front-end interfaces. Orchestration is part of your job, not some other team’s.

A green pipeline does not mean correct data

Dagster’s blog argues that traditional orchestrators work at the level of the task. That lets them show the state of the pipeline, but not the state of the underlying assets, which are what users actually care about.

Dagster’s own comparison with Airflow puts it more bluntly: Airflow focuses on running tasks, while Dagster focuses on the assets those tasks produce.

Bear in mind that this is a vendor’s view, not a neutral assessment. According to the Apache Airflow documentation, Airflow is an open-source platform for developing, scheduling and monitoring workflows, and it does exactly that. The two approaches answer two different questions.

Imagine you are deploying an LLM assistant for the customer service team of a retail chain. The assistant answers from an order summary table that is refreshed every night.

One morning a manager asks why the assistant is reporting the wrong order status. In the task-based UI, every task is green. But the question you have to answer is a different one: when was the order summary table last updated, and from which version of the source data?

An example: renaming a pipeline from verbs to nouns

Dagster’s “Thinking in Assets” guide recommends shifting design from tasks to assets. The concrete rule is to name assets with nouns, after the object being produced, not after the computation step. Applied to the hypothetical retail pipeline above:

Task-based name	Asset-based name	Question the asset name helps answer
fetch_orders_from_crm	raw_orders	When did the raw data arrive?
clean_and_dedupe	cleaned_orders	Is the clean table newer than the raw one?
build_summary	order_summary	Which version is the assistant reading?
embed_and_index	order_summary_index	Does the index match the summary table?

In Dagster, each row in the middle column corresponds to a Python function. According to the documentation, that function computes the contents of the asset, such as a database table or a file. If an asset takes several steps, the @graph_asset decorator lets you bundle a multi-step pipeline and present it as a single asset.

Rename things this way and the debugging question changes too. You stop asking “which task failed?” and start asking “which asset is older than the asset it depends on?” The second is the question customers actually ask, even if they phrase it differently.

When should you choose Dagster at a customer?

Following that logic, Dagster fits best when most of the work is creating and maintaining tables: the data layer for RAG, feature tables, reports that an LLM application reads. If the customer mainly needs to schedule jobs that produce no data, such as calling APIs, sending emails or running maintenance jobs, Airflow’s task model is more natural.

Prefect takes a third path. Its documentation describes it as an open-source orchestration engine that turns ordinary Python functions into production data pipelines. For an FDE who needs to stand up a pipeline within a few weeks at a customer, that low-friction approach is valuable.

The difference from Dagster lies in the starting point. Going by each side’s own description, with Prefect you start from existing Python functions and turn them into a pipeline; with Dagster you start from the table or file you need to produce and then write the function that computes its contents.

Practical advice: do not propose switching orchestrators just because you like assets. If the customer already runs Airflow, name DAGs and tasks after their outputs, record which table depends on which, and you get most of the benefit of asset-based thinking without touching their infrastructure.

Limits and risks

The biggest risk now is the product roadmap. With Prefect acquiring Dagster Labs, look closely at the official direction before putting Dagster into a system the customer will have to run on its own for years.

The second signal comes from the job market. ServiceNow’s FDE job ad lists Airflow and Prefect alongside MLflow, Kubeflow, SageMaker and Vertex AI, but not Dagster. A single job ad is a small sample. Even so, if you are aiming for similar roles, prioritise Airflow and Prefect.

What to learn first

Learn the concept first and the tools second. Get solid on Airflow, the task-based benchmark that both Dagster and Prefect are measured against. Write a Prefect flow from a few Python functions you already have to see how they become a pipeline. Then build a small Dagster project with three or four assets so that asset-based thinking becomes a reflex.

On your CV and in interviews, do not just list tool names. Explain which table your pipeline keeps fresh, who uses it, and how you detect stale data. Telling it that way shows you think in terms of outputs the customer can use, not merely that you know a tool.

Do you already think in assets?

A quick self-test: open the most recent pipeline you wrote and answer three questions in one minute. Which table is the final output users read? When was it last updated? Which table does it depend on directly?

If you have to open each task’s logs to answer, your pipeline is still being viewed through tasks. Renaming with nouns and recording dependencies is the first step to fixing that.

Tools change owners, change names or merge. The habit of asking “which data is correct?” goes with you to every customer.