Flows and Automation

Key concepts of automation and scheduling using Flows

In dex, Flows are our native orchestration engine—the equivalent of DAGs (Directed Acyclic Graphs) in traditional data tools like Airflow or Dagster. Instead of writing orchestration code or manually managing dependencies, you build Flows visually by defining steps that represent ingestion, transformation, or custom scripts.

What makes dex different is that all dependencies between transformation models are automatically inferred based on their references (ref()). That means you can focus on building high-quality logic, and dex will:

• Automatically detect the correct execution order

• Group models into flow nodes

• Manage parallelism and dependencies

• Run everything in the correct sequence without manual DAG definition

You don’t have to wire hundreds of steps or write orchestration logic. Just build your models—and dex handles the flow.

Flows

A Flow is the primary orchestration unit in dex. It defines a sequence of operations that must be executed in a specific order. Flows can be triggered manually, on a schedule, or re-executed based on previous results.

Flows support automation for:

• Data ingestion pipelines (Connections)

• Data transformation processes (Transformations)

• Custom or ad hoc Python scripts

Each Flow maintains a status, which reflects the outcome of its most recent execution.

Nodes

Each Flow is composed of Nodes, which are logical steps within the orchestration. Nodes represent grouped tasks that will be executed as part of the Flow. These tasks can be dependent on one another—dependencies are automatically inferred by dex based on model references or explicit configuration.

A single Node can result in one or multiple Tasks depending on its type.

Node Types

Type	Description
Connection	Handles data ingestion—moving data from an external source into your cloud.
Transformation	Executes your SQL or Python models. A node will expand into one task per model.
Custom Script	Runs arbitrary scripts that don't fit into ingestion or modeling workflows.

Runs

A Run is a single execution of a Flow. It can be triggered manually or via a schedule, and it is responsible for orchestrating all associated tasks in order, based on node dependencies.

Runs provide full visibility into the status, logs, and output of each step in the automation.

Run Statuses

Status	Meaning
SCHEDULED	The run is planned for a future time.
QUEUED	The run is waiting for available resources.
RUNNING	The run is actively executing its nodes and tasks.
SUCCEEDING	Finalizing with successful results.
SUCCEEDED	All tasks completed successfully.
FAILING	Encountered errors and entering failure logic.
FAILED	The run was terminated due to errors.
ABORTED	The run was manually stopped before completion.
TIMED_OUT	The run exceeded its maximum execution time.
ABORTING	In the process of being intentionally terminated.

Run Types

Type	Trigger
ORCHESTRATED	Triggered automatically by a schedule.
MANUAL	Started by a user from the dex UI or API.
RERUN_ALL	Re-executes all tasks in a previous run.
RERUN_FAILED	Re-executes only the failed tasks.
RERUN_TASK	Re-executes a specific task from a past run.

Tasks

Tasks represent the actual work units that get executed during a Flow Run. Every Node in a Flow expands into one or more Tasks.

• A Connection Node maps to one Task per connector.

• A Transformation Node maps to one Task per model.

• A Custom Script Node maps to one standalone Task.

Tasks are executed in sequence or parallel depending on their upstream dependencies, ensuring correct data flow and logic.

Examples of Tasks:

• Ingesting data from a Salesforce connector

• Running the orders.sql transformation model

• Executing a cleanup or export script

Jobs

A Job is the underlying execution unit of a Task. It represents one attempt at running the logic defined in the task.

Jobs track:

• Execution start and end time

• Status (success/failure)

• Logs and errors

• Retry history

You can inspect Jobs to troubleshoot any step in your pipeline.

Summary of Concepts

Concept	Definition
Flow	The overall pipeline configuration and structure of steps to run
Node	A step in a Flow that represents a group of related tasks
Task	The specific action executed for a node (e.g. run a model, ingest a dataset)
Run	A full execution of a Flow (manual, scheduled, or retried)
Job	A single execution instance of a Task, including logs, status, and retry data