Creating and Managing Flows

Creating and Managing Flows

Flows are dex’s orchestration engine—a powerful, visual way to schedule, automate, and monitor your data pipelines. A Flow defines the order in which ingestion, transformation, and custom tasks run and provides complete visibility into how your data moves through your system.

This guide walks you through the process of creating, configuring, and managing Flows within dex.

What is a Flow?

A Flow is a directed set of Nodes that execute data tasks in a defined order. Flows can run:

• Ingestion pipelines

• Model transformations

• Custom scripts and logic

Dependencies between tasks are inferred automatically based on model references—no manual DAG coding is required.

Think of Flows as dex's version of DAGs, but with smart automation and a no-code experience.

Creating a New Flow

1. Go to the Flows section in the left-hand menu.

2. Click New Flow.

You’ll be taken to the Flow creation form. Fill in the following fields:

• Name: A required name that uniquely identifies your Flow.

• Description (optional): A short explanation of what this Flow does (e.g., “Runs daily financial models and updates dashboards”).

Schedule

You can configure a schedule to automatically trigger this Flow at specific intervals.

You have two options:

Basic Mode

Use the dropdowns to create a simple recurring schedule:

• Select the frequency (minute, hour, day, etc.)

• Set the time (e.g., “Every day at 02:00 AM”)

A preview will appear below showing the exact timing.

Advanced Mode

Toggle the Write cron expression switch to input a custom cron expression directly.

For example:

• 0 2 * * * – runs daily at 2:00 AM

• 0 */6 * * * – runs every 6 hours

Need help with cron syntax? Just ask any AI model to write it for you!

Once the form is filled out, click Continue to move to node configuration.

Step-by-Step: Configuring Flow Nodes

Each Flow is composed of Nodes, which are responsible for executing actions like running models, ingesting data, or executing custom scripts.

1. Choose a Node Type

Select from:

• Transformation: Executes models and data transformations

• Connection: Runs configured connectors for ingestion

• Custom Script: Executes your custom Python logic

2. Configure Transformation Node Resources

When creating a transformation node, you must define which models will be included or excluded.

You can:

• Use model names directly

• Use tag selectors: tag:staging

• Use path selectors: path:models/core/

Using dependency modifiers:

• +tag:finance: includes upstream dependencies

• tag:finance+: includes downstream dependencies

• +tag:finance+: includes both directions

You can include multiple values. Just press Enter or type a comma after each.

3. Save the Node

After defining the node’s resources, click Save. The node will be added to your flow, and dex will automatically generate the corresponding tasks for each model selected.

You’ll see all tasks listed in the Node Components table with identifiers and dependency metadata.

Examples

Consider a project with the following models:

models/
├── staging/
│   ├── stg_customers.sql   # tag: staging
│   ├── stg_orders.sql      # tag: staging
│   ├── stg_payments.sql    # tag: staging
│
├── marts/
│   ├── mart_revenue.sql    # tag: mart, revenue
│   ├── mart_customers.sql  # tag: mart
│   ├── mart_orders.sql     # tag: mart, orders

Example 1: Include All staging Models

Include: tag:staging

Resulting Models:
staging.stg_customers
staging.stg_orders
staging.stg_payments

Example 2: Include mart Models but Exclude revenue

Include: tag:staging
Exlude: tag:revenue

Resulting Models:
marts.mart_customers
marts.mart_orders

Task Generation Process

When a Flow runs:

1. The Include and Exclude logic generates a list of selected models.

2. Each model is converted into an independent task within the Flow.

3. dex resolves dependencies automatically and executes models in the proper order.

Creating a Custom Script Node

You can execute arbitrary Python logic in a Flow using a Custom Script Node.

Steps:

1. Prepare your script:

   • Create a directory named dex-custom-scripts/

   • Add a folder for your script (e.g. data_transfer/)

   • Inside it, create a file named __main__.py

   • Optional: Add a requirements.txt for dependencies

2. Commit and push these files to your dex environment.

3. In the Flow editor:

   • Create a new node

   • Select node type: Custom Script

   • Choose the script package you pushed

   • (Optional) Add parameters in key-value format

4. Click Save. Your custom script will now run as part of your Flow.

Accessing Parameters in Python:

import sys

param1 = sys.argv[1]
param2 = sys.argv[2]

print(f"Parameter 1: {param1}")
print(f"Parameter 2: {param2}")

Using Environment Variables in Scripts

If you're using Custom Script Nodes, you can access environment-specific Variables and Secrets by using the standard Python os.getenv() function.

Example: Accessing Environment Variables in a Script

import os

my_var = os.getenv("MY_VAR")
another_var = os.getenv("ANOTHER_VAR")

Organizing Your Flow

A Flow can contain multiple nodes arranged to match your desired execution sequence. You can group related steps (e.g. finance, marketing, ingestion) into their own nodes and orchestrate them in the proper order.

dex provides a visual interface to drag, connect, and structure nodes for optimal clarity and execution flow.

This modular approach makes your pipelines easier to manage, monitor, and scale.