# Creating and Managing Flows

## Creating and Managing Flows

Flows are dex’s orchestration engine—a powerful, visual way to schedule, automate, and monitor your data pipelines. A Flow defines the order in which ingestion, transformation, and custom tasks run and provides complete visibility into how your data moves through your system.

This guide walks you through the process of creating, configuring, and managing Flows within dex.

### What is a Flow?

A **Flow** is a directed set of [**Nodes**](/lakehouse-platform/flows-and-automation.md) that execute data tasks in a defined order. Flows can run:

* Ingestion pipelines
* Model transformations
* Custom scripts and logic

Dependencies between tasks are inferred automatically based on model references—no manual DAG coding is required.

{% hint style="info" %}
 Think of Flows as dex's version of DAGs, but with smart automation and a no-code experience.
{% endhint %}

### Creating a New Flow

1. Go to the **Flows** section in the left-hand menu.
2. Click **New Flow**.

You’ll be taken to the Flow creation form. Fill in the following fields:

* **Name**: A required name that uniquely identifies your Flow.
* **Description** *(optional)*: A short explanation of what this Flow does (e.g., “Runs daily financial models and updates dashboards”).

#### Schedule

You can configure a schedule to automatically trigger this Flow at specific intervals.

You have two options:

**Basic Mode**

Use the dropdowns to create a simple recurring schedule:

* Select the frequency (`minute`, `hour`, `day`, etc.)
* Set the time (e.g., “Every day at 02:00 AM”)

A preview will appear below showing the exact timing.

**Advanced Mode**

Toggle the **Write cron expression** switch to input a custom cron expression directly.

For example:

* `0 2 * * *` – runs daily at 2:00 AM
* `0 */6 * * *` – runs every 6 hours

{% hint style="info" %}
Need help with cron syntax? Just ask any AI model to write it for you!
{% endhint %}

Once the form is filled out, click **Continue** to move to node configuration.

### Step-by-Step: Configuring Flow Nodes

Each Flow is composed of **Nodes**, which are responsible for executing actions like running models, ingesting data, or executing custom scripts.

#### 1. Choose a Node Type

Select from:

* **Transformation**: Executes models and data transformations
* **Connection**: Runs configured connectors for ingestion
* **Custom Script**: Executes your custom Python logic

#### 2. Configure Transformation Node Resources

When creating a transformation node, you must define which models will be included or excluded.

You can:

* Use model names directly
* Use tag selectors: `tag:staging`
* Use path selectors: `path:models/core/`

**Using dependency modifiers:**

* `+tag:finance`: includes upstream dependencies
* `tag:finance+`: includes downstream dependencies
* `+tag:finance+`: includes both directions

{% hint style="success" %}
You can include multiple values. Just press Enter or type a comma after each.
{% endhint %}

#### 3. Save the Node

After defining the node’s resources, click **Save**. The node will be added to your flow, and dex will automatically generate the corresponding tasks for each model selected.

You’ll see all tasks listed in the **Node Components** table with identifiers and dependency metadata.

### Examples

Consider a project with the following models:

```
models/
├── staging/
│   ├── stg_customers.sql   # tag: staging
│   ├── stg_orders.sql      # tag: staging
│   ├── stg_payments.sql    # tag: staging
│
├── marts/
│   ├── mart_revenue.sql    # tag: mart, revenue
│   ├── mart_customers.sql  # tag: mart
│   ├── mart_orders.sql     # tag: mart, orders
```

#### Example 1: Include All `staging` Models

```
Include: tag:staging

Resulting Models:
staging.stg_customers
staging.stg_orders
staging.stg_payments
```

#### Example 2: Include `mart` Models but Exclude `revenue`

```
Include: tag:staging
Exlude: tag:revenue

Resulting Models:
marts.mart_customers
marts.mart_orders
```

### Task Generation Process

When a Flow runs:

1. The **Include** and **Exclude** logic generates a list of selected models.
2. Each model is converted into an independent task within the Flow.
3. dex resolves dependencies automatically and executes models in the proper order.

### Creating a Custom Script Node

You can execute arbitrary Python logic in a Flow using a **Custom Script Node**.

#### Steps:

1. **Prepare your script**:
   * Create a directory named `dex-custom-scripts/`
   * Add a folder for your script (e.g. `data_transfer/`)
   * Inside it, create a file named `__main__.py`
   * Optional: Add a `requirements.txt` for dependencies
2. **Commit and push** these files to your dex environment.
3. In the Flow editor:
   * Create a new node
   * Select node type: `Custom Script`
   * Choose the script package you pushed
   * (Optional) Add parameters in key-value format
4. Click **Save**. Your custom script will now run as part of your Flow.

#### Accessing Parameters in Python:

```python
import sys

param1 = sys.argv[1]
param2 = sys.argv[2]

print(f"Parameter 1: {param1}")
print(f"Parameter 2: {param2}")
```

### Using Environment Variables in Scripts

If you're using **Custom Script Nodes**, you can access environment-specific [Variables and Secrets](/lakehouse-platform/variables-and-secrets.md) by using the standard Python `os.getenv()` function.

#### Example: Accessing Environment Variables in a Script

```python
import os

my_var = os.getenv("MY_VAR")
another_var = os.getenv("ANOTHER_VAR")
```

### Organizing Your Flow

A Flow can contain multiple **nodes** arranged to match your desired execution sequence. You can group related steps (e.g. finance, marketing, ingestion) into their own nodes and orchestrate them in the proper order.

dex provides a visual interface to drag, connect, and structure nodes for optimal clarity and execution flow.

This modular approach makes your pipelines easier to manage, monitor, and scale.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.dexlabs.io/lakehouse-platform/flows-and-automation/creating-and-managing-flows.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.