# Develop with dex

In dex, developing data pipelines is built around structured, modular, and version-controlled workflows. Whether you're transforming raw data into clean, analytics-ready datasets or orchestrating complex dependencies across projects, dex offers a powerful developer experience designed for scalability and collaboration.

At the core of dex development is the concept of writing **`models`** that define transformations, enriched with a set of tools and patterns to support data quality, reusability, governance, and automation. These tools include tests, documentation, seeds, snapshots, macros, variables, and more—all working together in a unified development framework.

Development in dex happens through Git-based workflows. Each environment is tied to a Git branch, and all changes go through commit-based versioning. This ensures every deployment is traceable, reproducible, and aligned with your organization’s software development lifecycle.

Below is a summary of the key building blocks you’ll use when developing with dex:

### Core Development Concepts

<table data-column-title-hidden data-view="cards"><thead><tr><th>Concept</th><th>Descripti</th><th data-hidden data-card-target data-type="content-ref"></th><th data-hidden data-card-cover data-type="files"></th></tr></thead><tbody><tr><td>Sync Sources</td><td>Configure access to external datasets already present in your cloud storage. Automatically detect schema and register sources for development.</td><td></td><td></td></tr><tr><td>Accessing Data Sources</td><td>Explore table schema, data previews, logs, and lineage directly within dex. Easily reference external sources in your models using the <code>source()</code> function.</td><td></td><td></td></tr><tr><td><strong>Core Execution Actions</strong></td><td>Preview, Run, Test, and Build—core actions that allow you to inspect, validate, and materialize models during development and execution.</td><td><a href="/pages/UbsBXebmZC0V6Ysn82KM">/pages/UbsBXebmZC0V6Ysn82KM</a></td><td></td></tr><tr><td><strong>Models</strong></td><td>SQL or Python files that define how raw data should be transformed into clean, structured outputs. Each model is typically a <code>SELECT</code> statement saved as a <code>.sql</code> file.</td><td><a href="/pages/cy930qe0HlEBML4p5iiA">/pages/cy930qe0HlEBML4p5iiA</a></td><td></td></tr><tr><td><strong>Layers</strong></td><td>Structured stages of transformation (e.g., raw, cleaned, trusted) that help organize data pipelines for clarity, scalability, and governance.</td><td><a href="/pages/fqrJFMBNB71PWUyy1J9F">/pages/fqrJFMBNB71PWUyy1J9F</a></td><td></td></tr><tr><td><strong>Tests</strong></td><td>Assertions that validate data quality, such as uniqueness, non-nullness, referential integrity, or custom conditions. Tests help catch issues early in the pipeline.</td><td><a href="/pages/uDZMpgWozGG7HG2Dli3O">/pages/uDZMpgWozGG7HG2Dli3O</a></td><td></td></tr><tr><td><strong>Documentation</strong></td><td>Human-readable descriptions added to models, columns, and resources. Enables automatic generation of data catalog and lineage tools.</td><td><a href="/pages/FxFmP67QBL6Gx9vXucjC">/pages/FxFmP67QBL6Gx9vXucjC</a></td><td></td></tr><tr><td><strong>Seeds</strong></td><td>Static data tables loaded from CSV or Parquet files. Useful for reference data like country codes, mappings, or product lists.</td><td><a href="/pages/idTtZCckpLUqVqyrIvB8">/pages/idTtZCckpLUqVqyrIvB8</a></td><td></td></tr><tr><td><strong>Snapshots</strong></td><td>Track changes over time in slowly changing dimensions (SCD). Snapshots allow capturing and diffing changes in source records.</td><td><a href="/pages/nnppTxUyLaO5dhwHYOlL">/pages/nnppTxUyLaO5dhwHYOlL</a></td><td></td></tr><tr><td><strong>Jinja and Macros</strong></td><td>Jinja is a templating language that allows dynamic SQL generation. Macros are reusable functions written in Jinja to standardize logic across models.</td><td><a href="/pages/zEYTZ82ibD95rhknrlMR">/pages/zEYTZ82ibD95rhknrlMR</a></td><td></td></tr><tr><td><strong>Project and Environment Variables</strong></td><td>User-defined variables that allow parameterizing model logic and configuration. Can vary by project or environment and support secure storage.</td><td><a href="/pages/XpQUL23aALDSTbZN8zYx">/pages/XpQUL23aALDSTbZN8zYx</a></td><td></td></tr><tr><td><strong>Packages</strong></td><td>Collections of models, macros, and other assets that can be reused across projects. Enables modularity and sharing of business logic.</td><td><a href="/pages/MPnrmir9LKGqFVfv0ePu">/pages/MPnrmir9LKGqFVfv0ePu</a></td><td></td></tr><tr><td><strong>Hooks and Operations</strong></td><td>Custom scripts triggered before or after model execution. Common use cases include granting permissions, sending alerts, or refreshing external systems.</td><td><a href="/pages/envtj0kyTViavopEmEJf">/pages/envtj0kyTViavopEmEJf</a></td><td></td></tr><tr><td><strong>Tags</strong></td><td>Lightweight metadata used to group or classify models. Useful for running subsets of models (e.g., nightly jobs or finance-related models).</td><td><a href="/pages/10QJltQYkMN6MOfIDqbm">/pages/10QJltQYkMN6MOfIDqbm</a></td><td></td></tr></tbody></table>

Explore each concept in the sections that follow to learn how dex empowers scalable, collaborative, and production-grade analytics engineering.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.dexlabs.io/lakehouse-platform/develop-with-dex.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.