Accessing Data Sources

Accessing Data Sources

Once a source has been synced to your environment, it becomes fully accessible for exploration, inspection, and usage within your models in dex. This includes sources automatically ingested by native Connectors or manually configured via Sync Sources.

Browsing Source Files

To view a synced source:

1. Navigate to the Develop section in the left-hand menu.

2. On the Explorer tab, locate the .yml file of the synced source dataset. These files describe the schema (metadata) of your external dataset.

3. Click the .yml file to open it in the editor. The file will display a structured list of tables and fields found in that dataset.

Inspecting Tables and Previewing Data

Table Description Menu

To inspect a specific table:

1. Right-click on the .yml file of the synced source.

2. Go to Table Description and choose the table you want to inspect.

3. A right-hand panel will appear showing:

   • Table name and last modified time

   • Storage metadata (partitioning, clustering, size, etc.)

   • Schema including field names, types, and nullability

   • Preview of the raw data

   • Execution logs

   • Lineage for tracking upstream and downstream dependencies

This feature allows you to explore datasets directly within dex, without switching tools or writing queries manually.

Referencing Sources in Models

When developing models, you can reference raw tables from external datasets using the source() function. This is helpful when your transformations depend on data that hasn’t been modeled yet, such as staging or bronze-level data.

To reference a source:

1. Right-click the .yml file

2. Navigate to Copy as Source and select the table you want to use.

3. dex will copy the correct syntax to your clipboard.

Example:

{{ source('dex-dsm-ecommerce_landing', 'orders') }}

Paste this reference into your transformation model to query the external data.

Example in a real query:

select
    order_status as order_status,
    safe_cast(order_delivered_customer_date as datetime) as order_delivered_customer_date,
    safe_cast(order_purchase_timestamp as datetime) as order_purchase_timestamp,
    safe_cast(order_estimated_delivery_date as datetime) as order_estimated_delivery_date,
    regexp_extract(customer_id, r'^.{0,3}') as customer_id,
    order_id as order_id,
    safe_cast(order_delivered_carrier_date as datetime) as order_delivered_carrier_date,
    safe_cast(order_approved_at as datetime) as order_approved_at
from 
    {{ source('dex-dsm-ecommerce_landing', 'orders') }}

Ref vs Source

When modeling data in dex, it's important to use the appropriate reference function:

• ref() is used to reference another model inside your project. It ensures proper dependency resolution and enables dex to build DAGs automatically.

  Example:

  {{ ref('customers_orders') }}

• source() is used to reference external datasets that are not defined as models (e.g., raw tables in your warehouse). It is used primarily in the bronze layer or when ingesting data.

  Example:

  {{ source('dex-dsm-ecommerce_landing', 'orders') }}

Use ref() for internal model dependencies and source() to bring in raw, external data.