dbt: Models, ref() & the DAG

dbt (data build tool) transforms data already loaded into a warehouse -- handling the 'T' in an ELT pipeline. Analysts write transformation logic as SQL SELECT statements; dbt handles dependency ordering, materialization, testing, and documentation around them.

Models & ref()

-- models/staging/stg_orders.sql
SELECT
  id,
  customer_id,
  order_date,
  status
FROM {{ source('raw', 'orders') }}

-- models/marts/fct_orders.sql
-- ref() compiles to the fully-qualified table name at run time --
-- dbt automatically knows this model depends on stg_orders, and
-- infers the correct build order from this reference
SELECT
  o.id,
  o.customer_id,
  o.order_date,
  c.name AS customer_name
FROM {{ ref('stg_orders') }} o
JOIN {{ ref('stg_customers') }} c ON o.customer_id = c.id

Sources: Declaring Raw Upstream Tables

# models/staging/_sources.yml
# Declares raw tables loaded upstream by a separate tool (e.g. Fivetran)
# -- gives dbt full lineage visibility, from raw source to final model
version: 2

sources:
  - name: raw
    tables:
      - name: orders
      - name: customers

The DAG & Running Models

dbt run                    # build all models in dependency order
dbt run --select stg_orders+  # run stg_orders and everything downstream
dbt test                   # run configured data quality tests
dbt docs generate          # build a browsable docs site with lineage graph

The DAG (dependency graph) is inferred entirely from ref() and source() calls scattered across the project's SQL -- no separate graph-definition step is needed.