Skip to main content
Once files are extracted (or, for CSVs, directly after download), you publish them to the warehouse. The warehouse is where your data lives as queryable tables, organized into four layers that build on each other from raw to dashboard-ready.

​
The four layers

The warehouse uses a layered model where each layer has a specific purpose. You build SQL models to transform data as it moves up the layers.

Bronze

Raw source data. Files land here automatically after extraction. One table per group. No transformations applied — this is the data exactly as extracted.

Silver

Cleaned and typed data. Explicit type casts (no implicit conversions), no JOINs, one source per model. This is where you enforce data types and clean up inconsistencies.

Gold

Joined data. Combine silver models across sources here. Gold is where cross-source analysis becomes possible.

Platinum

Dashboard-ready aggregations. Platinum models can include chart configuration that controls how data appears in dashboards.
The pipeline: bronze → silver → gold → platinum

​
Publishing to bronze

When you run warehouse_manage(publish=True) for a group, all extracted CSVs for that group are loaded into a bronze table. Column mappings and value mappings are applied automatically at publish time — the warehouse always sees canonical names and values.
The output reports the table name, how many files were loaded, and the total rows inserted.
Publishing requires a workspace. Changes don’t reach production until you promote the workspace with workspace_manage(operation="promote", workspace_id="<id>").
To check the current state of a warehouse table:
Status shows the table name, column list, row count, and which files are currently loaded.

​
Rebuilding a table

If you need to rebuild a table from scratch — for example, after changing your schema columns — use force=True:
force=True drops and recreates the table before loading.

​
Querying the warehouse

Use warehouse_query to explore data at any layer using DuckDB SQL. This is how you profile data before writing SQL models and verify that published data looks correct.
Pass workspace_id to query a workspace’s data (where in-progress changes live). Omit it to query production.

​
Useful query patterns

Profile a table’s columns and statistics:
Check distinct values in a column:
Inspect data types:
Preview rows:
SUMMARIZE returns column-level statistics: min, max, approximate distinct count, null count, and more. It’s the fastest way to understand a new table.

​
Building SQL models

Bronze tables are automatically created by the pipeline. Silver, gold, and platinum models are SQL models that you write and save using sql_model_save. Each model is a SQLMesh SQL file with a MODEL block that defines its name, layer, and column descriptions. See SQL Models for how to write and save models. Silver model rules:
  • Explicit type casts on every column (CAST(col AS INTEGER) or col::INTEGER)
  • No JOINs — one source per model
  • Source CTEs must be SELECT * FROM source with no transforms
Gold model rules:
  • JOINs across silver models are allowed here
  • No aggregations — save those for platinum
Platinum model rules:
  • Aggregations and final calculations
  • Can include a @dashboard YAML annotation for chart configuration

​
Table history

Each time you publish and promote, the warehouse records a snapshot. You can inspect this history to understand when data changed and what workspace made the change:
The output shows each snapshot version with its timestamp, the workspace that created it, and whether it’s been promoted to production.

​
Removing data

To remove specific files from a warehouse table:
To drop an entire table:
Omitting file_ids drops the whole table.