> ## Documentation Index
> Fetch the complete documentation index at: https://docs.soriaanalytics.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Pipeline Overview

> How Soria moves data from public sources into your warehouse — two paths depending on file type.

Every data source in Soria flows through the same stages: discover files, organize them into groups, define what data you want, and load it into the warehouse. The exact steps differ depending on whether your source publishes PDFs/Excel or CSVs.

<Note>
  All pipeline changes — running scrapers, extracting data, publishing to the warehouse — require a workspace. Create one first with `workspace_manage(operation="create")`. A workspace isolates your changes so nothing reaches production until you promote it.
</Note>

## The two paths

<CardGroup cols={2}>
  <Card title="PDFs & Excel" icon="file-pdf" href="/pipeline/extraction">
    Unstructured files that need AI extraction. Gemini detects the right pages, extracts tabular data into CSVs, and validates accuracy before the data reaches your warehouse.
  </Card>

  <Card title="CSVs" icon="table" href="/pipeline/warehouse">
    Already-structured files. No extraction needed — the challenge is schema drift (column names that change across files). Schema mappings resolve this automatically.
  </Card>
</CardGroup>

***

## PDFs and Excel

Use this path when your source publishes reports, data books, or spreadsheets that aren't already machine-readable CSVs.

<Steps>
  <Step title="Scraper discovers and downloads files">
    A scraper fetches the source URL, finds downloadable files, and downloads them. Each file is stored with its URL, filename, and date.

    See [Scrapers](/pipeline/scrapers) for how to create and test scrapers.
  </Step>

  <Step title="Files are assigned to groups">
    After download, files are automatically matched to groups by regex pattern. A group is a named collection of files that share the same data structure — for example, all PDFs from a quarterly enrollment report.

    Groups are how the pipeline knows which schema and extraction settings to apply to a set of files. You define groups with `group_manage`.
  </Step>

  <Step title="Define the schema">
    You tell the pipeline what data to extract by defining schema columns on the group. These columns become the headers of the extracted CSV. Use `schema_manage` to set them.
  </Step>

  <Step title="Detection identifies relevant pages">
    Gemini scans each PDF or Excel file and finds the pages or sheets that contain data matching your schema. Only those pages are carried forward — irrelevant content is filtered out.

    Run detection with `detection_run`.
  </Step>

  <Step title="Extraction pulls data into CSVs">
    Gemini extracts the tabular data from the detected pages and writes it into a CSV using your schema columns as headers. Each source file produces a child CSV file.

    Run extraction with `extraction_run`.
  </Step>

  <Step title="Validation checks accuracy">
    Heuristics and Gemini verify the extracted CSV against the source document. Corrections are applied automatically where possible.

    Validation runs automatically after extraction. You can also trigger it manually with `validation_run`.
  </Step>

  <Step title="Value mapping normalizes extracted values">
    Because AI extraction can produce slightly inconsistent values for the same concept (for example: `"total"`, `"overall"`, and `"all"` for the same category), value mapping indexes all distinct values and lets you normalize them to canonical forms.

    Use `value_manage` to index values and define your mappings.
  </Step>

  <Step title="Publish to the warehouse">
    Load the extracted CSVs into the warehouse. Column and value mappings are applied automatically at publish time.

    See [Warehouse](/pipeline/warehouse) for how the data lands and how to query it.
  </Step>
</Steps>

***

## CSVs

Use this path when your source already publishes clean CSV files. No AI extraction is needed — the challenge is that column names often change across files or over time (schema drift).

<Steps>
  <Step title="Scraper discovers and downloads files">
    Same as the PDF/Excel path — a scraper finds and downloads files from the source URL.
  </Step>

  <Step title="Files are assigned to groups">
    Files are matched to groups by regex pattern. CSV groups work the same way as PDF groups.
  </Step>

  <Step title="Define the schema">
    You define canonical schema columns for the group — the authoritative names for each field regardless of what the CSV headers say.
  </Step>

  <Step title="Schema mappings resolve header drift">
    `schema_mappings` maps each source CSV header to one of your canonical schema columns. When a CSV arrives with a renamed column, the mapping ensures it lands in the right place. Soria suggests mappings automatically based on name similarity; you confirm or adjust them.
  </Step>

  <Step title="Publish to the warehouse">
    Load the CSVs into the warehouse. Mappings are applied at publish time so the warehouse always sees canonical column names.
  </Step>
</Steps>

***

## What are groups?

When a scraper downloads files, each file needs to be matched to the right processing configuration. Groups are how that works.

A **group** is a named collection of files that share the same data structure. You define a regex pattern on each group, and files whose names match that pattern are automatically assigned to it. For example, a group named `Enrollment PDFs` with pattern `enrollment.*\.pdf` would collect all enrollment PDF files.

Groups are also where you attach:

* A **schema** (the columns you want to extract)
* An **extractor** (for Excel/CSV files)
* **Value mappings** (for normalizing inconsistent values)

For PDFs, you typically create a parent group for the raw PDFs and a child group for each data type you want to extract from them. Multiple child groups can detect different data from the same parent files.

***

## Workspaces

A workspace is an isolated copy of your pipeline data. All write operations — running scrapers, extracting data, editing SQL models, publishing to the warehouse — happen inside a workspace. Nothing reaches production until you explicitly promote the workspace.

```
workspace_manage(operation="create", scraper_name="my_source")
```

When your changes are ready:

```
workspace_manage(operation="promote", workspace_id="<id>")
```