Skip to main content
Every data source in Soria flows through the same stages: discover files, organize them into groups, define what data you want, and load it into the warehouse. The exact steps differ depending on whether your source publishes PDFs/Excel or CSVs.
All pipeline changes — running scrapers, extracting data, publishing to the warehouse — require a workspace. Create one first with workspace_manage(operation="create"). A workspace isolates your changes so nothing reaches production until you promote it.

​
The two paths

PDFs & Excel

Unstructured files that need AI extraction. Gemini detects the right pages, extracts tabular data into CSVs, and validates accuracy before the data reaches your warehouse.

CSVs

Already-structured files. No extraction needed — the challenge is schema drift (column names that change across files). Schema mappings resolve this automatically.

​
PDFs and Excel

Use this path when your source publishes reports, data books, or spreadsheets that aren’t already machine-readable CSVs.
1

Scraper discovers and downloads files

A scraper fetches the source URL, finds downloadable files, and downloads them. Each file is stored with its URL, filename, and date.See Scrapers for how to create and test scrapers.
2

Files are assigned to groups

After download, files are automatically matched to groups by regex pattern. A group is a named collection of files that share the same data structure — for example, all PDFs from a quarterly enrollment report.Groups are how the pipeline knows which schema and extraction settings to apply to a set of files. You define groups with group_manage.
3

Define the schema

You tell the pipeline what data to extract by defining schema columns on the group. These columns become the headers of the extracted CSV. Use schema_manage to set them.
4

Detection identifies relevant pages

Gemini scans each PDF or Excel file and finds the pages or sheets that contain data matching your schema. Only those pages are carried forward — irrelevant content is filtered out.Run detection with detection_run.
5

Extraction pulls data into CSVs

Gemini extracts the tabular data from the detected pages and writes it into a CSV using your schema columns as headers. Each source file produces a child CSV file.Run extraction with extraction_run.
6

Validation checks accuracy

Heuristics and Gemini verify the extracted CSV against the source document. Corrections are applied automatically where possible.Validation runs automatically after extraction. You can also trigger it manually with validation_run.
7

Value mapping normalizes extracted values

Because AI extraction can produce slightly inconsistent values for the same concept (for example: "total", "overall", and "all" for the same category), value mapping indexes all distinct values and lets you normalize them to canonical forms.Use value_manage to index values and define your mappings.
8

Publish to the warehouse

Load the extracted CSVs into the warehouse. Column and value mappings are applied automatically at publish time.See Warehouse for how the data lands and how to query it.

​
CSVs

Use this path when your source already publishes clean CSV files. No AI extraction is needed — the challenge is that column names often change across files or over time (schema drift).
1

Scraper discovers and downloads files

Same as the PDF/Excel path — a scraper finds and downloads files from the source URL.
2

Files are assigned to groups

Files are matched to groups by regex pattern. CSV groups work the same way as PDF groups.
3

Define the schema

You define canonical schema columns for the group — the authoritative names for each field regardless of what the CSV headers say.
4

Schema mappings resolve header drift

schema_mappings maps each source CSV header to one of your canonical schema columns. When a CSV arrives with a renamed column, the mapping ensures it lands in the right place. Soria suggests mappings automatically based on name similarity; you confirm or adjust them.
5

Publish to the warehouse

Load the CSVs into the warehouse. Mappings are applied at publish time so the warehouse always sees canonical column names.

​
What are groups?

When a scraper downloads files, each file needs to be matched to the right processing configuration. Groups are how that works. A group is a named collection of files that share the same data structure. You define a regex pattern on each group, and files whose names match that pattern are automatically assigned to it. For example, a group named Enrollment PDFs with pattern enrollment.*\.pdf would collect all enrollment PDF files. Groups are also where you attach:
  • A schema (the columns you want to extract)
  • An extractor (for Excel/CSV files)
  • Value mappings (for normalizing inconsistent values)
For PDFs, you typically create a parent group for the raw PDFs and a child group for each data type you want to extract from them. Multiple child groups can detect different data from the same parent files.

​
Workspaces

A workspace is an isolated copy of your pipeline data. All write operations — running scrapers, extracting data, editing SQL models, publishing to the warehouse — happen inside a workspace. Nothing reaches production until you explicitly promote the workspace.
When your changes are ready: