Skip to main content
Extraction is the process that turns unstructured PDFs and Excel files into clean, structured CSVs. CSVs skip this entire section — if your source publishes CSVs, see Warehouse for what happens next. The extraction pipeline has four stages: detection, extraction, validation, and value mapping. They run in sequence on each file.

​
Detection

Before Soria can extract data from a PDF, it needs to know which pages contain the data you care about. A report might be 80 pages long, but only pages 12–15 contain the enrollment table you want. Detection uses Gemini to scan each file and identify the pages (for PDFs) or sheets (for Excel) that contain data matching your group’s schema columns. Those pages are collected into a sub-file that carries only the relevant content forward to extraction. For PDFs, detection works on child groups. You create a child group under the parent group that holds the raw PDFs, and detection writes the detected sub-PDFs into that child group. This lets you extract multiple different data types from the same set of PDFs — one child group per data type.
Detection is idempotent — it skips files that have already been processed. Use force=True to re-detect.
You must define schema columns on the group before running detection. The schema tells Gemini what kind of data to look for.

​
Extraction

Once detection has identified the relevant pages, extraction uses Gemini to pull the tabular data out of those pages and write it into a CSV file. The output CSV uses your schema columns as headers. Because the schema is defined upfront, there’s no schema drift at extraction time — the columns are always canonical.
Each source file produces a child CSV file. Extraction is idempotent — it skips files that already have a CSV. Use force=True to re-extract.

​
Testing extraction

Before running extraction on all files, you can test your extraction setup against a single file:
Test mode extracts and validates without creating any CSV files or database records. The output shows the row count, column names, validation results, and a sample of the extracted data.

​
Excel and CSV extractors

For Excel and CSV files, extraction uses a Python class called a SimpleExtractor rather than AI. You implement an extract(reader) generator method that yields DataFrames in chunks.
The reader provides two methods:
  • reader.read(n_rows=N) — read the first N rows as a raw DataFrame (no headers, all strings), useful for peeking at structure
  • reader.iter(start_row=N, columns=[...], chunk_size=M) — yield DataFrames in chunks starting at a given row
Save and manage extractors with extractor_manage. Only polars is available for DataFrames — pandas is not installed.

​
Validation

After extraction, validation cross-checks the extracted CSV against the source document. It uses a combination of heuristics and Gemini to verify that the data looks correct and applies corrections where possible. Validation runs automatically after extraction. You can also run it manually — for example, to re-validate after fixing an extraction issue:
Like extraction, validation is idempotent. Use force=True to re-validate files that already have results.

​
Value mapping

When AI extracts data, it can produce slightly inconsistent values for the same underlying concept. For example, an unlabeled “total” row in a table might be extracted as "total", "overall", or "all" depending on the context. These are all the same thing, but they’d appear as three separate values in your warehouse. Value mapping solves this. It has two steps: 1. Index values — scan all extracted CSVs for a given column and collect every distinct value found.
2. Map values — normalize inconsistent values to canonical forms. When you map one value to another, the target automatically becomes the canonical. You don’t need to create canonicals upfront.
The read output shows:
  • Canonicals — normalized forms that other values will be mapped to
  • Unmapped — values not yet assigned to a canonical, with similarity suggestions
  • Mapped — values that have been normalized, showing the canonical they map to
To apply mappings:
Value mappings are applied automatically when you publish to the warehouse — you don’t need to re-run extraction.

​
Schema drift for CSVs

CSVs skip detection, extraction, and validation entirely. But they have their own challenge: schema drift — column names that change across files or over time. For example, a state Medicaid enrollment file might use "Plan Name" in 2022 and "MCO Name" in 2024. Both mean the same thing, but they’d land in different columns without schema mappings. schema_mappings handles this by mapping source CSV headers to canonical schema columns. Soria suggests matches automatically based on name similarity. You confirm the suggestions and manually map anything it can’t match.
The read output shows current mappings, unmapped schema columns, and unmapped source headers with similarity-based suggestions. To apply a mapping:
When a new CSV arrives with a column that’s already mapped, it’s handled automatically. When a new unmapped header appears, scraper_run reports it as schema drift so you can map it before publishing.