Detection & Extraction

Detection and extraction are the two AI processing steps that turn raw files into structured data. Detection locates the pages or sheets that contain schema-matching data. Extraction reads those pages and writes the data into CSV records aligned with your schema.

Pipeline order

Run these steps in sequence for each group:

group_manage

Create the group and set the file pattern that assigns files to it. For PDFs, create a child group — detection reads files from the parent and writes sub-PDFs into the child.

schema_manage

Define the schema columns for the group. The AI uses these during both detection and extraction to understand what data to look for.

detection_run

Identify which pages (PDFs) or sheets (Excel) contain data that matches your schema. Produces filtered sub-files in the child group.

extraction_run

Extract tabular data from the detected files into CSV records aligned with your schema columns.

CSVs skip detection entirely — Soria processes them directly with a programmatic extractor. Detection applies only to PDFs and Excel files.

group_manage

Organize files into groups by regex pattern. Groups are the unit that schemas, detection, and extraction operate on. For PDF pipelines, you need a child group: the parent group holds the original PDFs, and the child group holds the detected sub-PDFs (one per relevant page range). Multiple child groups can extract different data types from the same parent PDFs.

string

The group to read, update, or delete. Required for read, update, and delete.

string

The scraper that owns the group. Required when creating top-level groups.

boolean

default:"false"

Return the group’s details: name, pattern, file list, and child groups.

object[]

Create one or more groups. Each object requires name and pattern:

[{ "name": "Benefit Tables", "pattern": "benefit.*\\.pdf" }]

Patterns are Python regex applied to file names. Pass group_id to create child groups under an existing group, or scraper_id to create top-level groups.

object

Update the group’s name, pattern, or both. Changing the pattern re-applies it to all files in the scraper.

boolean

default:"false"

Delete the group. Files are retained — their group_id is set to null.

string

Required for create, update, and delete operations.

Examples

{
  "scraper_id": "scraper-uuid",
  "workspace_id": "workspace-uuid",
  "create": [
    { "name": "Rate Filings", "pattern": "rate_filing.*\\.pdf" }
  ]
}

{
  "group_id": "parent-group-uuid",
  "workspace_id": "workspace-uuid",
  "create": [
    { "name": "Benefit Tables", "pattern": "" }
  ]
}

{
  "group_id": "group-uuid",
  "read": true
}

detection_run

Scan PDFs and Excel files to identify which pages or sheets contain data that matches your group’s schema. For PDFs, Soria creates a filtered sub-PDF for each relevant page range and stores it in the child group. Extraction then runs against these sub-PDFs. Detection is idempotent — files that already have detection results are skipped unless you pass force=True.

For PDFs, group_id must be a child group. Detection reads files from the parent group and writes results into the child group. Passing a top-level group will return an error.

Parameters

string

required

The workspace to run detection in.

string

required

The group to run detection for. Must be a child group for PDF files.

string[]

Limit detection to specific file IDs. Defaults to all files in the group.

string

The prompt to use for detection. Defaults to the group’s configured prompt.

string

Ad-hoc instructions passed to the AI in addition to the prompt. Use this to guide detection for a particular run without changing the saved prompt.

boolean

default:"false"

Re-detect files that already have detection results.

Examples

{
  "workspace_id": "workspace-uuid",
  "group_id": "child-group-uuid"
}

{
  "workspace_id": "workspace-uuid",
  "group_id": "child-group-uuid",
  "file_ids": ["file-uuid-1", "file-uuid-2"]
}

{
  "workspace_id": "workspace-uuid",
  "group_id": "child-group-uuid",
  "force": true,
  "instructions": "Focus only on pages containing premium rate tables, not summary pages."
}

If detection is taking a long time, check the server logs before retrying. Large PDFs and API rate limits are common causes of slow runs. Try with a single file_id first to verify detection is working correctly.

extraction_run

Extract tabular data from files into structured CSV records aligned with your schema columns. Extraction creates child file records containing the extracted CSV data. Soria routes each file to the appropriate extraction method automatically:

File type	Extraction method	Requirements
PDF	AI extraction via Gemini	`schema_manage` must define columns
Excel / CSV	Programmatic extraction via `SimpleExtractor`	`extractor_manage` must define an extractor

Extraction is idempotent — files that already have a CSV are skipped unless you pass force=True.

Parameters

string

required

The workspace to run extraction in.

string

The group containing files to extract. Required unless running in test mode.

string[]

Limit extraction to specific file IDs. Defaults to all files in the group. In test mode, pass exactly one file ID.

string

Prompt to use for AI extraction on PDFs. Defaults to the group’s configured prompt.

string

Ad-hoc extraction instructions for PDF AI mode. Passed to the AI alongside the prompt without modifying saved settings.

boolean

default:"false"

Re-extract files that already have a CSV.

string

Name of the programmatic extractor to use for Excel/CSV files. Defaults to the group’s default extractor.

string

Python extractor code to run directly without reading from storage. Used for testing changes before saving.

boolean

default:"false"

Dry-run mode. Extracts and validates but does not create CSV records or write to the database. Returns a sample of the output and any validation errors.

string

default:":5,-5:"

Slice syntax controlling how many rows appear in test mode output. Examples:

:10 — first 10 rows
-10: — last 10 rows
:5,-5: — first 5 and last 5 rows (default)

Examples

{
  "workspace_id": "workspace-uuid",
  "group_id": "group-uuid"
}

{
  "workspace_id": "workspace-uuid",
  "group_id": "group-uuid",
  "file_ids": ["file-uuid-1", "file-uuid-2"]
}

{
  "workspace_id": "workspace-uuid",
  "extractor_code": "class MyExtractor(SimpleExtractor):\n    def extract(self, reader):\n        sample = reader.read(n_rows=20)\n        headers = [str(v).strip() for v in sample.row(0)]\n        for chunk in reader.iter(start_row=1, columns=headers):\n            yield chunk",
  "file_ids": ["file-uuid-1"],
  "test": true,
  "rows": ":10"
}

{
  "workspace_id": "workspace-uuid",
  "group_id": "group-uuid",
  "force": true,
  "instructions": "Extract only rows where the coverage type is 'Individual'. Ignore aggregate rows."
}

Use test=True with a single file_id to verify extraction output before running on the full group. The test response includes row count, column names, validation warnings, and a data sample.

Overview

Ingestion Tools

Pipeline Tools

Warehouse & Models

Intelligence Tools

Detection & Extraction

Pipeline order

group_manage

Examples

detection_run

Parameters

Examples

extraction_run

Parameters

Examples

​Pipeline order

​group_manage

​Examples

​detection_run

​Parameters

​Examples

​extraction_run

​Parameters

​Examples

Pipeline order

group_manage

Examples

detection_run

Parameters

Examples

extraction_run

Parameters

Examples