Schema

The schema pipeline tools let you build and maintain the column definitions Soria uses when extracting data from your files. Use schema_manage to define what columns exist and schema_mappings to connect source file headers to those columns.

schema_manage

Define, update, or read the schema columns for a file group. A schema tells Soria which columns to extract when it processes files in that group.

workspace_id is required for any write operation (update_columns). Read-only calls work without it.

Parameters

string

required

The scraper that owns the group. Get this from scraper_manage.

string

required

The group to manage the schema for. Get this from group_manage.

boolean

default:"false"

Return the current schema and its columns.

boolean

default:"false"

Generate or backfill vector embeddings for canonical columns. Embeddings power the similarity suggestions in schema_mappings. Safe to run multiple times — only processes columns missing embeddings.

object[]

Replace the schema’s column list. Each object is a column definition:

Field	Type	Notes
`id`	string (UUID)	Omit to create a new column; include to update an existing one
`name`	string	Column name (required)
`column_source`	`"header"` \| `"derived"`	Where the value comes from — a CSV/Excel header (`"header"`) or a computed value (`"derived"`). Defaults to `"header"`.
`is_nullable`	boolean	Whether the column can be empty. Defaults to `true`.
`is_required`	boolean	Whether extraction must produce a value. Defaults to `false`.
`description`	string	Optional description used by the AI during extraction

Columns not included in update_columns are deleted, along with any mappings that reference them. Always include every column you want to keep when calling with update_columns.

string

Required for write operations (update_columns). Identifies which workspace schema to use.

Column lifecycle

Create: include a column object without an id field.
Update: include a column object with its existing id.
Delete: omit the column from the list entirely.

You can combine read=True and update_columns in a single call to update and immediately confirm the result.

Examples

{
  "scraper_id": "scraper-uuid",
  "group_id": "group-uuid",
  "read": true
}

{
  "scraper_id": "scraper-uuid",
  "group_id": "group-uuid",
  "workspace_id": "workspace-uuid",
  "update_columns": [
    {
      "name": "Plan Name",
      "column_source": "header",
      "is_required": true,
      "description": "The name of the insurance plan"
    },
    {
      "name": "Premium Amount",
      "column_source": "header",
      "is_nullable": false
    },
    {
      "name": "Reporting Period",
      "column_source": "derived",
      "description": "Computed from the file name"
    }
  ]
}

{
  "scraper_id": "scraper-uuid",
  "group_id": "group-uuid",
  "workspace_id": "workspace-uuid",
  "update_columns": [
    {
      "id": "col-uuid-1",
      "name": "Plan Name",
      "column_source": "header",
      "is_required": true
    },
    {
      "id": "col-uuid-2",
      "name": "Monthly Premium",
      "column_source": "header"
    }
  ]
}

{
  "scraper_id": "scraper-uuid",
  "group_id": "group-uuid",
  "embed": true
}

schema_mappings

Map the raw column headers found in your source files (CSVs, Excel sheets) to the canonical schema columns you defined with schema_manage. This is necessary when column names in your files differ from your schema column names, or when those names change over time (schema drift). Soria uses the mappings to correctly align source data to your schema during extraction. When you call with read=True, Soria returns:

Current mappings — which source headers are already mapped and to which schema column
Unmapped schema columns — schema columns that have no source mapping yet
Similarity suggestions — candidate schema columns ranked by semantic similarity to each unmapped source header (requires embeddings — run schema_manage with embed=True first)

workspace_id is required when using update.

Parameters

string

required

The group whose column mappings you want to read or update.

boolean

default:"false"

Return current mappings, unmapped schema columns, and similarity suggestions for any unmapped source columns.

object

A dictionary mapping source column names (as they appear in the file header) to target schema column UUIDs. Pass null as the value to delete an existing mapping.

{
  "Source Header Name": "target-column-uuid",
  "Old Column Name": null
}

string

Required for update operations.

Examples

{
  "group_id": "group-uuid",
  "read": true
}

{
  "group_id": "group-uuid",
  "workspace_id": "workspace-uuid",
  "update": {
    "Plan_Name": "col-uuid-plan-name",
    "Monthly Premium ($)": "col-uuid-premium",
    "Coverage Type": "col-uuid-coverage"
  }
}

{
  "group_id": "group-uuid",
  "workspace_id": "workspace-uuid",
  "update": {
    "Old Column Header": null
  }
}

{
  "group_id": "group-uuid",
  "workspace_id": "workspace-uuid",
  "read": true,
  "update": {
    "Issuer Name": "col-uuid-issuer"
  }
}

Handling schema drift

When a data source changes its column names (for example, "Plan Name" becomes "Product Name"), existing mappings no longer match. To fix drift:

Read current state

Call schema_mappings with read=True to see which source columns are unmapped and review the similarity suggestions.

Check suggestions

Each unmapped source column includes ranked suggestions with similarity scores. A score of 85% or higher is usually a reliable match.

Apply the new mappings

Call schema_mappings with update pointing the new header names to the correct column UUIDs. Delete any mappings for headers that no longer exist.

If similarity suggestions are missing or unhelpful, run schema_manage with embed=True to generate or refresh column embeddings.

Overview

Ingestion Tools

Pipeline Tools

Warehouse & Models

Intelligence Tools

schema_manage

Parameters

Column lifecycle

Examples

schema_mappings

Parameters

Examples

Handling schema drift

​schema_manage

​Parameters

​Column lifecycle

​Examples

​schema_mappings

​Parameters

​Examples

​Handling schema drift

schema_manage

Parameters

Column lifecycle

Examples

schema_mappings

Parameters

Examples

Handling schema drift