scraper_run

Use scraper_run to either test a scraper in development or execute a saved scraper to download files. Test mode lets you validate your discover_files() logic and preview the file list it returns — without downloading anything or touching the workspace. Run mode dispatches the scraper to download all discovered files and triggers the full processing pipeline.

Parameters

string

required

The name of the scraper to run or test.

boolean

When true, runs the scraper in test mode. Executes discover_files() and returns the files found without downloading. Requires code and url.

string

The Python scraper code to execute. Required in test mode.

string

The URL to pass to the scraper. Required in test mode.

boolean

When true, overwrites existing file metadata even if the file has already been discovered. Use this to pick up date or filename corrections after updating scraper code.

string

The workspace to write discovered files into. Required for non-test runs.

Modes

Test mode
Run mode

Test mode runs your discover_files() implementation against the target URL and returns the file metadata it produces. No files are downloaded and no workspace records are created.Use test mode during development to:

Confirm your scraper can reach the target site
Verify the returned url, filename, date, and page_url values look correct
Catch date format errors before they are rejected by the pipeline

Test mode previews up to 5 files. If more files are found, the response reports the total count and notes how many were omitted.

scraper_run(
    scraper_name="kaufman_hall",
    test=True,
    code="""
from soria.scrapers.core.base_scraper import SimpleScraper, get_html, make_absolute_url

class KaufmanHallScraper(SimpleScraper):
def discover_files(self) -> list[dict]:
    html = get_html(self.url)
    links = html.select("a.report-download")
    return [
        {
            "url": make_absolute_url(a["href"], self.url),
            "filename": a.text.strip(),
            "date": a.find_parent("tr").select_one(".date").text.strip(),
            "page_url": self.url,
        }
        for a in links
    ]
""",
    url="https://www.kaufmanhall.com/resources/research",
)

Example response:

Test passed for 'kaufman_hall'

Files found: 12
  - kaufman-hall-margin-2024-q1.pdf  date=2024-Q1  url=https://...
  - kaufman-hall-margin-2023-q4.pdf  date=2023-Q4  url=https://...
  - kaufman-hall-margin-2023-q3.pdf  date=2023-Q3  url=https://...
  - kaufman-hall-margin-2023-q2.pdf  date=2023-Q2  url=https://...
  - kaufman-hall-margin-2023-q1.pdf  date=2023-Q1  url=https://...
  ... and 7 more

If the test times out (over 20 seconds), the most common causes are: the target site is blocking automated requests (WAF such as Akamai or Cloudflare), or an infinite loop in discover_files(). For sites that block scraping, use scraper_upload_urls instead.

Run mode dispatches the saved scraper to download all discovered files and ingest them into the specified workspace. The scraper must already have code saved via scraper_manage.

scraper_run(
    scraper_name="kaufman_hall",
    workspace_id="ws_abc123",
)

Example response:

Scraper 'kaufman_hall' completed
Files discovered: 12
Files downloaded: 3
Duplicates skipped: 9

Pipeline:
  Files grouped: 3/3
  Files column-mapped: 3

The response breaks down what happened at each stage:

Field	Description
`Files discovered`	Total files returned by `discover_files()`
`Files downloaded`	New files that were not previously ingested
`Duplicates skipped`	Files already present in the workspace (matched by content hash)
`Existing metadata updated`	Files whose metadata was refreshed (when `force_refresh_metadata=True`)
`Files grouped`	Files successfully assigned to a file group
`Files column-mapped`	CSV/spreadsheet files with columns mapped to the workspace schema
`New column variants`	New column name variants registered for schema mapping
`Detection enqueued`	PDF files queued for table detection

Schema drift detection

If previously mapped column headers no longer appear in the downloaded files — or if new unmapped headers are present — the response includes a schema drift report:

Schema drift detected:
  Group "kaufman_hall_margins":
    Unmapped headers: ['Hospital Name', 'Fiscal Year']
    Available schema columns: ['facility_name', 'report_year', 'operating_margin']
  Use schema_mappings to map these columns.

Use schema_mappings to resolve unmapped headers before re-running the scraper.

Run mode dispatches work asynchronously. If the tool call hangs, the scraper is likely still running on the server — do not retry blindly, as that will trigger duplicate downloads. Check server logs to verify progress before retrying.

Overview

Ingestion Tools

Pipeline Tools

Warehouse & Models

Intelligence Tools

Parameters

Modes

Schema drift detection

​Parameters

​Modes

​Schema drift detection

Parameters

Modes

Schema drift detection