scraper_manage

Use scraper_manage to read existing scraper code or save new and updated scraper code. It is the primary tool for authoring and versioning the Python scraper classes that Soria uses to discover and download files from data sources.

Parameters

string

required

Unique name for the scraper (e.g., kaufman_hall). Used to identify the scraper across all tools.

boolean

When true, returns the existing scraper code and metadata for scraper_name.

object

A dict with the scraper code and target URL to save. Required keys:

code — the Python source code implementing the scraper class
url — the target URL the scraper will run against

string

The workspace to associate the scraper with. Required whenever save includes code.

Scraper interface

Every scraper is a Python class that extends SimpleScraper and implements a single method: discover_files(). The base class handles all downloading — your job is to return a list of file metadata dicts.

from soria.scrapers.core.base_scraper import SimpleScraper, get_html, make_absolute_url

class MyScraperScraper(SimpleScraper):
    def discover_files(self) -> list[dict]:
        html = get_html(self.url)
        # Parse HTML to find file links
        return [
            {
                "url": "https://example.com/reports/q1-2024.csv",
                "filename": "q1-2024.csv",
                "date": "2024-Q1",
                "page_url": self.url,
            }
        ]

The class name must follow the pattern {ScraperName}Scraper — for example, a scraper named kaufman_hall should define a class called KaufmanHallScraper.

`discover_files()` return format

Each dict in the returned list must include all four keys:

Key	Type	Description
`url`	string	Direct download URL for the file
`filename`	string	File name to save the file as
`date`	string	Publication date in ISO format (see below)
`page_url`	string	The page where the file was found — use `self.url` if there is only one page

Date format: Dates must be normalized to ISO format. Non-ISO dates are rejected at discovery time.

Format	Example
`YYYY`	`2024`
`YYYY-MM`	`2024-01`
`YYYY-MM-DD`	`2024-01-15`
`YYYY-QN`	`2024-Q1`

Capability tiers

Choose the simplest approach that works for your data source. More powerful tiers are slower and more expensive.

Tier 1 — Direct HTTP (`get_html()` / `get_json()`)

The default and preferred approach. Fast, cheap, and reliable. Works for any site that serves static HTML or a JSON API.

class MyScraperScraper(SimpleScraper):
    # Optional: add custom headers (merged with DEFAULT_HEADERS)
    SCRAPER_HEADERS = {
        "Cookie": "session=abc123",
    }

    def discover_files(self) -> list[dict]:
        html = get_html(self.url)
        links = html.select("a.report-link")
        return [
            {
                "url": make_absolute_url(a["href"], self.url),
                "filename": a.text.strip() + ".pdf",
                "date": a["data-date"],
                "page_url": self.url,
            }
            for a in links
        ]

Tier 2 — AI-driven browser automation (`browser_task()`)

Use when the site is dynamic or JS-heavy and you do not want to write selectors by hand. browser_task() sends a natural-language instruction to an AI browser agent and returns structured data. It does not require needs_browser=True.

class MyScraperScraper(SimpleScraper):
    def discover_files(self) -> list[dict]:
        result = self.browser_task(
            "Find all downloadable PDF reports on this page. Return their URLs, filenames, and dates.",
            output_schema={
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "url": {"type": "string"},
                        "filename": {"type": "string"},
                        "date": {"type": "string"},
                    },
                },
            },
        )
        return [
            {"url": r["url"], "filename": r["filename"], "date": r["date"], "page_url": self.url}
            for r in result
        ]

Tier 3 — Playwright (`self.page`)

Use when you need precise, deterministic browser interactions — clicking tabs, filling forms, waiting for specific DOM states. Requires needs_browser = True. The page starts on self.url with bot protection already cleared.

class MyScraperScraper(SimpleScraper):
    needs_browser = True

    def discover_files(self) -> list[dict]:
        self.page.click("#reports-tab")
        self.page.wait_for_selector(".report-list")
        # Parse self.page.content() with BeautifulSoup
        html = get_html(self.page.content())
        # ...
        return [...]

You can also set render_as_pdf = True to render HTML pages as PDFs via headless browser instead of downloading files directly. Use this for data sources that only publish HTML tables with no downloadable file.

Available in scraper namespace

These are available without any import inside a scraper class:

Name	Description
`get_html(url, headers=None)`	Fetch a URL and return a `BeautifulSoup` object
`get_json(url, method='GET', json_body=None, headers=None)`	Fetch a URL and return parsed JSON
`make_absolute_url(url, base)`	Resolve a relative URL against a base URL
`clean_filename(name)`	Sanitize a string to use as a filename
`DEFAULT_HEADERS`	Dict of default request headers (Chrome user agent, etc.)
`SCRAPER_HEADERS`	Class attribute — extend or override `DEFAULT_HEADERS` per scraper
`self.page`	Playwright sync `Page` object (only when `needs_browser=True`)
`self.browser_task(task, output_schema=None)`	AI-driven browser automation (any scraper)
`re`, `json`, `datetime`	Pre-loaded standard library modules

Additional libraries available for import:

polars (use polars for DataFrames — pandas is not installed)
requests, lxml, beautifulsoup4, pdfplumber, html5lib, xmltodict, tenacity
curl_cffi, openpyxl, xlrd, chardet, python-dateutil
All standard library modules: csv, io, collections, itertools, hashlib, time, urllib, math, etc.

Recommended workflow

Create the workspace and scraper

workspace_manage(operation="create", scraper_name="my_source")

This creates the scraper record and its associated workspace.

Fetch the target URL

Retrieve the source page to understand its structure before writing code.

Write the scraper code

Implement discover_files() using the appropriate capability tier. Start with Tier 1 (direct HTTP) and move up only if needed.

Test the scraper

scraper_run(scraper_name="my_source", code="...", url="...", test=True)

Test mode runs discover_files() and returns the files found without downloading anything.

Save the scraper code

scraper_manage(
    scraper_name="my_source",
    save={"code": "...", "url": "https://example.com/reports"},
    workspace_id="ws_abc123",
)

Run the scraper

scraper_run(scraper_name="my_source", workspace_id="ws_abc123")

Downloads all discovered files and triggers the processing pipeline.

Examples

Reading existing scraper code

scraper_manage(scraper_name="kaufman_hall", read=True)

The response includes the scraper’s URL, ID, and full Python source code.

Saving new scraper code

scraper_manage(
    scraper_name="kaufman_hall",
    save={
        "code": """
from soria.scrapers.core.base_scraper import SimpleScraper, get_html, make_absolute_url

class KaufmanHallScraper(SimpleScraper):
    def discover_files(self) -> list[dict]:
        html = get_html(self.url)
        rows = html.select("table.reports tr")
        files = []
        for row in rows[1:]:
            cells = row.select("td")
            if len(cells) < 2:
                continue
            link = cells[0].find("a")
            if not link:
                continue
            files.append({
                "url": make_absolute_url(link["href"], self.url),
                "filename": link.text.strip(),
                "date": cells[1].text.strip(),
                "page_url": self.url,
            })
        return files
""",
        "url": "https://www.kaufmanhall.com/resources/research",
    },
    workspace_id="ws_abc123",
)

Overview

Ingestion Tools

Pipeline Tools

Warehouse & Models

Intelligence Tools

scraper_manage

Parameters

Scraper interface

`discover_files()` return format

Capability tiers

Tier 1 — Direct HTTP (`get_html()` / `get_json()`)

Tier 2 — AI-driven browser automation (`browser_task()`)

Tier 3 — Playwright (`self.page`)

Available in scraper namespace

Recommended workflow

Examples

Reading existing scraper code

Saving new scraper code

​Parameters

​Scraper interface

​discover_files() return format

​Capability tiers

​Tier 1 — Direct HTTP (get_html() / get_json())

​Tier 2 — AI-driven browser automation (browser_task())

​Tier 3 — Playwright (self.page)

​Available in scraper namespace

​Recommended workflow

​Examples

​Reading existing scraper code

​Saving new scraper code

Parameters

Scraper interface

`discover_files()` return format

Capability tiers

Tier 1 — Direct HTTP (`get_html()` / `get_json()`)

Tier 2 — AI-driven browser automation (`browser_task()`)

Tier 3 — Playwright (`self.page`)

Available in scraper namespace

Recommended workflow

Examples

Reading existing scraper code

Saving new scraper code