Scrapers

A scraper is a Python class that knows how to find downloadable files on a public URL. You give it a URL; it returns a list of files with their download links, filenames, and dates. The base class handles the actual downloading — you only write the discovery logic.

Always test a scraper with test=True before running it for real. Test mode runs your discover_files() method and shows you what files it found, without downloading anything.

The SimpleScraper interface

Every scraper extends SimpleScraper and implements one method: discover_files(). It returns a list of dicts, each with four required keys:

Key	Type	Description
`url`	string	Direct download URL for the file
`filename`	string	Name to save the file as
`date`	string	Publication date in ISO format
`page_url`	string	The page the link was found on

Dates must be in one of these ISO formats: YYYY, YYYY-MM, YYYY-MM-DD, or YYYY-QN (for example, "2024-Q1"). Non-ISO dates are rejected at discovery time.

from soria.scrapers.core.base_scraper import SimpleScraper, get_html, make_absolute_url

class MySourceScraper(SimpleScraper):
    def discover_files(self) -> list[dict]:
        html = get_html(self.url)
        # Parse HTML, find file links
        return [
            {
                "url": "https://example.gov/data/report-2024.csv",
                "filename": "report-2024.csv",
                "date": "2024",
                "page_url": self.url,
            }
        ]

The class name should follow the pattern {ScraperName}Scraper — for example, KaufmanHallScraper.

Capability tiers

Choose the simplest approach that works for your source. Simpler scrapers are faster, cheaper, and more reliable.

Direct HTTP

Use get_html() or get_json() to fetch pages directly. Works for most public data portals where files are linked in standard HTML.

AI browser automation

Use self.browser_task() to describe what you want to find. An AI agent navigates the site and returns structured data. Good for dynamic or JS-heavy sites without writing selectors.

Playwright browser

Set needs_browser = True to get self.page, a Playwright sync Page. Use when you need precise, deterministic browser control — clicking tabs, filling forms, waiting for elements.

Tier 1: Direct HTTP

The default — no extra configuration needed. get_html() returns a BeautifulSoup object; get_json() fetches and parses JSON.

class NationalHealthExpenditureDataScraper(SimpleScraper):
    def discover_files(self) -> list[dict]:
        soup = get_html(self.url)
        files = []
        for a in soup.find_all("a", href=True):
            href = a["href"]
            if not href.lower().endswith((".xlsx", ".csv", ".pdf")):
                continue
            url = make_absolute_url(href, self.url)
            filename = href.rsplit("/", 1)[-1].split("?")[0]
            files.append({
                "url": url,
                "filename": filename,
                "date": "2024",
                "page_url": self.url,
            })
        return files

Tier 2: AI browser automation

Call self.browser_task() with a plain-language description of what to find. Pass an output_schema to get back structured JSON. This works in any scraper — you don’t need needs_browser = True.

class MyScraper(SimpleScraper):
    def discover_files(self) -> list[dict]:
        result = self.browser_task(
            "Find all downloadable PDF reports on this page. Return their URLs, filenames, and dates.",
            output_schema={
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "url": {"type": "string"},
                        "filename": {"type": "string"},
                        "date": {"type": "string"},
                    },
                },
            },
        )
        return [
            {"url": r["url"], "filename": r["filename"], "date": r["date"], "page_url": self.url}
            for r in result
        ]

Tier 3: Playwright browser

Set needs_browser = True to get self.page — a Playwright sync Page that starts on self.url with bot protection already cleared. Use this when you need to click through tabs, wait for dynamic content, or interact with forms.

class MyScraper(SimpleScraper):
    needs_browser = True

    def discover_files(self) -> list[dict]:
        # self.page already starts on self.url
        self.page.click("#reports-tab")
        self.page.wait_for_selector(".report-list")
        # parse self.page.content() with BeautifulSoup
        return [...]

Available in the scraper namespace

These are available without importing anything:

Name	Description
`get_html(url, headers=None)`	Fetch a page and return a BeautifulSoup object
`get_json(url, method='GET', json_body=None, headers=None)`	Fetch and parse JSON
`make_absolute_url(url, base)`	Resolve a relative URL against a base URL
`clean_filename(name)`	Sanitize a filename
`DEFAULT_HEADERS`	Default request headers (Chrome User-Agent, etc.)
`re`, `json`, `datetime`	Pre-loaded standard library modules

You can also define SCRAPER_HEADERS on your class to add or override request headers for every request this scraper makes (merged with DEFAULT_HEADERS):

class MyScraper(SimpleScraper):
    SCRAPER_HEADERS = {
        "Cookie": "session=abc123",
    }

Available for import: polars (use instead of pandas — pandas is not installed), requests, lxml, beautifulsoup4, pdfplumber, html5lib, xmltodict, tenacity, curl_cffi, openpyxl, xlrd, chardet, python-dateutil, and all Python standard library modules.

Testing and running a scraper

Always test before running for real.

Create a workspace

workspace_manage(operation="create", scraper_name="my_source")

Test your scraper code

Test mode runs discover_files() and shows you the files found — no downloads happen.

scraper_run(
    scraper_name="my_source",
    code="...",
    url="https://example.gov/data",
    test=True
)

The output shows how many files were found and previews the first five with their filenames, dates, and URLs.

Save the scraper code

Once the test output looks correct, save the code:

scraper_manage(
    scraper_name="my_source",
    save={"code": "...", "url": "https://example.gov/data"},
    workspace_id="<workspace_id>"
)

Run the scraper to download files

scraper_run(scraper_name="my_source", workspace_id="<workspace_id>")

The output reports how many files were discovered, downloaded, grouped, and — for CSVs — column-mapped. For PDFs, it reports how many detection jobs were enqueued.

If a site blocks automated requests (Akamai, Cloudflare WAF), scraper_run may time out. In that case, use scraper_upload_urls to generate presigned upload URLs and provide the files manually, then confirm with scraper_confirm_uploads.

Render HTML pages as PDF

Some data sources only publish HTML tables with no downloadable files. Set render_as_pdf = True on your scraper and all discovered URLs will be rendered to PDF by a headless browser and fed into the extraction pipeline.

class MyScraper(SimpleScraper):
    render_as_pdf = True

    def discover_files(self) -> list[dict]:
        # Return page URLs instead of file download URLs
        return [
            {"url": "https://example.gov/table/2024-q1", "filename": "2024-q1.pdf", "date": "2024-Q1", "page_url": self.url}
        ]

Get Started

Data Pipeline

Workspaces

Dashboards

News Intelligence

The SimpleScraper interface

Capability tiers

Direct HTTP

AI browser automation

Playwright browser

Tier 1: Direct HTTP

Tier 2: AI browser automation

Tier 3: Playwright browser

Available in the scraper namespace

Testing and running a scraper

Render HTML pages as PDF

​The SimpleScraper interface

​Capability tiers

Direct HTTP

AI browser automation

Playwright browser

​Tier 1: Direct HTTP

​Tier 2: AI browser automation

​Tier 3: Playwright browser

​Available in the scraper namespace

​Testing and running a scraper

​Render HTML pages as PDF

The SimpleScraper interface

Capability tiers

Tier 1: Direct HTTP

Tier 2: AI browser automation

Tier 3: Playwright browser

Available in the scraper namespace

Testing and running a scraper

Render HTML pages as PDF