> ## Documentation Index
> Fetch the complete documentation index at: https://docs.soriaanalytics.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Scrapers
> How to create scrapers that discover and download files from public data sources.
A scraper is a Python class that knows how to find downloadable files on a public URL. You give it a URL; it returns a list of files with their download links, filenames, and dates. The base class handles the actual downloading — you only write the discovery logic.
Always test a scraper with `test=True` before running it for real. Test mode runs your `discover_files()` method and shows you what files it found, without downloading anything.
## The SimpleScraper interface
Every scraper extends `SimpleScraper` and implements one method: `discover_files()`. It returns a list of dicts, each with four required keys:
| Key | Type | Description |
| ---------- | ------ | -------------------------------- |
| `url` | string | Direct download URL for the file |
| `filename` | string | Name to save the file as |
| `date` | string | Publication date in ISO format |
| `page_url` | string | The page the link was found on |
Dates must be in one of these ISO formats: `YYYY`, `YYYY-MM`, `YYYY-MM-DD`, or `YYYY-QN` (for example, `"2024-Q1"`). Non-ISO dates are rejected at discovery time.
```python theme={null}
from soria.scrapers.core.base_scraper import SimpleScraper, get_html, make_absolute_url
class MySourceScraper(SimpleScraper):
def discover_files(self) -> list[dict]:
html = get_html(self.url)
# Parse HTML, find file links
return [
{
"url": "https://example.gov/data/report-2024.csv",
"filename": "report-2024.csv",
"date": "2024",
"page_url": self.url,
}
]
```
The class name should follow the pattern `{ScraperName}Scraper` — for example, `KaufmanHallScraper`.
***
## Capability tiers
Choose the simplest approach that works for your source. Simpler scrapers are faster, cheaper, and more reliable.
Use `get_html()` or `get_json()` to fetch pages directly. Works for most public data portals where files are linked in standard HTML.
Use `self.browser_task()` to describe what you want to find. An AI agent navigates the site and returns structured data. Good for dynamic or JS-heavy sites without writing selectors.
Set `needs_browser = True` to get `self.page`, a Playwright sync Page. Use when you need precise, deterministic browser control — clicking tabs, filling forms, waiting for elements.
### Tier 1: Direct HTTP
The default — no extra configuration needed. `get_html()` returns a BeautifulSoup object; `get_json()` fetches and parses JSON.
```python theme={null}
class NationalHealthExpenditureDataScraper(SimpleScraper):
def discover_files(self) -> list[dict]:
soup = get_html(self.url)
files = []
for a in soup.find_all("a", href=True):
href = a["href"]
if not href.lower().endswith((".xlsx", ".csv", ".pdf")):
continue
url = make_absolute_url(href, self.url)
filename = href.rsplit("/", 1)[-1].split("?")[0]
files.append({
"url": url,
"filename": filename,
"date": "2024",
"page_url": self.url,
})
return files
```
### Tier 2: AI browser automation
Call `self.browser_task()` with a plain-language description of what to find. Pass an `output_schema` to get back structured JSON. This works in any scraper — you don't need `needs_browser = True`.
```python theme={null}
class MyScraper(SimpleScraper):
def discover_files(self) -> list[dict]:
result = self.browser_task(
"Find all downloadable PDF reports on this page. Return their URLs, filenames, and dates.",
output_schema={
"type": "array",
"items": {
"type": "object",
"properties": {
"url": {"type": "string"},
"filename": {"type": "string"},
"date": {"type": "string"},
},
},
},
)
return [
{"url": r["url"], "filename": r["filename"], "date": r["date"], "page_url": self.url}
for r in result
]
```
### Tier 3: Playwright browser
Set `needs_browser = True` to get `self.page` — a Playwright sync Page that starts on `self.url` with bot protection already cleared. Use this when you need to click through tabs, wait for dynamic content, or interact with forms.
```python theme={null}
class MyScraper(SimpleScraper):
needs_browser = True
def discover_files(self) -> list[dict]:
# self.page already starts on self.url
self.page.click("#reports-tab")
self.page.wait_for_selector(".report-list")
# parse self.page.content() with BeautifulSoup
return [...]
```
***
## Available in the scraper namespace
These are available without importing anything:
| Name | Description |
| ----------------------------------------------------------- | ------------------------------------------------- |
| `get_html(url, headers=None)` | Fetch a page and return a BeautifulSoup object |
| `get_json(url, method='GET', json_body=None, headers=None)` | Fetch and parse JSON |
| `make_absolute_url(url, base)` | Resolve a relative URL against a base URL |
| `clean_filename(name)` | Sanitize a filename |
| `DEFAULT_HEADERS` | Default request headers (Chrome User-Agent, etc.) |
| `re`, `json`, `datetime` | Pre-loaded standard library modules |
You can also define `SCRAPER_HEADERS` on your class to add or override request headers for every request this scraper makes (merged with `DEFAULT_HEADERS`):
```python theme={null}
class MyScraper(SimpleScraper):
SCRAPER_HEADERS = {
"Cookie": "session=abc123",
}
```
**Available for import:** `polars` (use instead of pandas — pandas is not installed), `requests`, `lxml`, `beautifulsoup4`, `pdfplumber`, `html5lib`, `xmltodict`, `tenacity`, `curl_cffi`, `openpyxl`, `xlrd`, `chardet`, `python-dateutil`, and all Python standard library modules.
***
## Testing and running a scraper
Always test before running for real.
```
workspace_manage(operation="create", scraper_name="my_source")
```
Test mode runs `discover_files()` and shows you the files found — no downloads happen.
```
scraper_run(
scraper_name="my_source",
code="...",
url="https://example.gov/data",
test=True
)
```
The output shows how many files were found and previews the first five with their filenames, dates, and URLs.
Once the test output looks correct, save the code:
```
scraper_manage(
scraper_name="my_source",
save={"code": "...", "url": "https://example.gov/data"},
workspace_id=""
)
```
```
scraper_run(scraper_name="my_source", workspace_id="")
```
The output reports how many files were discovered, downloaded, grouped, and — for CSVs — column-mapped. For PDFs, it reports how many detection jobs were enqueued.
If a site blocks automated requests (Akamai, Cloudflare WAF), `scraper_run` may time out. In that case, use `scraper_upload_urls` to generate presigned upload URLs and provide the files manually, then confirm with `scraper_confirm_uploads`.
***
## Render HTML pages as PDF
Some data sources only publish HTML tables with no downloadable files. Set `render_as_pdf = True` on your scraper and all discovered URLs will be rendered to PDF by a headless browser and fed into the extraction pipeline.
```python theme={null}
class MyScraper(SimpleScraper):
render_as_pdf = True
def discover_files(self) -> list[dict]:
# Return page URLs instead of file download URLs
return [
{"url": "https://example.gov/table/2024-q1", "filename": "2024-q1.pdf", "date": "2024-Q1", "page_url": self.url}
]
```