Manual Upload

Some data sources block automated scraping with WAF protection (Akamai, Cloudflare, etc.), require a login that cannot be automated, or simply distribute files as local downloads rather than web pages. For these cases, use the manual upload flow to bring files into Soria without a scraper. The flow has two steps:

scraper_upload_urls — generate presigned GCS PUT URLs so you can upload files directly to Soria’s storage
scraper_confirm_uploads — confirm the uploads and trigger the full processing pipeline

scraper_upload_urls

Generates presigned GCS PUT URLs for one or more files. Each URL is valid for a single upload of the exact file described by its content_hash and size_bytes.

Parameters

string

required

The name of a manual-type scraper. The scraper must already exist.

object[]

required

A list of file descriptors. Each item must include:

filename (string) — the name of the file, e.g. report-q1-2024.csv
content_hash (string) — SHA-256 hex digest of the file contents
size_bytes (integer) — file size in bytes

To compute content_hash on the command line:

sha256sum report.csv | cut -d' ' -f1

Response

For each file, the tool returns an upload_url (the presigned PUT URL) and a storage_key (used later in scraper_confirm_uploads). Files whose content hash already exists in storage are marked as duplicates and skipped.

Upload URLs for 'kaufman_hall':
  report-q1-2024.csv:
    storage_key: scrapers/kaufman_hall/abc123.csv
    upload_url: https://storage.googleapis.com/soria-data/...
  report-q4-2023.csv: DUPLICATE (already exists)

1 duplicate(s) skipped

Uploading a file

Use curl to PUT the file to the presigned URL. Set Content-Type to match the file type.

curl -X PUT "<upload_url>" \
  -H "Content-Type: text/csv" \
  --upload-file ./report-q1-2024.csv

For PDFs:

curl -X PUT "<upload_url>" \
  -H "Content-Type: application/pdf" \
  --upload-file ./report-q1-2024.pdf

scraper_confirm_uploads

Ingests uploaded files into a scraper and triggers the full processing pipeline: file record creation, group assignment, CSV schema mapping, and PDF table detection.

Parameters

string

required

The name of the manual scraper to ingest files into.

object[]

required

A list of file descriptors. Each item uses one of two formats depending on how the file was provided:Presigned URL flow (after using scraper_upload_urls):

filename (string) — file name
storage_key (string) — the storage_key returned by scraper_upload_urls
content_hash (string) — SHA-256 hex digest, same value used in scraper_upload_urls

Local file flow (file is on disk, accessible to the server):

filename (string) — file name
file_path (string) — absolute path to the file on disk

string

Optional workspace context for the scraper. Provide this if the scraper is associated with a specific workspace.

Response

Upload confirmed for 'kaufman_hall':
  Files created: 1
  Files grouped: 1
  Files column-mapped: 1

Full two-step flow

Compute file hashes and sizes

For each file you want to upload, compute its SHA-256 hash and byte size.

sha256sum report-q1-2024.csv | cut -d' ' -f1
# outputs: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

wc -c < report-q1-2024.csv
# outputs: 204800

Request presigned upload URLs

scraper_upload_urls(
    scraper_name="kaufman_hall",
    files=[
        {
            "filename": "report-q1-2024.csv",
            "content_hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
            "size_bytes": 204800,
        }
    ],
)

Save the upload_url and storage_key from the response.

Upload the file with curl

curl -X PUT "<upload_url>" \
  -H "Content-Type: text/csv" \
  --upload-file ./report-q1-2024.csv

The upload goes directly to GCS — no data passes through the Soria API.

Confirm the upload and trigger ingestion

scraper_confirm_uploads(
    scraper_name="kaufman_hall",
    files=[
        {
            "filename": "report-q1-2024.csv",
            "storage_key": "scrapers/kaufman_hall/abc123.csv",
            "content_hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
        }
    ],
    workspace_id="ws_abc123",
)

Soria verifies the file exists in storage and runs it through the full pipeline: group assignment, schema mapping, and PDF detection (if applicable).

Local file alternative

If you have files on disk that the Soria server can access directly, you can skip scraper_upload_urls and confirm them in a single call using file_path:

scraper_confirm_uploads(
    scraper_name="kaufman_hall",
    files=[
        {
            "filename": "report-q1-2024.csv",
            "file_path": "/path/to/report-q1-2024.csv",
        }
    ],
    workspace_id="ws_abc123",
)

Soria reads the file from disk, computes the hash, uploads it to storage, and runs the pipeline — all in one step.

Overview

Ingestion Tools

Pipeline Tools

Warehouse & Models

Intelligence Tools

Manual Upload

scraper_upload_urls

Parameters

Response

Uploading a file

scraper_confirm_uploads

Parameters

Response

Full two-step flow

Local file alternative

​scraper_upload_urls

​Parameters

​Response

​Uploading a file

​scraper_confirm_uploads

​Parameters

​Response

​Full two-step flow

​Local file alternative

scraper_upload_urls

Parameters

Response

Uploading a file

scraper_confirm_uploads

Parameters

Response

Full two-step flow

Local file alternative