News Pipeline

The news pipeline runs four sequential steps: fetch → extract → cluster → summarize. You can run all steps together, run individual steps to reprocess data, check run status, and manage the branch configurations that control pipeline behavior.

Pipeline steps

Fetch

Retrieves articles from your configured sources for the date range defined in the branch’s fetch_config. Articles are deduplicated by URL within a branch — re-fetching the same date range is safe.

Extract

Scores each unscored article for relevance (1–10) and extracts structured fields: event_type, primary_entity, and a short event_summary. Articles that meet the relevance threshold go into the relevant bucket; the rest go to discard.

Cluster

Groups relevant articles into events based on semantic similarity. New articles are matched against events from the last 72 hours so that ongoing stories accumulate articles across daily runs rather than creating duplicate events.

Summarize

Generates a headline label and a structured intelligence brief for each event that does not yet have one. The brief includes sections for “The news”, “Why it matters”, and “Context”.

Running the pipeline

Run the full pipeline

Run all four steps against the production branch:

news_pipeline(action="run")

Run against a specific branch:

news_pipeline(action="run", branch_id="your-branch-id")

Run specific steps

Pass a steps list to run only certain steps. This is useful when you want to re-score or re-cluster without re-fetching.

# Fetch only
news_pipeline(action="run", steps=["fetch"])

# Extract and cluster (skips fetch and summarize)
news_pipeline(action="run", steps=["extract", "cluster"])

# Re-summarize events that already have articles
news_pipeline(action="run", steps=["summarize"])

Valid step names are fetch, extract, cluster, and summarize. When you run a subset of steps, each step operates on whatever data is already in the branch — for example, running extract without fetch scores articles already stored from a prior fetch.

Backfills with date ranges

Use date_from and date_to to fetch articles for a specific date range. This overrides the branch’s days_back setting.

# Backfill a specific week
news_pipeline(
    action="run",
    date_from="2026-03-01",
    date_to="2026-03-07",
)

date_to defaults to today if omitted. Both values use YYYY-MM-DD format.

Checking run status

news_pipeline(action="status")

# Check status for a specific branch
news_pipeline(action="status", branch_id="your-branch-id")

The response includes the latest run’s status (pending, running, completed, or failed), the current step if still running, and counts for articles fetched, articles scored, and events created.

Managing branches

List all branches

news_branches(action="list")

The production branch is marked [PRODUCTION] in the output.

Get branch details

news_branches(action="get", branch_id="your-branch-id")

Returns the branch name, schedule status, description, and fetch_config.

Create a branch

news_branches(action="create", name="my-experiment")

You can pass a config dict with any branch fields:

news_branches(
    action="create",
    name="my-experiment",
    config={
        "description": "Testing a tighter scoring prompt",
        "extraction_prompt": "Focus only on UnitedHealth and Humana...",
        "days_back": 7,
    },
)

Clone the production branch

Cloning copies the production branch’s full configuration — including its fetch_config, prompts, and clustering settings — into a new branch. This is the recommended way to start a new experiment.

news_branches(
    action="clone",
    branch_id="00000000-0000-0000-0000-000000000001",
    name="my-clone",
)

Update a branch

news_branches(
    action="update",
    branch_id="your-branch-id",
    config={
        "extraction_prompt": "Updated prompt text...",
        "schedule_enabled": True,
        "schedule_cron": "0 8 * * *",
    },
)

You can update the branch name by passing name directly:

news_branches(
    action="update",
    branch_id="your-branch-id",
    name="renamed-branch",
)

Enable a schedule

Custom branches can run on their own schedule using standard cron syntax. The schedule_timezone field defaults to America/New_York.

news_branches(
    action="update",
    branch_id="your-branch-id",
    config={
        "schedule_enabled": True,
        "schedule_cron": "0 */6 * * *",       # Every 6 hours
        "schedule_timezone": "America/Chicago",
    },
)

Delete a branch

news_branches(action="delete", branch_id="your-branch-id")

You cannot delete the production branch. Deleting a branch removes all articles and events stored under it.

Branch config reference

Field	Type	Description
`fetch_config`	dict	Source definitions and search queries for the fetch step
`extraction_prompt`	string	Custom system prompt for the extract/score step
`extraction_config`	dict	Additional extraction settings
`filter_prompt`	string	Custom prompt for the filter step
`filter_config`	dict	Filter settings, including `min_relevant_score`
`cluster_prompt`	string	Custom prompt for the cluster step
`cluster_config`	dict	Clustering settings
`summary_prompt`	string	Custom prompt for the summarize step
`days_back`	int	How many days back to fetch (overridden by `date_from`/`date_to`)
`schedule_enabled`	bool	Whether the branch runs on a schedule
`schedule_cron`	string	Cron expression (5-field, e.g. `"0 8 * * *"`)
`schedule_timezone`	string	Timezone for the schedule (default: `America/New_York`)
`newsletter_audience_ids`	list	Audience IDs to send a newsletter after scheduled production runs

Viewing events

Events are clusters of related articles that the pipeline has grouped around a single real-world occurrence. Each event has a label, a structured summary, and an article count.

List events

news_events(action="list")

Filter by status:

# Only pending review
news_events(action="list", status="pending")

# Approved events
news_events(action="list", status="approved")

# Rejected events
news_events(action="list", status="rejected")

Limit results and target a specific branch:

news_events(
    action="list",
    branch_id="your-branch-id",
    limit=10,
)

The output shows each event’s label, article count, and a truncated preview of its summary.

Viewing articles

Articles are individual news items with a relevance score and bucket classification.

List articles

news_articles(action="list")

Filter by bucket:

# Only relevant articles
news_articles(action="list", bucket="relevant")

# Discarded articles
news_articles(action="list", bucket="discard")

Limit results and target a specific branch:

news_articles(
    action="list",
    branch_id="your-branch-id",
    bucket="relevant",
    limit=50,
)

Each article in the output shows its relevance score (e.g. [8]), bucket, title, source name, and URL.

Use bucket="discard" with a custom extraction_prompt on a test branch to tune your scoring threshold. Run news_pipeline(action="run", steps=["extract"]) after updating the prompt to re-score existing articles without re-fetching.

Production schedule

The production branch runs automatically on its configured schedule. You do not need to trigger it manually. After each scheduled run, if newsletter_audience_ids are configured on the branch, Soria automatically sends a newsletter with the latest events. To check when the production branch last ran:

news_pipeline(action="status")

Get Started

Data Pipeline

Workspaces

Dashboards

News Intelligence

News Pipeline

Pipeline steps

Running the pipeline

Run the full pipeline

Run specific steps

Backfills with date ranges

Checking run status

Managing branches

List all branches

Get branch details

Create a branch

Clone the production branch

Update a branch

Enable a schedule

Delete a branch

Branch config reference

Viewing events

List events

Viewing articles

List articles

Production schedule

​Pipeline steps

​Running the pipeline

​Run the full pipeline

​Run specific steps

​Backfills with date ranges

​Checking run status

​Managing branches

​List all branches

​Get branch details

​Create a branch

​Clone the production branch

​Update a branch

​Enable a schedule

​Delete a branch

​Branch config reference

​Viewing events

​List events

​Viewing articles

​List articles

​Production schedule

Pipeline steps

Running the pipeline

Run the full pipeline

Run specific steps

Backfills with date ranges

Checking run status

Managing branches

List all branches

Get branch details

Create a branch

Clone the production branch

Update a branch

Enable a schedule

Delete a branch

Branch config reference

Viewing events

List events

Viewing articles

List articles

Production schedule