> ## Documentation Index
> Fetch the complete documentation index at: https://docs.soriaanalytics.com/llms.txt
> Use this file to discover all available pages before exploring further.

# News Pipeline

> How to run the news pipeline, manage branches, and query events and articles.

The news pipeline runs four sequential steps: **fetch → extract → cluster → summarize**. You can run all steps together, run individual steps to reprocess data, check run status, and manage the branch configurations that control pipeline behavior.

## Pipeline steps

<Steps>
  <Step title="Fetch">
    Retrieves articles from your configured sources for the date range defined in the branch's `fetch_config`. Articles are deduplicated by URL within a branch — re-fetching the same date range is safe.
  </Step>

  <Step title="Extract">
    Scores each unscored article for relevance (1–10) and extracts structured fields: `event_type`, `primary_entity`, and a short `event_summary`. Articles that meet the relevance threshold go into the `relevant` bucket; the rest go to `discard`.
  </Step>

  <Step title="Cluster">
    Groups relevant articles into events based on semantic similarity. New articles are matched against events from the last 72 hours so that ongoing stories accumulate articles across daily runs rather than creating duplicate events.
  </Step>

  <Step title="Summarize">
    Generates a headline label and a structured intelligence brief for each event that does not yet have one. The brief includes sections for "The news", "Why it matters", and "Context".
  </Step>
</Steps>

***

## Running the pipeline

### Run the full pipeline

Run all four steps against the production branch:

```python theme={null}
news_pipeline(action="run")
```

Run against a specific branch:

```python theme={null}
news_pipeline(action="run", branch_id="your-branch-id")
```

### Run specific steps

Pass a `steps` list to run only certain steps. This is useful when you want to re-score or re-cluster without re-fetching.

```python theme={null}
# Fetch only
news_pipeline(action="run", steps=["fetch"])

# Extract and cluster (skips fetch and summarize)
news_pipeline(action="run", steps=["extract", "cluster"])

# Re-summarize events that already have articles
news_pipeline(action="run", steps=["summarize"])
```

<Note>
  Valid step names are `fetch`, `extract`, `cluster`, and `summarize`. When you run a subset of steps, each step operates on whatever data is already in the branch — for example, running `extract` without `fetch` scores articles already stored from a prior fetch.
</Note>

### Backfills with date ranges

Use `date_from` and `date_to` to fetch articles for a specific date range. This overrides the branch's `days_back` setting.

```python theme={null}
# Backfill a specific week
news_pipeline(
    action="run",
    date_from="2026-03-01",
    date_to="2026-03-07",
)
```

`date_to` defaults to today if omitted. Both values use `YYYY-MM-DD` format.

### Checking run status

```python theme={null}
news_pipeline(action="status")
```

```python theme={null}
# Check status for a specific branch
news_pipeline(action="status", branch_id="your-branch-id")
```

The response includes the latest run's status (`pending`, `running`, `completed`, or `failed`), the current step if still running, and counts for articles fetched, articles scored, and events created.

***

## Managing branches

### List all branches

```python theme={null}
news_branches(action="list")
```

The production branch is marked `[PRODUCTION]` in the output.

### Get branch details

```python theme={null}
news_branches(action="get", branch_id="your-branch-id")
```

Returns the branch name, schedule status, description, and `fetch_config`.

### Create a branch

```python theme={null}
news_branches(action="create", name="my-experiment")
```

You can pass a `config` dict with any branch fields:

```python theme={null}
news_branches(
    action="create",
    name="my-experiment",
    config={
        "description": "Testing a tighter scoring prompt",
        "extraction_prompt": "Focus only on UnitedHealth and Humana...",
        "days_back": 7,
    },
)
```

### Clone the production branch

Cloning copies the production branch's full configuration — including its `fetch_config`, prompts, and clustering settings — into a new branch. This is the recommended way to start a new experiment.

```python theme={null}
news_branches(
    action="clone",
    branch_id="00000000-0000-0000-0000-000000000001",
    name="my-clone",
)
```

### Update a branch

```python theme={null}
news_branches(
    action="update",
    branch_id="your-branch-id",
    config={
        "extraction_prompt": "Updated prompt text...",
        "schedule_enabled": True,
        "schedule_cron": "0 8 * * *",
    },
)
```

You can update the branch name by passing `name` directly:

```python theme={null}
news_branches(
    action="update",
    branch_id="your-branch-id",
    name="renamed-branch",
)
```

### Enable a schedule

Custom branches can run on their own schedule using standard cron syntax. The `schedule_timezone` field defaults to `America/New_York`.

```python theme={null}
news_branches(
    action="update",
    branch_id="your-branch-id",
    config={
        "schedule_enabled": True,
        "schedule_cron": "0 */6 * * *",       # Every 6 hours
        "schedule_timezone": "America/Chicago",
    },
)
```

### Delete a branch

```python theme={null}
news_branches(action="delete", branch_id="your-branch-id")
```

<Warning>
  You cannot delete the production branch. Deleting a branch removes all articles and events stored under it.
</Warning>

### Branch config reference

| Field                     | Type   | Description                                                       |
| ------------------------- | ------ | ----------------------------------------------------------------- |
| `fetch_config`            | dict   | Source definitions and search queries for the fetch step          |
| `extraction_prompt`       | string | Custom system prompt for the extract/score step                   |
| `extraction_config`       | dict   | Additional extraction settings                                    |
| `filter_prompt`           | string | Custom prompt for the filter step                                 |
| `filter_config`           | dict   | Filter settings, including `min_relevant_score`                   |
| `cluster_prompt`          | string | Custom prompt for the cluster step                                |
| `cluster_config`          | dict   | Clustering settings                                               |
| `summary_prompt`          | string | Custom prompt for the summarize step                              |
| `days_back`               | int    | How many days back to fetch (overridden by `date_from`/`date_to`) |
| `schedule_enabled`        | bool   | Whether the branch runs on a schedule                             |
| `schedule_cron`           | string | Cron expression (5-field, e.g. `"0 8 * * *"`)                     |
| `schedule_timezone`       | string | Timezone for the schedule (default: `America/New_York`)           |
| `newsletter_audience_ids` | list   | Audience IDs to send a newsletter after scheduled production runs |

***

## Viewing events

Events are clusters of related articles that the pipeline has grouped around a single real-world occurrence. Each event has a label, a structured summary, and an article count.

### List events

```python theme={null}
news_events(action="list")
```

Filter by status:

```python theme={null}
# Only pending review
news_events(action="list", status="pending")

# Approved events
news_events(action="list", status="approved")

# Rejected events
news_events(action="list", status="rejected")
```

Limit results and target a specific branch:

```python theme={null}
news_events(
    action="list",
    branch_id="your-branch-id",
    limit=10,
)
```

The output shows each event's label, article count, and a truncated preview of its summary.

***

## Viewing articles

Articles are individual news items with a relevance score and bucket classification.

### List articles

```python theme={null}
news_articles(action="list")
```

Filter by bucket:

```python theme={null}
# Only relevant articles
news_articles(action="list", bucket="relevant")

# Discarded articles
news_articles(action="list", bucket="discard")
```

Limit results and target a specific branch:

```python theme={null}
news_articles(
    action="list",
    branch_id="your-branch-id",
    bucket="relevant",
    limit=50,
)
```

Each article in the output shows its relevance score (e.g. `[8]`), bucket, title, source name, and URL.

<Tip>
  Use `bucket="discard"` with a custom `extraction_prompt` on a test branch to tune your scoring threshold. Run `news_pipeline(action="run", steps=["extract"])` after updating the prompt to re-score existing articles without re-fetching.
</Tip>

***

## Production schedule

The production branch runs automatically on its configured schedule. You do not need to trigger it manually. After each scheduled run, if `newsletter_audience_ids` are configured on the branch, Soria automatically sends a newsletter with the latest events.

To check when the production branch last ran:

```python theme={null}
news_pipeline(action="status")
```