> For the complete documentation index, see [llms.txt](/llms.txt).
> Markdown versions of each page are available by appending .md to any URL.

# Measure and improve a factory

Measure factory activity and costs, evaluate completed conversations, compare agent configurations, and turn failures into follow-up work.

Warp Factories tracks what your factory produces and how well it performs, so you can spot a problem, test a fix, and decide whether to keep it.

| Feature | What it tells you | Keep in mind |
| --- | --- | --- |
| Dashboard metrics | How much work the factory produced, and what it cost. | Some metrics require the GitHub App, and measurement runs count as activity. |
| Scorers | Whether completed conversations meet criteria you define. | Scorers classify conversations; they don’t grade quality on a numeric scale. |
| Benchmarks | How different configurations perform on the same tasks. | Results don’t pick a winner or include full cost. |
| Self-improvement | Which repeated failures need follow-up work. | It can open pull requests, but you review and adopt every change yourself. |

## Read dashboard metrics

The control room dashboard shows activity, cost, autonomy, and evaluation results:

| Metric | What it shows | Keep in mind |
| --- | --- | --- |
| **Total runs** | Agent runs, with breakdowns by agent type, status, source, model, and more. | Includes Warp’s own measurement and improvement runs. |
| **PRs opened** | Pull requests created from factory work. | Collected differently from **PRs merged**. |
| **PRs merged** | Pull requests that merged. | Requires the GitHub App; counting starts when you install it. |
| **Autonomy %** | The share of merged work that shipped without human edits. | Based on recorded PR signals, not a quality judgment. |
| **Time saved** | A rough estimate of engineer-hours saved by merged factory PRs. | A rule of thumb for spotting trends, not a billing or ROI figure. |
| **PR latency** | How long pull requests spend in each stage. | Can be incomplete, depending on webhook coverage. |
| **Cost per PR** | An estimate of credits spent per pull request. | A lower-bound estimate. Dollar amounts are for display, not billing. |
| **Most expensive PRs** | The highest-cost pull requests. | Full detail requires the GitHub App. |
| **Scorer cards** | Results from your Scorers. | Reflects the criteria your team defined. |
| **Self-improvement PRs** | The three newest Self-improvement pull requests. | Not affected by the dashboard date range. |

**Time saved** converts merged factory pull requests into an estimate of engineer-hours, and the per-teammate view divides that estimate by your team size. **Cost per PR** shows the median by default, plus mean, **By complexity**, and **By size** views. Treat both as directional estimates rather than financial figures.

Caution

Merge, latency, and per-PR detail metrics require the GitHub App, and collection starts when you install it; earlier history isn’t backfilled. **PRs opened** and **PRs merged** come from different sources, so their ratio isn’t a merge rate. Run counts include evaluation, benchmark, and Self-improvement runs.

Use the dashboard to pick which runs to investigate, not to conclude what caused a change. A higher run count with a flat PR count could mean harder tasks, retries, measurement activity, or unclear agent instructions. Open the runs to find out.

## Configure Scorers

A **Scorer** uses an LLM judge to classify completed conversations against criteria you write, such as “did the agent run the tests before opening a PR?” Scorers classify conversations rather than grading them on a numeric scale. Keep each Scorer focused on one question so its failures point to a specific fix. Benchmarks also run a separate built-in scorer, **Correctness**, which you don’t configure.

Configure these fields:

-   **Judge instructions** - The criteria the judge checks for.
-   **Classifications** - The labels the judge can assign, each with a score.
-   **Pass threshold** - The score a conversation needs to pass.
-   **Sample rate** - The portion of eligible conversations to evaluate.
-   **Judge model** - The model that acts as the judge.

You can apply a Scorer to all agents or only the ones you choose, and pause it at any time. Shortly after an eligible conversation completes, the judge evaluates it and records a classification, a score, and its reasoning.

| Mode | Use it when | You get |
| --- | --- | --- |
| Manual | You want to score one conversation or test new judge instructions. | An evaluation of the selected conversation. |
| Periodic | You want an ongoing sample to track over time. | A baseline to compare against after a change. |

Changing **Pass threshold** only changes how past scores display as pass or fail. The recorded classification, score, and reasoning stay the same.

## Compare configurations with benchmarks

A benchmark compares configurations of a single agent on the same fixed tasks, so you can test a model, harness, or runner change before adopting it. A suite includes:

-   **Agent** - The agent whose configurations you compare.
-   **Tasks** - Fixed prompts with success criteria. Task definitions freeze when you launch.
-   **Configurations** - The harness, model, and runner combinations to test.
-   **Scorers** - Your classification Scorers, applied to every trial.
-   **Repetitions** - The number of trials per task and configuration.

You can create a benchmark task from a completed run’s detail pane, and Warp copies the run’s input into the task. Add success criteria before you launch.

Every benchmark also runs **Correctness**, a built-in scorer that marks each trial as pass or fail against the task’s success criteria. Results include a per-configuration scoreboard, pass rate by Scorer, a cost and quality scatter plot, and per-task comparisons. Warp doesn’t combine these signals into one score or pick a winner; you weigh the results and decide. Credit totals exclude inference usage, so they understate the full cost.

Run enough repetitions to separate a real difference from one lucky trial, and weigh factors the suite doesn’t capture, like required tools, security policy, provider availability, and how serious each failure type is.

## Configure and review Self-improvement

Turn on **Self-improvement** for each Scorer whose failures you want investigated automatically. The factory-level **Analysis model** setting chooses the model that analyzes failures and groups related findings. Factories managed from files in a GitHub repo can’t change this setting in the control room.

Self-improvement investigates failing results, clusters related causes, and files follow-up tasks as ordinary agent runs. A follow-up run can propose changes to application code, or to prompts, skills, and configuration when the factory’s configuration repo is available. It can open pull requests, but a pull request isn’t guaranteed, and nothing is adopted without your review.

The **Self-improvement PRs** card shows the three newest Self-improvement pull requests, regardless of the dashboard date range. Each pull request includes a **Regressions addressed** section that links the failing runs, the Scorer result, and the investigation run, so you can trace a proposed change back to its evidence.

## Run a practical improvement loop

Change one measurable thing at a time:

1.  **Define a Scorer.** Pick one agent and one failure mode you can observe. Write the judge instructions and classifications, then compare the judge’s results against a few conversations you review yourself.
2.  **Collect a baseline.** Let periodic scoring run until results reflect normal work. Record the Scorer settings, date range, and relevant costs.
3.  **Inspect failures.** Read the judge’s reasoning and the underlying conversations. Look for causes like missing context, unclear instructions, or missing tools. Turn on Self-improvement when the same failure keeps repeating.
4.  **Benchmark a candidate.** Compare configurations of that agent on the same tasks, with enough repetitions to trust the difference.
5.  **Review and adopt.** If the evidence supports the change, make it. Review Self-improvement pull requests with the same standards as human-authored ones.
6.  **Keep monitoring.** Leave the Scorer active and compare new results against your baseline. Pause or revise the Scorer when its criteria no longer match what your team needs.

A small, measured change gives you stronger evidence than a broad reconfiguration with several possible causes.

## Next step

Record an adopted change in [factory definitions as code](./factory-as-code) so your team can review the factory configuration.