Measure and improve a factory

Warp Factories tracks what your factory produces and how well it performs, so you can spot a problem, test a fix, and decide whether to keep it.

Feature	What it tells you	Keep in mind
Dashboard metrics	How much work the factory produced, and what it cost.	Some metrics require the GitHub App, and measurement runs count as activity.
Scorers	Whether completed conversations meet criteria you define.	Scorers classify conversations; they don’t grade quality on a numeric scale.
Benchmarks	How different configurations perform on the same tasks.	Results don’t pick a winner or include full cost.
Self-improvement	Which repeated failures need follow-up work.	It can open pull requests, but you review and adopt every change yourself.

Read dashboard metrics

The control room dashboard shows activity, cost, autonomy, and evaluation results:

Metric	What it shows	Keep in mind
Total runs	Agent runs, with breakdowns by agent type, status, source, model, and more.	Includes Warp’s own measurement and improvement runs.
PRs opened	Pull requests created from factory work.	Collected differently from PRs merged.
PRs merged	Pull requests that merged.	Requires the GitHub App; counting starts when you install it.
Autonomy %	The share of merged work that shipped without human edits.	Based on recorded PR signals, not a quality judgment.
Time saved	A rough estimate of engineer-hours saved by merged factory PRs.	A rule of thumb for spotting trends, not a billing or ROI figure.
PR latency	How long pull requests spend in each stage.	Can be incomplete, depending on webhook coverage.
Cost per PR	An estimate of credits spent per pull request.	A lower-bound estimate. Dollar amounts are for display, not billing.
Most expensive PRs	The highest-cost pull requests.	Full detail requires the GitHub App.
Scorer cards	Results from your Scorers.	Reflects the criteria your team defined.
Self-improvement PRs	The three newest Self-improvement pull requests.	Not affected by the dashboard date range.

Time saved converts merged factory pull requests into an estimate of engineer-hours, and the per-teammate view divides that estimate by your team size. Cost per PR shows the median by default, plus mean, By complexity, and By size views. Treat both as directional estimates rather than financial figures.

Caution

Merge, latency, and per-PR detail metrics require the GitHub App, and collection starts when you install it; earlier history isn’t backfilled. PRs opened and PRs merged come from different sources, so their ratio isn’t a merge rate. Run counts include evaluation, benchmark, and Self-improvement runs.

Use the dashboard to pick which runs to investigate, not to conclude what caused a change. A higher run count with a flat PR count could mean harder tasks, retries, measurement activity, or unclear agent instructions. Open the runs to find out.

Configure Scorers

A Scorer uses an LLM judge to classify completed conversations against criteria you write, such as “did the agent run the tests before opening a PR?” Scorers classify conversations rather than grading them on a numeric scale. Keep each Scorer focused on one question so its failures point to a specific fix. Benchmarks also run a separate built-in scorer, Correctness, which you don’t configure.

Configure these fields:

• Judge instructions - The criteria the judge checks for.
• Classifications - The labels the judge can assign, each with a score.
• Pass threshold - The score a conversation needs to pass.
• Sample rate - The portion of eligible conversations to evaluate.
• Judge model - The model that acts as the judge.

You can apply a Scorer to all agents or only the ones you choose, and pause it at any time. Shortly after an eligible conversation completes, the judge evaluates it and records a classification, a score, and its reasoning.

Mode	Use it when	You get
Manual	You want to score one conversation or test new judge instructions.	An evaluation of the selected conversation.
Periodic	You want an ongoing sample to track over time.	A baseline to compare against after a change.

Changing Pass threshold only changes how past scores display as pass or fail. The recorded classification, score, and reasoning stay the same.

Compare configurations with benchmarks

A benchmark compares configurations of a single agent on the same fixed tasks, so you can test a model, harness, or runner change before adopting it. A suite includes:

• Agent - The agent whose configurations you compare.
• Tasks - Fixed prompts with success criteria. Task definitions freeze when you launch.
• Configurations - The harness, model, and runner combinations to test.
• Scorers - Your classification Scorers, applied to every trial.
• Repetitions - The number of trials per task and configuration.

You can create a benchmark task from a completed run’s detail pane, and Warp copies the run’s input into the task. Add success criteria before you launch.

Every benchmark also runs Correctness, a built-in scorer that marks each trial as pass or fail against the task’s success criteria. Results include a per-configuration scoreboard, pass rate by Scorer, a cost and quality scatter plot, and per-task comparisons. Warp doesn’t combine these signals into one score or pick a winner; you weigh the results and decide. Credit totals exclude inference usage, so they understate the full cost.

Run enough repetitions to separate a real difference from one lucky trial, and weigh factors the suite doesn’t capture, like required tools, security policy, provider availability, and how serious each failure type is.

Configure and review Self-improvement

Turn on Self-improvement for each Scorer whose failures you want investigated automatically. The factory-level Analysis model setting chooses the model that analyzes failures and groups related findings. Factories managed from files in a GitHub repo can’t change this setting in the control room.

Self-improvement investigates failing results, clusters related causes, and files follow-up tasks as ordinary agent runs. A follow-up run can propose changes to application code, or to prompts, skills, and configuration when the factory’s configuration repo is available. It can open pull requests, but a pull request isn’t guaranteed, and nothing is adopted without your review.

The Self-improvement PRs card shows the three newest Self-improvement pull requests, regardless of the dashboard date range. Each pull request includes a Regressions addressed section that links the failing runs, the Scorer result, and the investigation run, so you can trace a proposed change back to its evidence.

Run a practical improvement loop

Change one measurable thing at a time:

1. Define a Scorer. Pick one agent and one failure mode you can observe. Write the judge instructions and classifications, then compare the judge’s results against a few conversations you review yourself.
2. Collect a baseline. Let periodic scoring run until results reflect normal work. Record the Scorer settings, date range, and relevant costs.
3. Inspect failures. Read the judge’s reasoning and the underlying conversations. Look for causes like missing context, unclear instructions, or missing tools. Turn on Self-improvement when the same failure keeps repeating.
4. Benchmark a candidate. Compare configurations of that agent on the same tasks, with enough repetitions to trust the difference.
5. Review and adopt. If the evidence supports the change, make it. Review Self-improvement pull requests with the same standards as human-authored ones.
6. Keep monitoring. Leave the Scorer active and compare new results against your baseline. Pause or revise the Scorer when its criteria no longer match what your team needs.

A small, measured change gives you stronger evidence than a broad reconfiguration with several possible causes.

Next step

Record an adopted change in factory definitions as code so your team can review the factory configuration.