Product

One check for every prompt and model change, in three commands

ParetoOps tells you, on every pull request, whether a prompt or model change made your AI worse or more expensive. It doesn't run evals and never calls a model: it reads the results your eval tool already wrote and does the cost-per-successful-task math. analyze shows what's worth paying for,gate enforces your limits in CI, and watch stops a failing run before it burns its full budget.

Free

analyze — find the frontier

Every configuration you tested, ranked by what a successful task actually cost, with the ones nothing beats on both cost and accuracy marked out.

  • 95% Wilson lower bound: how low the real success rate could plausibly be, so a handful of lucky trials doesn't look like a result.
  • OPTIMAL ≠ meets your bar: it means nothing beats that option on cost andaccuracy together. Gemini Flash is OPTIMAL here — cheapest, and nothing both cheaper and more accurate exists — even though it misses your 90% --min-acc. Only the sweet spot clears the bar.
  • DOMINATED means overpaying: here, the legacy Opus 4.8 setup costs about 9× more per success than Haiku 4.5 and is less accurate — not because the model is bad, but because it's an old default nobody revisited. That drift is exactly what a one-off benchmark misses, and what this catches.
  • Sweet spot: the cheapest frontier configuration that clears your accuracy bar, set with --min-acc, computed for you, not eyeballed off a chart.
  • Prices tokens correctly: the built-in catalog accounts for prompt caching and reasoning-token rates.
50%60%70%80%90%100%$0.002$0.005$0.01$0.02$0.05Cost per successful task (log scale) →Success rate ↑Your accuracy bar: 90%Claude Sonnet 5Claude Haiku 4.5Claude Opus 4.8 (legacy)Gemini 2.5 Flash
On the frontierSweet spot: cheapest frontier option above the barDominated: costs more for the same or worse results
Free Pro

gate — stop regressions before they ship

A configuration that fails your thresholds fails the build. On Pro, every pull request is also compared with your last known good run, task by task, with the diff posted as a comment.

What "paired task by task" means: the gate lines up each task's result in the new run against the same task in the baseline run, and tests the differences, with a confidence interval, instead of comparing two averages. Averages hide a regression: a change can fix a few tasks and break more, and run-to-run noise can look like either. In one study, rewording a prompt moved results 11 to 58 times more than rerunning it (Chen, Qian et al., 2026), so a drop of a few points on a single run tells you little. On the sample data, a change that still clears a 50% accuracy bar is significantly worse than the baseline on both accuracy and cost, and the gate fails it (see the report).

  • Works in any CI (Free): threshold gating, job summaries, and PR comments in GitHub Actions with a free offline key, through the ParetoOps CI Gate action (uses: certiqa/paretoops-action@v1).
  • Regression gate (Pro): pareto-ops ci compares the pull request's run with your last good run on the same tasks, so a drop that still clears your threshold doesn't slip through.
  • Baselines on a git branch (Pro): no artifact storage or database — the last good run is just a commit.
  • Survives price changes (Pro): re-prices old and new runs from token counts, so a provider's price change isn't mistaken for a regression.

See a full sample Pro report →

ParetoOpscommented on this pull request❌ FAILED
🔁 Change vs Baseline (200 shared tasks) — WORSE
MetricBaselineThis changeChange (95% CI)Verdict
Accuracy81.5%66.5%-15.0pp (-22.5pp to -7.5pp)WORSE
Cost / Success$0.0155$0.0219×1.42 (×1.24–×1.64)WORSE

The change still clears the 50% accuracy threshold, so a threshold check alone would have passed it.

Excerpt of the CI gate's pull request comment with the Pro regression table (full report). Real output from the sample regression dataset.
Pro

watch — stop paying for evals that already failed

Wraps any eval runner that writes one result per line, and ends the run as soon as the result is statistically clear — not after every task has been paid for.

  • Stops on your limits: minimum accuracy, maximum cost per success, a hard --max-spend cap, or doing worse than a baseline run.
  • Statistically safe by default: checks after every task while keeping the false-stop rate at or below 5% (--alpha), and waits for at least 20 trials (--min-tasks) before it will stop at all.
  • Wraps any runner: point it at the command that writes your eval's results, one JSON line per task, and it tails the file.

The same analysis, from Python

pip install pareto-ops installs the identical native CLI plus a Python API, for teams that want the frontier in a notebook or a pipeline instead of a terminal.

  • Free: ParetoOptimizer, every importer, the pricing catalog.
  • Pro: the 3D frontier (enable_3d_optimization) and confidence-bound ranking (use_statistical_confidence), unlocked with a license_key.

Run it on your own eval results

Free, no sign-up, no telemetry. Pro adds the regression gate, baselines and early stopping.