Skip to content

Add a CI gate

pareto-ops gate checks one configuration against your limits. It exits with code 0 when the configuration passes, 1 when it doesn’t (or the input can’t be read) and 2 for a usage error, so any CI system can use it. If a configuration has fewer than 10 trials, the gate warns that the result is preliminary.

Terminal window
npx pareto-ops gate results.json --config claude-haiku-4-5 --min-acc 0.9 --max-cost-per-success 0.01
Evaluating CI Gate for Config: Claude Haiku 4.5
- Success Rate: 92.0% (Threshold: 90.0%)
- Cost per Success: $0.0035
- Status: PARETO OPTIMAL
✅ CI Gate PASSED. Configuration meets economic & accuracy requirements.
Flag Meaning
--config <id> The configuration to check, by configId
--min-acc <n> Minimum success rate, from 0 to 1 (default 0.9)
--max-cost-per-success <n> Maximum cost per successful task, in USD
--format <format> Input format (default: detected automatically)

The ParetoOps CI Gate action is the quickest way in. It installs ParetoOps and runs it on Linux, macOS and Windows runners. Without a key it runs the threshold gate and writes a short job summary:

- uses: actions/checkout@v7
- run: npm ci && npm run evals -- --output results.json
- uses: certiqa/paretoops-action@v1
with:
results: results.json
config: claude-haiku-4-5
min-acc: 0.9
max-cost-per-success: 0.01

With a free key: PR comment and full job summary

Section titled “With a free key: PR comment and full job summary”

Add a free license key and the action runs pareto-ops ci. It writes the full job summary with the frontier chart and keeps one comment up to date on the pull request:

permissions:
contents: read
pull-requests: write # the pull request comment
steps:
- uses: actions/checkout@v7
- run: npm ci && npm run evals -- --output results.json
- uses: certiqa/paretoops-action@v1
with:
results: results.json
config: claude-haiku-4-5
min-acc: 0.9
license-key: ${{ secrets.PARETO_LICENSE_KEY }}

PARETO_LICENSE_KEY is the key you added as a repository secret. The action posts the comment with the workflow’s own GITHUB_TOKEN, which GitHub creates for every run, so there is nothing else to set up. Pull requests from forks don’t get your secrets, so on those the action runs the keyless threshold check, and the change is fully gated once it reaches your base branch.

If you leave out config, the action gates the sweet spot: the cheapest configuration per success that clears min-acc. Every input and output is listed in the action’s README.

The action is a thin wrapper, so you can also run the CLI directly. Pass the key and token as environment variables:

- name: ParetoOps CI gate
run: npx pareto-ops ci results.json --config claude-haiku-4-5 --min-acc 0.9
env:
PARETO_LICENSE_KEY: ${{ secrets.PARETO_LICENSE_KEY }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

On Linux runners you can use the container image instead: uses: docker://ghcr.io/certiqa/pareto-ops:v1 with args: ci results.json ....

Need a free license key? Generate yours in seconds at paretoops.dev/free. The key is verified offline and the CLI sends no telemetry; we store the email you register with to issue and support the key.

Use pareto-ops gate for threshold gating in any shell or pipeline:

evals:
script:
- npm run evals -- --output results.json
- npx pareto-ops gate results.json --config claude-haiku-4-5 --min-acc 0.9

Thresholds catch a configuration that’s bad in absolute terms. Pro adds the checks teams need on every pull request:

  • a regression gate against your last known good run, paired task by task (ci --baseline, or the action’s baseline input), so a small drop is caught even when the result still clears your thresholds;
  • baselines stored on a git branch, with no artifact storage to manage;
  • early stopping, so a failing eval run ends before it spends the full budget.

Join the interest list to hear when Pro launches.