Add a CI gate
pareto-ops gate checks one configuration against your limits. It exits with code 0 when the
configuration passes, 1 when it doesn’t (or the input can’t be read) and 2 for a usage error,
so any CI system can use it. If a configuration has fewer than 10 trials, the gate warns that the
result is preliminary.
npx pareto-ops gate results.json --config claude-haiku-4-5 --min-acc 0.9 --max-cost-per-success 0.01Evaluating CI Gate for Config: Claude Haiku 4.5- Success Rate: 92.0% (Threshold: 90.0%)- Cost per Success: $0.0035- Status: PARETO OPTIMAL
✅ CI Gate PASSED. Configuration meets economic & accuracy requirements.| Flag | Meaning |
|---|---|
--config <id> |
The configuration to check, by configId |
--min-acc <n> |
Minimum success rate, from 0 to 1 (default 0.9) |
--max-cost-per-success <n> |
Maximum cost per successful task, in USD |
--format <format> |
Input format (default: detected automatically) |
GitHub Actions
Section titled “GitHub Actions”The ParetoOps CI Gate action is the quickest way in. It installs ParetoOps and runs it on Linux, macOS and Windows runners. Without a key it runs the threshold gate and writes a short job summary:
- uses: actions/checkout@v7- run: npm ci && npm run evals -- --output results.json- uses: certiqa/paretoops-action@v1 with: results: results.json config: claude-haiku-4-5 min-acc: 0.9 max-cost-per-success: 0.01With a free key: PR comment and full job summary
Section titled “With a free key: PR comment and full job summary”Add a free license key and the action runs pareto-ops ci. It writes the full job summary with the
frontier chart and keeps one comment up to date on the pull request:
permissions: contents: read pull-requests: write # the pull request comment
steps: - uses: actions/checkout@v7 - run: npm ci && npm run evals -- --output results.json - uses: certiqa/paretoops-action@v1 with: results: results.json config: claude-haiku-4-5 min-acc: 0.9 license-key: ${{ secrets.PARETO_LICENSE_KEY }}PARETO_LICENSE_KEY is the key you added as a repository secret. The action posts the comment with the
workflow’s own GITHUB_TOKEN, which GitHub creates for every run, so there is nothing else to set up. Pull
requests from forks don’t get your secrets, so on those the action runs the keyless threshold check, and the
change is fully gated once it reaches your base branch.
If you leave out config, the action gates the sweet spot: the cheapest configuration per success that clears
min-acc. Every input and output is listed in the
action’s README.
Without the action
Section titled “Without the action”The action is a thin wrapper, so you can also run the CLI directly. Pass the key and token as environment variables:
- name: ParetoOps CI gate run: npx pareto-ops ci results.json --config claude-haiku-4-5 --min-acc 0.9 env: PARETO_LICENSE_KEY: ${{ secrets.PARETO_LICENSE_KEY }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}On Linux runners you can use the container image instead:
uses: docker://ghcr.io/certiqa/pareto-ops:v1 with args: ci results.json ....
Need a free license key? Generate yours in seconds at paretoops.dev/free. The key is verified offline and the CLI sends no telemetry; we store the email you register with to issue and support the key.
GitLab CI and other CI systems
Section titled “GitLab CI and other CI systems”Use pareto-ops gate for threshold gating in any shell or pipeline:
evals: script: - npm run evals -- --output results.json - npx pareto-ops gate results.json --config claude-haiku-4-5 --min-acc 0.9In Pro (coming soon)
Section titled “In Pro (coming soon)”Thresholds catch a configuration that’s bad in absolute terms. Pro adds the checks teams need on every pull request:
- a regression gate against your last known good run, paired task by task (
ci --baseline, or the action’sbaselineinput), so a small drop is caught even when the result still clears your thresholds; - baselines stored on a git branch, with no artifact storage to manage;
- early stopping, so a failing eval run ends before it spends the full budget.
Join the interest list to hear when Pro launches.