LLM Cost Economics & Cost of Pass
What is cost per successful task (or cost of pass) for LLMs?
Cost per successful task (also called cost-of-pass in academic literature) is the total inference spend on a set of benchmark tasks divided by the number of tasks that actually passed. Because failed model calls still cost money, a model with a low token price that fails often frequently costs significantly more per useful output than a more expensive, reliable model.
Why does comparing price per token mislead LLM buyers (price reversal)?
Price per token only reflects nominal provider rates per million tokens. It ignores how many tokens a model consumes to solve a prompt, how many reasoning or "thinking" tokens it generates, and how often it fails. In published research, in 32% of model pairs studied, the model with the lower list price cost more in total—in extreme cases up to 28x more—due to verbosity and reasoning token overhead.
How do reasoning tokens and prompt caching impact LLM eval budgets?
Reasoning models (like o1, o3, DeepSeek R1, or Claude with extended thinking) generate hidden internal reasoning tokens that are billed as output tokens, drastically multiplying cost per call. Conversely, prompt caching can discount repeated prefix tokens by 50% to 90%. ParetoOps includes a built-in pricing catalog covering cache-read, cache-write, and reasoning token rates so your economic evaluations reflect real invoice pricing.
Should I always use the most capable frontier model?
Not necessarily. Research demonstrates that routing requests to the cheapest model capable of handling each task can match frontier performance at up to 98% lower cost. However, on difficult reasoning tasks where failures trigger costly human intervention or cascading retries, a frontier model is often the cheapest per successful task. Optimal choice requires empirical measurement on your specific workload.
What is a Pareto frontier in LLM evaluation, and what does dominated mean?
In multi-objective optimization, a model configuration is dominated when an alternative is at least as accurate and no more expensive per successful task, and strictly better on one of those dimensions. The set of non-dominated configurations forms the Pareto frontier (marked OPTIMAL). ParetoOps finds the frontier and recommends the "sweet spot": the lowest-cost configuration that meets your quality threshold.
How many benchmark tasks do I need for trustworthy LLM evaluation?
Small sample sizes (e.g. 20–30 tasks) produce noisy success rates with wide error margins. ParetoOps computes a 95% Wilson score confidence interval alongside every success rate to show the statistical lower bound. For reliable CI gating and model selection, we recommend at least 30 to 100 representative tasks per configuration.
CI/CD Gating & Eval Integrations
I only evaluate one model. Is ParetoOps still useful?
Yes. The main use is checking every prompt, retrieval or model-version change against your last good run, which works with a single model. The model comparison is the second use: when a new model ships, a price changes, or you want to route easy requests to a cheaper option, you run the same eval on the alternatives and see which one costs least per successful task. A configuration is more than a model, so two prompts on the same model count as two options.
How do I catch prompt and model regressions in CI/CD before merging?
With ParetoOps CI gating (`pareto-ops gate` or `pareto-ops ci`), every pull request can be evaluated against your minimum accuracy and maximum cost-per-success thresholds. The gate exits with code 0 on pass or 1 on failure, blocking regressions in GitHub Actions, GitLab CI, Jenkins, or any CI runner. A threshold only checks a fixed number, so a change can clear it and still be worse than before. Pro adds a paired, task-by-task comparison against your last good run, with confidence intervals, which catches that.
How does early stopping save money on failing LLM evaluation runs?
Running full benchmark suites on failing configurations wastes significant API spend. ParetoOps Pro watches your eval stream in real time and uses sequential statistical testing to stop an eval as soon as the evidence shows the run cannot clear your accuracy or cost thresholds. In our sample, a run clearly below its bar stopped after 86 of 200 tasks (57% of spend avoided). Savings depend on how far below the bar a run is.
Which LLM eval frameworks does ParetoOps support?
ParetoOps reads export outputs directly from Promptfoo, LangSmith, Braintrust, and DeepEval, as well as standard CSV and JSON trial logs. Each trial requires a configuration identifier, cost in USD, and pass/fail boolean (latency in milliseconds is optional).
Does ParetoOps call model APIs or send my evaluation data to external servers?
No. ParetoOps is 100% offline and deterministic. It performs local arithmetic on the benchmark results your eval framework generates. It makes no API calls to LLM providers, sends zero telemetry, and verifies license keys completely offline.
Why is ParetoOps closed source, and how can I verify what it does?
ParetoOps ships as compiled native binaries. In place of source, every release artifact is scanned before publication, output is deterministic, JSON and report formats are stable, and the binary never calls a model API and sends no telemetry. You can check the no-network claim by running it with networking disabled in CI. Details are on the Trust page at paretoops.dev/trust.
Can I use ParetoOps in GitHub Actions or GitLab CI for free?
Yes. The `pareto-ops gate` command is free and needs no account or key, and `pareto-ops ci` (job summary and pull request comment) is free with a free offline key from paretoops.dev/free. Both run in any CI pipeline. On GitHub, the ParetoOps CI Gate action (`uses: certiqa/paretoops-action@v1`) runs either one in a single step. The paired regression gate against a baseline is Pro.
Is there a Python SDK for ParetoOps?
Yes. Install `pip install pareto-ops` to compute Pareto frontiers, model recommendations, and confidence bounds directly within Python workflows or Jupyter notebooks.
Plans & Licensing
What does ParetoOps Pro add over Free?
Pro is for engineering teams shipping LLM changes through pull requests: a paired, task-by-task regression gate against your last good run (with confidence intervals), baselines stored on a git branch, early stopping for failing eval runs, step-level cost attribution for agentic runs, and a 3D frontier with P95 latency. Pull request comments are part of Free with a free key.
When is ParetoOps Pro available and how do I get early access?
Pro and Enterprise are rolling out soon. Pro will cost $99 per month, billed yearly, per GitHub organization. Join the interest list to hear when it launches and to request a 30-day trial license key before general availability. Early customers keep their launch price.
How is ParetoOps Pro licensed?
Pro is licensed per GitHub organization or team with an offline cryptographically signed license key stored in CI secrets. It never phones home.
Do I need an account to use the Free version?
Local analysis (`analyze`) and threshold gating (`gate`) need no sign-up, credit card or key. The CI gate with PR comments (`ci`) needs a free license key, which you generate at paretoops.dev/free with your email.