Skip to content

Supported Eval Frameworks & Integrations

ParetoOps does not execute model API calls directly. Instead, it reads the benchmark and evaluation results generated by your existing eval frameworks, analyzes the cost-accuracy tradeoffs on the Pareto frontier, and gates your CI pipeline.

Formats are detected automatically. To specify a format explicitly, pass --format <type> (promptfoo, langsmith, braintrust, deepeval, csv, or pareto).


Promptfoo is an open-source CLI and library for evaluating LLM outputs against test cases.

Run your Promptfoo evaluation suite and output the results as JSON:

Terminal window
npx promptfoo eval --output promptfoo-results.json
Terminal window
npx pareto-ops analyze promptfoo-results.json --min-acc 0.9
- name: Run Promptfoo Evals
run: npx promptfoo eval --output results.json
- name: ParetoOps CI Gate
run: npx pareto-ops gate results.json --config my-model-config --min-acc 0.9 --max-cost-per-success 0.01

LangSmith from LangChain logs inputs, outputs, token counts, and eval scores for LLM applications.

Export your evaluation run dataset from the LangSmith UI or using the LangSmith Python SDK:

from langsmith import Client
import json
client = Client()
run_stats = client.read_project(project_name="my-eval-project")
# Export trials with configId, costUsd, passed, durationMs
Terminal window
pareto-ops analyze langsmith-export.json --format langsmith --min-acc 0.85

Braintrust tracks LLM evaluations, scores, and token usage across prompt iterations.

Export experiment trial results as JSON:

Terminal window
braintrust eval --json-output braintrust-results.json
Terminal window
pareto-ops analyze braintrust-results.json --format braintrust --min-acc 0.9

DeepEval evaluates LLM pipelines on unit-test metrics (hallucination, faithfulness, G-Eval).

Configure your test runner to dump evaluation results:

Terminal window
deepeval test run --output-file deepeval-results.json
Terminal window
pareto-ops analyze deepeval-results.json --format deepeval --min-acc 0.95

If you use a custom eval harness, you can feed ParetoOps a lightweight JSON array or CSV table.

[
{
"configId": "gpt-4o-mini-v1",
"configName": "GPT-4o mini (Prompt v1)",
"model": "gpt-4o-mini",
"provider": "openai",
"costUsd": 0.0012,
"passed": true,
"durationMs": 350
}
]
configId,configName,model,provider,costUsd,passed,durationMs
claude-haiku,Claude Haiku 4.5,claude-haiku-4-5,anthropic,0.0032,true,280
gemini-flash,Gemini 2.5 Flash,gemini-2-5-flash,google,0.0015,false,190