Supported Eval Frameworks & Integrations
ParetoOps does not execute model API calls directly. Instead, it reads the benchmark and evaluation results generated by your existing eval frameworks, analyzes the cost-accuracy tradeoffs on the Pareto frontier, and gates your CI pipeline.
Formats are detected automatically. To specify a format explicitly, pass --format <type> (promptfoo, langsmith, braintrust, deepeval, csv, or pareto).
1. Promptfoo
Section titled “1. Promptfoo”Promptfoo is an open-source CLI and library for evaluating LLM outputs against test cases.
Exporting from Promptfoo
Section titled “Exporting from Promptfoo”Run your Promptfoo evaluation suite and output the results as JSON:
npx promptfoo eval --output promptfoo-results.jsonAnalyzing with ParetoOps
Section titled “Analyzing with ParetoOps”npx pareto-ops analyze promptfoo-results.json --min-acc 0.9CI Gate in GitHub Actions
Section titled “CI Gate in GitHub Actions”- name: Run Promptfoo Evals run: npx promptfoo eval --output results.json
- name: ParetoOps CI Gate run: npx pareto-ops gate results.json --config my-model-config --min-acc 0.9 --max-cost-per-success 0.012. LangSmith
Section titled “2. LangSmith”LangSmith from LangChain logs inputs, outputs, token counts, and eval scores for LLM applications.
Exporting from LangSmith
Section titled “Exporting from LangSmith”Export your evaluation run dataset from the LangSmith UI or using the LangSmith Python SDK:
from langsmith import Clientimport json
client = Client()run_stats = client.read_project(project_name="my-eval-project")# Export trials with configId, costUsd, passed, durationMsAnalyzing with ParetoOps
Section titled “Analyzing with ParetoOps”pareto-ops analyze langsmith-export.json --format langsmith --min-acc 0.853. Braintrust
Section titled “3. Braintrust”Braintrust tracks LLM evaluations, scores, and token usage across prompt iterations.
Exporting from Braintrust
Section titled “Exporting from Braintrust”Export experiment trial results as JSON:
braintrust eval --json-output braintrust-results.jsonAnalyzing with ParetoOps
Section titled “Analyzing with ParetoOps”pareto-ops analyze braintrust-results.json --format braintrust --min-acc 0.94. DeepEval
Section titled “4. DeepEval”DeepEval evaluates LLM pipelines on unit-test metrics (hallucination, faithfulness, G-Eval).
Exporting from DeepEval
Section titled “Exporting from DeepEval”Configure your test runner to dump evaluation results:
deepeval test run --output-file deepeval-results.jsonAnalyzing with ParetoOps
Section titled “Analyzing with ParetoOps”pareto-ops analyze deepeval-results.json --format deepeval --min-acc 0.955. Generic JSON & CSV Format
Section titled “5. Generic JSON & CSV Format”If you use a custom eval harness, you can feed ParetoOps a lightweight JSON array or CSV table.
JSON Schema
Section titled “JSON Schema”[ { "configId": "gpt-4o-mini-v1", "configName": "GPT-4o mini (Prompt v1)", "model": "gpt-4o-mini", "provider": "openai", "costUsd": 0.0012, "passed": true, "durationMs": 350 }]CSV Schema
Section titled “CSV Schema”configId,configName,model,provider,costUsd,passed,durationMsclaude-haiku,Claude Haiku 4.5,claude-haiku-4-5,anthropic,0.0032,true,280gemini-flash,Gemini 2.5 Flash,gemini-2-5-flash,google,0.0015,false,190