Tracely – Turn AI agent production failures into CI regression tests

Trace-native CI/CD for AI agents Production failures become regression tests. Tracely grades every agent trace as it lands, clusters the failures into issues, freezes the bad runs into hermetic replayable cases — and blocks the pull request that would ship them again. production trace → failure detection → regression test → CI gate Open the dashboard Star on GitHub hallucination · judge cited sources0.98 PASS grounded answer0.94 PASS no fabrication0.99 PASS gate history · 7d pass rate 96% · 1 blocked merge tracely — traces / tr_9f2c41env=prod rec agentsupport_agent 8.4s llmplan · claude-sonnet-5 1.2s toolsearch_orders 0.9s toolrefund_api ▲ timeout · unhandled 3.1s llmrespond · claude-sonnet-5 1.4s evaluatorsgrounded · PASSrefund_policy · PASSresolution · FAIL GATE ✗ BLOCKED PR #214 reintroduces cluster #12 instruments any agent stack OpenAIAnthropicGoogle GeminiLangChainLangGraphLiteLLMCrewAIMistralany OTLP sourceOpenAIAnthropicGoogle GeminiLangChainLangGraphLiteLLMCrewAIMistralany OTLP source The loop From production incident to CI gate. Four moves, no hand-authored datasets. Each one is derived from the last, and the trace starts it all. 01 Trace Every production run streams in over OTLP — durable blob first, one indexed row per span. Agent, conversation and tool semantics are first-class columns, not strings. 02 Detect Online evaluators grade each trace as it lands — LLM-as-judge at conversation, run or span level. One FAIL on a blocking evaluator flips the verdict. 03 Freeze One click promotes a failing trace into a hermetic case: recorded input, tool and LLM fixtures bundled, a fail-to-pass contract attached. 04 Gate tracely replay re-runs the suite on every PR — offline, deterministic, $0 in model spend — and exits non-zero on the merge that regresses it. You never author a test set. Production already wrote the perfect failing example — Tracely freezes it and guards it forever. 1 click → frozen case$0 hermetic replay3 lines to instrumentevery PR gated Derived from the trace Everything downstream of one artifact. Scores, clusters, cases, gates and trends are all computed from the trace — so they never disagree with each other, or with production. Evaluators as columns Judges live where you look: every evaluator is a column on the trace table, and scores stream into the grid live over SSE as they run. Failure clustering Structural + semantic clustering groups failing traces into named issues — with a description, a proposed fix and a suggested evaluator to catch it next time. Hermetic replay Cases replay against recorded tool and LLM fixtures — no API keys, no model spend, no flakes. Add --live when you want real calls. The PR gate tracely gate exits non-zero, posts the commit status and upserts a PR comment. Two lines of YAML in the workflow you already have. Trends & meta-analysis Daily failure and gate pass-rates, plus per-agent cross-metric analysis: Spearman correlations, z-score outliers, LLM-synthesized findings. Judge calibration Label judge verdicts against human review, get per-evaluator agreement, and catch an over-flagging judge before you let it gate a release. Ship The PR that re-breaks prod never merges. The gate replays your promoted cases on every pull request, exits non-zero on failure, and posts the verdict as a commit status and PR comment. .github/workflows/tracely.yml # .github/workflows/tracely.ymlon: pull_requestjobs: gate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 # → run your agent here; traces stream in with env=ci - uses: Jwuthri/Tracely/.github/actions/tracely-gate@master with: agent: support-agent api: https://tracely.your-co.dev key: ${{ secrets.TRACELY_KEY }} fix: retry refund_api on timeout #214 · mira wants to merge 3 commits Tracely gate / gate (pull_request) Required · failing after 42s Details 11 passed · 1 failed — case rc_118 (cluster #12 · refund tool timeout) regressed at step 4/6. Trajectory diff attached. Merge blocked. 8f31c2a — await refund_api(order) 12/12 passed · gate green, merge unblocked Instrument Three lines in. Every span out. tracely.init() switches on auto-instrumentation for whatever your agent already uses — OpenAI, Anthropic, LangChain / LangGraph, LiteLLM — and stamps agent, conversation and turn onto every span. Zero span code in your codebase; manual context managers remain the escape hatch. tokens & costtools & retrievalsthinking spansmulti-turn conversations Read the SDK docs agent.pypip install "tracely-ai[openai]" import tracely_sdk as tracelyfrom openai import OpenAI tracely.init(endpoint="https://tracely.your-co.dev", api_key=os.environ["TRACELY_KEY"], service_name="support-agent", env="prod") with tracely.trace(agent="support-agent", conversation="conv-42"): OpenAI().chat.completions.create(...) # traced — zero span code Tracely Ship agents that don't regress. Open the dashboard Star on GitHub