I built a free tool to evaluate AI agent outputs (human labels and LLM judges)

Notifications You must be signed in to change notification settings

Fork 0

Star 0

BranchesTags

Open more actions menu

Folders and files

NameName

Last commit message

Last commit date

Latest commit

History

4 Commits

4 Commits

.github

.github

fixtures

fixtures

scripts

scripts

src

src

test-files

test-files

test

test

.gitignore

.gitignore

CONTRIBUTING.md

CONTRIBUTING.md

LICENSE

LICENSE

README.md

README.md

index.html

index.html

package-lock.json

package-lock.json

package.json

package.json

tsconfig.json

tsconfig.json

vite.config.ts

vite.config.ts

Repository files navigation

Verdict is a local, open source tool for evaluating AI features. Load the traces from any AI agent, chatbot, RAG assistant, or LLM feature. Review each conversation and mark it Pass or Fail with a comment. Discover what kinds of failures happen with grounded theory error analysis. Then build and validate an LLM as a judge against your human labels, and export that judge to monitor quality in production.

Everything runs in your browser. No backend, no accounts, and no data leaves your machine unless you choose to call an LLM API. Ship it as a single web page that works offline.

Live demo: https://antoinef23.github.io/verdict/

Why this exists: shipping an AI feature and eyeballing a few outputs (the "vibe check") does not tell you if it is good, how it fails, or whether it is improving. Real evaluation is a loop. You look at your data, label it, find the failure modes, measure them, then automate the measurement with a judge you have proven trustworthy. Verdict is a purpose built tool for running that loop.

Credit and inspiration

This tool is heavily inspired by the AI evaluation methodology of Hamel Husain. The core ideas come straight from his teaching: look at your data, run error analysis with open and axial coding, and only trust an LLM judge once it has been shown to agree with human reviewers (measured with true positive and true negative rates). If you want the theory behind Verdict, watch:

AI Evals Crash Course in 50 minutes (Hamel Husain): https://creatoreconomy.so/p/ai-evaluations-crash-course-in-50-minutes-hamel-husain

Verdict is an independent implementation of these ideas. It is not affiliated with or endorsed by Hamel Husain.

The evaluation loop

Look. Read real traces from your AI feature.

Label. Mark each conversation Pass or Fail and write a short comment.

Code. Tag why each failure happened with short free text codes (open coding).

Cluster. Group those codes into failure categories (axial coding).

Judge. Build one LLM judge per failure category.

Validate. Prove the judge agrees with humans (confusion matrix, TPR, TNR, Cohen's kappa).

Ship. Export the trusted judge and run it on production traffic no human has time to read.

Steps 1 to 4 are human work, and Verdict makes them fast. Steps 5 to 7 let you scale that judgment to conversations no human will read, but only after the judge has proven it matches your reviewers.

What Verdict does

Reads traces from anything. Format agnostic ingest: OpenTelemetry (OTLP) exports, OpenInference and gen_ai and OpenLLMetry spans, Langfuse exports, plain [{role, content}] conversations, NDJSON, and a fallback that renders any other JSON as a gradable timeline.

Renders conversations, not raw JSON. A clean chat timeline with user and assistant messages and tool calls (collapsible inputs and outputs), plus a "show raw" toggle on every step.

Fast human review. A queue with progress and filters, Pass or Fail plus a comment, and keyboard shortcuts built for moving through hundreds of conversations. Autosaves to your browser.

Grounded theory error analysis. Add free text open codes to failures, then have an LLM cluster them into a failure taxonomy with per category counts.

LLM as a judge, validated. One binary judge per failure category, each with a tunable, versioned prompt and a model you choose. Validate against your human labels with a train and test split and a full confusion matrix (TPR, TNR, precision, F1, accuracy, Cohen's kappa). Export the judge with its prompt, model, and metrics to run in production.

Anonymization. Redact PII (emails, phones, IPs, URLs, cards, IBANs, plus a name and company heuristic and your own term list) in the timeline, in exports, and in everything sent to an LLM.

Local and private. Everything runs in the browser. Feedback lives in local storage and exports to CSV or JSON. The only network calls are the optional LLM API requests you trigger.

Quick start

Just use it, no install: open the live demo above, or download the built dist/index.html and open it in any browser. It is one file and works offline.

Run from source:

npm install npm run dev # dev server with hot reload npm run build # produces dist/index.html, one self contained file npm test # unit and DOM tests npm run typecheck

Try it right away with the bundled sample data (see Sample data).

How the LLM judge is evaluated (the important part)

The whole point of a judge is to replace a human reviewer on data no human will read. That is only safe if you can prove the judge labels almost as well as a human. Verdict does this the way any classifier is evaluated: it treats your human labels as ground truth, has the judge label the same conversations, and compares the two.

Ground truth

Every conversation on this platform is labeled by a human. For a given failure category X, a conversation is a positive if the reviewer's codes map to category X, and a negative otherwise. The "fully coded" checkbox lets you mark a failure as exhaustively coded, so its negatives for other categories are trustworthy.

The confusion matrix

For one category, compare each conversation's human label to the judge's label. Every conversation falls into one of four cells:

Judge says YES Judge says NO

Human YES TP FN

Human NO FP TN

TP (true positive): both agree the failure is present.

TN (true negative): both agree it is absent.

FP (false positive): the judge flagged it, the human did not. The judge over flags.

FN (false negative): the judge missed one the human caught. The judge under flags.

The metrics

TPR (recall, sensitivity) = TP / (TP + FN) TNR (specificity) = TN / (TN + FP) Precision = TP / (TP + FP) F1 = 2 * Precision * TPR / (Precision + TPR) Accuracy = (TP + TN) / (TP + FP + FN + TN)

TPR is the share of real failures the judge catches. Low TPR means it misses failures.

TNR is the share of clean conversations the judge correctly leaves alone. Low TNR means it raises false alarms.

Precision is how often the judge is right when it says YES.

F1 is a single number balancing precision and recall.

Accuracy is the overall share correct. It is misleading when a failure is rare, because a judge that always says NO can still score high. Do not rely on accuracy alone.

Cohen's kappa (the headline number)

Two labelers can agree a lot just by chance, especially when one class is rare. Cohen's kappa corrects agreement for chance, so it is the best single "is the judge good enough" number.

po (observed agreement) = Accuracy pe (chance agreement) = ( (TP + FN) * (TP + FP) + (FN + TN) * (FP + TN) ) / N^2 kappa = (po - pe) / (1 - pe)

Rough reading: below 0.4 is weak, 0.4 to 0.6 is moderate, 0.6 to 0.8 is substantial, above 0.8 is near perfect. Aim for high kappa together with high TPR.

Why a train and test split

If you tune the judge prompt while looking at the same conversations you score on, you overfit and the numbers lie. Verdict splits your labeled set into a train part and a held out test part (stratified so both keep the same balance of positives). Tune on train, and trust the test numbers. Verdict reports both and clears a version's metrics whenever you edit its prompt or model, so stale numbers never mislead you.

The decision

When a judge reaches strong agreement on the held out test set (high kappa, high TPR, acceptable FP), it is a trustworthy stand in for a human on that category. Export it and run it on unlabeled production traffic. If it does not reach that bar, keep improving the prompt or model, or keep a human in the loop for that category.

Tutorial for product managers

You do not need to be technical to run a real evaluation. This walkthrough uses the bundled 100 conversation demo, so you can do the whole loop in about 20 minutes, then repeat it on your own feature's traces.

1. Get some traces. An AI feature (agent, chatbot, RAG assistant, classifier) produces

conversations. Your engineers can usually export these as JSON from your observability stack (OpenTelemetry, Langfuse, Arize Phoenix, LangSmith, or a plain log). Verdict reads all of these. No export yet? Use the demo files in test-files/.

1. Look at your data. This is the most important step. Open Verdict, click Load export, and

pick demo-100-conversations.json. Read a few conversations end to end. You will start noticing patterns immediately: the agent makes things up, ignores instructions, mishandles tools.

1. Label Pass or Fail plus a comment. For each conversation decide: did the AI do its job? Hit

Pass or Fail and write a short comment on why. Keep it binary. "Good enough to ship to a user?" beats a 1 to 5 score, which hides disagreement. Shortcuts: j and k to move, p and f to grade, / to jump to the comment.

1. Open code the failures. Set the queue filter to Fail. On each failure, add one or more

short error codes in the feedback bar, for example hallucinated figure, ignored constraint, ignored tool error. Invent the words that fit. Previously used codes autocomplete so wording stays consistent. Tick fully coded once you have captured every problem in that conversation.

1. Axial code into a failure taxonomy. Click Analyze fails. You will see every distinct code

and its frequency. Cluster them into higher level failure categories. With an API key, click Run with API. Without a key, click Copy prompt, paste it into ChatGPT or Claude, and paste the JSON back. Verdict shows the categories with counts. Click filter queue on a category to review just those failures.

1. Build and validate a judge per category. Click Judges. There is one tab per failure

category. Pick the model, review or edit the judge prompt, and click Validate on labeled set. Read the confusion matrix and metrics (there is a "How to read these numbers" guide in the panel). The headline is Cohen's kappa on the test split. High FP means it over flags, so tighten the prompt. High FN means it misses cases, so broaden it. Save a new version, re validate, and watch the numbers move.

1. Ship the judge to monitor production. Once a judge is good enough, click Export judge (or

Export all judges). You get an artifact with the exact prompt, model, and measured metrics. Hand it to engineering to run on the thousands of production conversations no human will read, and track your failure rate per category over time.

Tips distilled from Hamel Husain's approach

Look at data before dashboards. Reading 30 to 100 real traces teaches you more than any metric.

Use binary Pass or Fail. It forces a real decision and makes disagreement visible.

Build one judge per failure mode, not one giant judge. Narrow judges are easier to validate and fix.

A judge is only as trustworthy as its agreement with humans. Report TPR, TNR, and kappa, not vibes.

Watch the rare class trap. If a failure is rare, a high accuracy can hide that the judge never catches it. Look at TPR and kappa.

Re validate whenever you change the prompt or model.

Sample data

Everything below lives in this repo, so anyone can exercise the tool without their own data.

test-files/ has one dense, ready to drag sample per input format (the same conversations in different shapes, so you can confirm they all parse identically):

File Format

01-otlp-genai.json OTLP export with gen_ai attributes

02-openinferen

[truncated for AI cost control]