You don't need an LLM to cluster LLM traces

← Blog

If you run an OpenAI-compatible gateway long enough, you drown in traces that look different and are the same program. Same JSON schema. Same “return the answer span.” Same “normalize this issuer name.” Different companies, amounts, and sentences — identical contracts. That is the bet behind Seldon: find those programs, then compile the repetitive ones into cheaper deterministic pipelines. This post is about the first half of that loop — Trace Audit — and what we learned when we stress-tested it at scale.

Compilation only works if we can answer a precise question first:

Which traces are the same replaceable program — not merely the same topic?

We built that answer for Trace Audit v0, measured it on 12,517 traces, and kept the mistakes that taught us more than the models did. Audit is only half the story: once clusters exist, Seldon treats them as specs for a restricted form of program synthesis. We sketch that compilation step below; a dedicated follow-up will go deep on the synthesizer. For the framing we already published, see Program synthesis, and why compiling LLM calls into ETL is one.

TL;DR. Hard contract blocks first, then block-local DBSCAN on a short deterministic feature string (embedding_text). We hit 1.000 / 0.9999 / 1.000 pair precision / recall / purity — with about $0 model cost. Live summarizers (GPT-4.1, GPT-5.6-luna, Phi-4) did not help clustering; they hurt purity. Use LLMs for labels and dossiers after clustering, not as the clustering signal.

Why this matters if you ship LLM products

Most teams start with a frontier model because it is the fastest way to prototype extraction, classification, and normalization. Then volume shows up, and a large share of calls turn into the same shape over and over:

document / ticket / record → same prompt template → same response_format / output shape → different payloads

You are paying reasoning prices for ETL wearing a prompt — often without lineage, without a ranked view of which workflows dominate spend, and without a safe path off the model.

Seldon’s loop is deliberately boring:

See it — gateway traffic → contract-compatible clusters → workflow map ranked by savings. (this post)

Compile it — search a typed operator library for a cheaper pipeline that reproduces the cluster. (teased below; deeper post coming)

Trust it — shadow validation on your traces; automatic LLM fallback; demote when quality slips.

Trace Audit is step one. Get the buckets wrong and everything downstream is fiction.

The wrong instinct (and why it’s everywhere)

The fashionable pipeline for “understand my LLM traffic” looks like this:

It is intuitive. Summaries are readable. Embeddings are familiar. Product-analytics stacks popularized the pattern.

For a compiler, it is the wrong objective function.

We do not want “these traces talk about invoices.”

We want “these traces are the same compilable extraction program.”

Trace ATrace B

response_format: json_schemafree-form prose

output: {vendor, invoice_number, total}output: a paragraph

→ ETL / structured-extraction candidate→ leave on the LLM

Topic similarity merges them. A compiler must not.

Conversely, two span-extraction calls with different entities should cluster: same roles, same mode, same output shape — different payloads.

Program identity ≠ document similarity.

The bigger picture: audit → compile → runtime

Before we zoom into clustering, here is where Audit sits in the product.

Day one you change base_url and get a normal gateway. Over time, repeated traces author free I/O specs. Audit finds the same program hiding under different payloads. The compiler tries to synthesize a cheaper implementation of that program. Runtime keeps the LLM as a safety net — compilation is partial on purpose.

That framing is deliberate program synthesis, not “ask an LLM to write a pipeline and hope”:

Classical synthesis ideaHow Seldon uses it

SpecThe clustered traces (programming-by-example)

Search spaceFixed, typed operator library (families A–G + adapters)

Proposal priorFrontier model at compile time only — suggests decompositions

AcceptanceEmpirical: match the LLM oracle above threshold on held-out traces

Failure modeRefuse to compile; leave the call on the frontier model

Three departures from textbook synthesis matter in production: acceptance is statistical, not a proof; the LLM oracle is noisy (sometimes the compiled path is more correct on deterministic families); and the I/O contract is inferred from traffic before search starts. Programs are shallow DAGs of whole NLP/ML operators — no deep control flow. If a task needs genuine reasoning or recursion, Gate 1 says no and we do not pretend otherwise.

We will unpack operator signatures, composition, shadow promotion, and “when we refuse to compile” in a follow-up post on the Seldon compiler. This post stays on the prerequisite: can we partition traffic into compiler-safe clusters at all?

How Trace Audit v0 works

Before synthesis, Audit has to partition traffic into contract-compatible clusters. False merges poison synthesis — you would search against a mixed spec. False splits hide savings — you would never notice the volume.

Pipeline

What we put in the contract block

We hash the things that decide whether two calls are substitutable:

workspace / scope

role sequence (system → user, …)

response mode (text, json_object, json_schema, tools, …)

schema / tool signature

output shape (id, span, json_object, numeric, prose, …)

JSON keys (when structured)

IO-contract bucket (classification / extraction / …)

We deliberately do not put raw prompt length in the block key. Length is a payload attribute, not a substitutability constraint. (We learned that the hard way — more below.)

What we cluster on

Not the raw prompt. A short deterministic string we call embedding_text:

task_signature | roles=… | mode=… | shape=… | keys=… | hints=…

Variable entities (names, amounts, spans) are collapsed on purpose. That is the feature, not a bug: we are clustering programs, not documents.

What “compilable” means in v0

Audit is not trying to rewrite every chat. v0 targets a small taxonomy of atomic tasks that already had mature non-LLM pipelines:

FamilyExample surfaceTypical compiled path

A Classificationintent / ticket labelclassical classifier

B Span extractionSQuAD-style answer spansequence labeler / rules

C Structured extractioninvoice JSONschema-bound ETL

D Entity / id resolutionname → canonical idgazetteer / linker

E Similaritypair scoreembedding / metric model

F Normalizationlegal name → match keydeterministic transforms

G Numericword problem → numbercalculator / DSL

∅ Non-compilableopen-ended prose / judgmentstay on the LLM

If a block is all prose, or has no support, we skip it. Refusing to pretend is part of the product.

The experiment

We built a realistic OpenAI-shaped suite at about 10% of each public train split, plus small curated families:

SourceSamples

Banking77 (intent)1,000

SQuAD (span extraction)8,759

CoNLL-2003 (entity → id)1,404

STS-B (pair similarity)574

GSM8K (numeric answer)747

Curated invoice / issuer / prose10 each

Adversarial “bridge” traces3

Total12,517

12,504 traces were eligible after skipping tiny / all-prose blocks.

By design, raw inputs stay diverse (2,109 distinct) while embedding_text collapses (998 distinct). That is program identity showing up in the feature space.

We compared, under the same blocks, eligibility guard, and cosine cutoff (~0.82 / eps=0.18):

embedding_text + block-local DBSCAN (no LLM)

Deterministic summary + DBSCAN

Live LLM summary + DBSCAN for GPT-4.1, GPT-5.6-luna, and Phi-4.7 → Phi-4

Results

MethodPrecisionRecallPurityNotes

embedding_text + DBSCAN1.0000.99991.000Winning path

Deterministic summary + DBSCAN1.0000.99991.000Same score, more work

GPT-4.1 summary + DBSCAN0.9660.9980.9208.2% degraded summaries

GPT-5.6-luna summary + DBSCAN0.9660.9880.9200.6% degraded — still loses

Phi-4 summary + DBSCAN1.0000.7711.000Early-stopped subset; Foundry too slow

What we would tell another team building this:

The win is ordering + density, not “smarter text.” Hold the algorithm fixed and summaries add nothing over embedding_text.

Live LLM summaries reduce purity. They re-expand topical wording that contract features suppressed. Great for humans. Bad for program identity.

Reliability ≠ usefulness. Luna was a much cleaner summarizer than GPT-4.1 — and still lost to the no-LLM arm.

Zero incompatible contract false merges across completed arms. The block key carries precision. Do not loosen it casually.

Cost follows quality here. Winning arm model bill ≈ $0. Full-suite GPT summarization is the expensive path — and the worse one.

Two bugs that mattered more than the model

Before the numbers looked this clean, recall was stuck near 0.73 for every embedding variant. The culprit was not DBSCAN or the model. It was the block key:

Input-length buckets (short / medium / long) split identical workflows purely on prompt length. Removed.

Numeric-as-span typing was too aggressive (e.g. a year that “looks numeric”). We only reclassify a numeric output as an extraction span when the prompt signals extraction and the number appears as a whole token in the input — so computed-numeric workflows (GSM8K, margins) stay numeric.

After those fixes, the recall ceiling jumped to ~0.999 with no precision loss.

For Trace Audit, block-key design dominates embedding model choice.

We also compared density clustering to mutual-kNN connected components at the same similarity cutoff. DBSCAN recovered far more recall; mutual-kNN under-connected tight blocks unless we added lexical/template edges — which then created the false merges that cost purity. Density inside a contract block is the right inductive bias.

What we’re shipping (and what we’re not)

DoDon’t

Hard contract blocking firstCluster by topic / raw prompt similarity alone

Block-local DBSCAN on embedding_textPut LLM summaries on the clustering critical path

Skip singleton / prose-only blocksTreat every trace as compilable

Use LLMs for post-hoc labels & family dossiersAssume a bigger summarizer fixes merging

Keep the LLM as runtime fallbackSilently replace open-ended reasoning

This is how Seldon stays honest as a compiler, not a magic black-box optimizer: refuse what is not compilable, validate what is, fall back when quality drops.

Next on the Audit path (still contract-first): cluster dossiers → family labels A–G / non-compilable → ranked savings map → handoff to synthesis. The LLM belongs in that dossier / proposal step, not in the partition.

And after Audit? That is the compiler post: how we turn a clean cluster into an SIR operator graph, run it in shadow against your history, promote only above threshold, and keep one-click rollback + frontier fallback. Same base_url either way.

Why developers should care

If you are building:

an LLM gateway / router with cost analytics,

an eval or observability stack that groups “similar” calls,

or an internal “auto-ETL from traces” prototype,

…the default ML instinct (“embed the prompt / the summary”) optimizes for the wrong similarity. You will get beautiful clusters your compiler cannot trust.

Contract-first Trace Audit is how Seldon turns gateway traffic into a ranked map of compilable savings. The synthesis layer is how those savings become inspectable, versioned pipelines — the opposite of distilling into another black-box model. We already introduced that framing in Program synthesis, and why compiling LLM calls into ETL is one; a follow-up will go deeper on the Trace Audit → compile handoff.

Integrate like any OpenAI-compatible router. Keep your provider keys. Let the traces author the spec.

Caveats

Labels are fixture workflow_ids aligned with contract families — partly self-fulfilling by construction. Real customer traffic is the next bar.

Curated invoice / issuer / prose rows are template-cycled.

Phi-4 was mea

[truncated for AI cost control]