待翻译：Observational reconstruction, then the inverse problem

AI 服务暂时不可用，以下为来源正文，待恢复后补全翻译。

← Blog We capture traces so we can stop paying frontier prices for the same data workflow, run thousands of times a day, wearing a prompt. Reconstructing those traces is necessary and mostly a known engineering problem. Turning them into a compiled route is not: many latent programs can produce the same prompt/response pair, and a cluster is not a license to serve. Seldon’s bet is simple to state. A large share of production LLM calls are not irreducibly open-ended reasoning. They are repeated behavioral contracts — classify this ticket, extract these fields, validate this schema — that a cheaper plan could serve if you could find them, prove a substitute, and fall back to the provider when you cannot. The developer experience is deliberately boring: change base_url, keep an OpenAI-compatible client, get routing, cost accounting, and traces. Over time the same gateway becomes an optimisation layer. That bet fails in two different ways, and the industry has mostly solved only the first. The first failure is blindness. If you cannot reconstruct what crossed the wire — streamed tool calls, schema re-asks, which attempt was served — you cannot attribute cost per task, you cannot build an eval set, and you cannot tell an incident review what the model actually did. LangSmith, Langfuse, Helicone, Phoenix, Portkey, and OpenAI Traces all implement this physics. Seldon’s router and Import Audit sit in the same family. This post treats that layer as a prerequisite, with enough mechanism that you can see where audits silently lie. The second failure is false authority. Once the exchanges exist, it is tempting to hash a prompt template, embed a summary, or ask a model to label the “workflow,” then compile the cluster. We have written as if that were enough: contract-first clustering asked which traces are the same replaceable program; program synthesis treated the cluster as the specification. Both remain useful as discovery. Neither is identity, and neither is serving authority. A sealed measurement on production Live traffic, plus a sequence of feasibility spikes, made that precise. Observability reconstructs sessions. A compiler needs an identity, a hypothesis, and a qualification edge — and black-box traces do not give you those three for free. This matters beyond Seldon. Any team that wants to cache, route, distill, or compile from traces — an agent gateway, an eval harness, a prompt-to-ETL research stack — eventually has to answer: same contract, or merely similar text? Get that wrong and you merge two users, split one job into twelve “opportunities,” or serve a compiled path to a request it was never qualified for. Fail-open to the provider is the right runtime when you cannot tell. It is not a theory of identity. What we are trying to do The loop we want an engineer to trust: Success is not “we recovered the hidden agent graph.” Success is: this request shape has been seen often enough to matter; this Candidate reproduced the oracle on held-out examples under named gates; this workspace chose to serve it; anything that does not match falls through to the model the application already called. The shipped product already walks that loop. Live Audit groups traces. An Optimization Case freezes one group as a Candidate, evaluates it, and — only after human-gated adjacent steps — binds @signature/{digest} in shadow, canary, or live. Import Audit does the same discovery on Langfuse, LangSmith, or Braintrust JSONL, in an isolated scope that must never write the live routing map. What the loop still compresses, and what this post is about, is the key. Today grouping, Candidate membership, compile topology, and serving all hang off one LLM-derived digest. That is the implementation we run. It is also the load-bearing mistake the rest of the challenges explain. Why the motivation is economic, not cosmetic A chatbot turn is typically one inference. An agentic task is a loop: plan, propose tools, consume results, retry, sometimes hand off. Practitioner bands put simple agents around five to fifteen LLM calls per completed task; research and coding agents run higher (Vortenza’s 2026 agent cost breakdown). Call count understates the bill. Each subsequent call re-sends the static prefix (system prompt, tool definitions) plus accumulating conversation and tool results. If N is model turns, S the static prefix, u new text per turn, and r average tool-result size: total_input_tokens ≈ N·S + u·N(N+1)/2 + r·N(N-1)/2 Naive history resend grows roughly with N². Tian Pan’s five-step climb 888 → 3,400 → 8,900 → 14,200 → 18,900 is a concrete instance (The token economy of multi-turn tool use). The claim you can verify on your own traces is narrower: tokens per completed task rise faster than calls per task. Two consequences follow. First, capture bugs multiply. A streaming stub that blinds one chatbot reply blinds one row. The same bug on an agent blinds fifteen billed inferences and shatters cost-per-task. Reconstruction quality is FinOps infrastructure, not a pretty session UI. Second, the compile target is usually an inner hop, not the whole agent. The outer loop may still need a frontier model for planning and tool choice. The hop that extracts invoice fields on every ticket is the thing you might replace — if you can recognise it across sessions, users, and paraphrases. Session keys keep two users apart. They do not tell you those users are doing the same paid work. That is why this is a general compiler problem, not an observability feature request. Process mining reconstructs cases. We need to identify workload contracts under a replacement policy. Challenge 1 — Reconstruct the observation without lying Engineers overload “replay.” Observational reconstruction asks: given stored evidence, what crossed the instrumented boundary? Execution time-travel asks: can I fork the agent from a checkpoint? LangGraph checkpointers do the latter; nodes re-execute and can diverge (time-travel docs). A payload log is not a checkpointer. If your “replay” never calls a model again, you are reconstructing. That is all this section claims. The industry data model is a session of turn-level traces, each a tree of typed spans. Fig. A — Session → turn traces → typed spans The session identifier is chosen by the application, not derived from the API key or wall-clock proximity. A turn is a tree: flattening it into “prompt / completion” destroys tool causality. Without the session key you have a bag of traces. Langfuse’s rule of thumb is the right one: one chatbot turn becomes one trace; the conversation becomes one session (good-trace FAQ). Every reconstruction is relative to a boundary. Fig. B — Gateway capture vs SDK instrumentation A gateway sees POST /v1/chat/completions. It does not see in-process tools, private RAG, or MCP unless the result later appears as role="tool". An SDK can wrap local functions. “We log all LLM traffic” is not “we reconstructed the agent.” Resent messages arrays are a claim by the client, not proof of what you previously served — clients truncate, summarise, and drop tool results. The failure modes that actually destroy audits are ordinary, and they have to be named as statuses rather than repaired in silence. Streaming. Relaying SSE correctly while persisting only a usage stub is the classic lie: the product looks fine, every stream is an analysis error. The fix is incremental assembly before yield: merge id/model when first seen, append delta.content, concatenate delta.tool_calls by (choice_index, tool_index) — providers split one argument JSON across many chunks — record finish_reason and terminal usage, persist complete | stream_error | client_disconnected | partial | truncated. Never materialise an empty assistant message and call capture complete. Tool-call-only outputs (content: null, finish_reason: tool_calls) are valid evidence; requiring output_text drops the program’s next step. Tools. Join role="tool" rows by tool_call_id. If the model proposed call_abc and no result appears, the row is result_not_observed. A later prose sentence that mentions a balance is not the tool return value. Attempts versus the served response. Schema re-asks and provider fallbacks are extra upstream calls for one client request. Showing only the winner hides the cost. Keep client request and effective attempt request distinct when the gateway inserts a correction turn. Two ledgers. Provider success and capture success diverge. Billing follows the provider. Audit completeness follows capture. Collapsing them produces the report in which a successful, billed call “did not happen” because a blob write timed out. Identity heuristics are not identity. API key, time adjacency, and overlapping history hashes collide under concurrent users and shared service accounts. Missing session headers yield unlinked or inferred, never a silent merge. Seldon’s Live router implements this contract: ordered messages after redaction, assembled streams, attempt records, optional conversation headers, source usage on the full population of successful provider calls even when a subset fails analysis. Import Audit is a second seat — vendor JSONL, coverage receipts for missing I/O, vendor-reported cost left unknown when absent — isolated from Live billing and routing. That is the prerequisite. It is also where most LLMOps products stop. The compiler cannot. Challenge 2 — The observation does not identify the program Let W be the latent task graph that actually ran (or that a Candidate might run), X the input, O the captured prompt/response. The forward map is W + X → O. Grouping and compilation attempt O → W. In general that inverse is not unique. Distinct graphs produce the same JSON: extract fields, then normalize dates, versus one fused typed-generation step validate against schema in-process, leaving no extra span on the wire two topological orders of the same operators a shorter pipeline that is observationally equivalent to a longer one A classifier can be made repeatable (temperature=0, fixed seed, structured output). Repeatability is not identifiability. Seeded JSON constrains shape, not membership. A committed invoice-shaped probe on our signature model alternated C.key_value_field_extraction with the same step plus F.date_currency_unit_normalization (stability 0.667). When the two calls agreed, they agreed. They did not prove that pipeline executed. Parsimony — “shortest ordered subgroup list that explains the output” — is a canonicalization heuristic. It is not a discovery theorem. Switching to an unordered 41-task set plus a frozen display order would remove permutation noise only after you have chosen a policy. It still would not recover latent W. What you can do without identifying W: fingerprint declared and observed wire facts; store versioned interpretations labelled as interpretations; qualify a concrete Candidate against a contract by behavior. Live Audit today does the first two in compressed form and treats the third as “held-out members of this digest.” That compression is Challenge 3. Challenge 3 — One hash is being asked to do three jobs A compiler intake has to answer three questions that are not the same relation. Fig. C — One observation cannot authorize three relations RelationQuestionAlgebraServe? Stable identitySame disjoint accounting unit?EquivalenceNo Semantic similaritySame kind of work, under a named policy?Overlapping, versionedNo ReplaceabilityMay Candidate X replace the provider for contract A inside envelope V?Bipartite, not transitiveOnly as a passed edge Non-transitivity is the part clustering papers skip. Candidate X may qualify for contracts A and B, Candidate Y for B and C, with nothing qualifying for both A and C. A “replaceability cluster” would mark A and C as the same program. They are not. v1 compresses those relations into one digest: workload_signature_digest = sha256("workload-signature [truncated for AI cost control]