翻訳待ち：How to Fingerprint AI Models When Prompts Lie

AI サービスが一時的に利用できないため、復旧後に翻訳を補完します。

Back to Research When evaluating third-party API gateways, proxy routers, or anonymous arena models, prompt-based identification is essentially useless. A basic system prompt or lightweight fine-tune can make Claude claim it is GPT-4, make Llama sound like Mistral, or alter refusal behaviors entirely. Persona and tone are malleable. But the underlying serving stack (tokenizers, merge tables, serving harnesses, and error validation boundaries) is rigid. To fake those, a proxy provider would need to intercept streams, translate tokens bidirectionally in real time, and rewrite metadata on the fly, which introduces severe latency and stream desyncs. Here is how our client-side fingerprinting engine identifies models deterministically using infrastructure artifacts instead of conversational cues. 1. Tokenizer Vocabularies and Byte Fallbacks Every lab trains a distinct tokenizer with custom vocabulary sizes, merge tables, and fallback rules. Because the tokenizer determines the exact input matrix shape of the model weights, a model cannot alter how it segments text without a full retrain. We send small, fixed text inputs to the endpoint and inspect the returned usage.prompt_tokens: Latin Pangrams:A fixed 212-character English string locks in baseline BPE merge behavior. CJK Segmentation:Chinese text shows high divergence. Chinese frontier models (Qwen, GLM, DeepSeek) encode CJK with high vocabulary density (low token count), while Western models fragment the same string into 2 to 4 times more tokens. Indentation and Code:Tests how 2-space vs 4-space whitespace blocks and code syntax brackets are compressed. Emoji and Unicode:Tests whether multi-byte unicode sequences resolve to dedicated tokens, grapheme clusters, or raw byte fallbacks. 2. Template Offsets and Hidden Harnesses Inference engines (vLLM, SGLang, TGI, TensorRT-LLM) wrap user prompts in chat templates (like ChatML or Jinja templates) and often inject default system prompts. By sending an empty prompt or a single 1-token payload and comparing the returned usage.prompt_tokens against the raw payload token count, we calculate the static overhead: offset = usage.prompt_tokens - payload_tokens This delta exposes hidden guardrail templates, platform wrappers, and chat formatting artifacts applied upstream before the prompt hits the model. 3. Error Taxonomy and Validation Ceilings Validating normal responses only tells part of the story. Feeding intentionally invalid parameters triggers backend validation written by specific engineering teams: Temperature ceilings: Sending temperature: 2.5 causes some runtimes to fail at >1.0, others at >2.0, with distinct error strings. Context refusals: Requesting max_tokens: 1000000000 forces the backend to reject the request and echo its true physical generation limit (e.g. "max_tokens must be ≤ 131072"). Vendor error codes: Safety filter triggers often return proprietary numeric codes (e.g. Zhipu AI internal safety code 1301). 4. Response Serialization Dialects Minor implementation details in how the server formats JSON reveal the runtime: The exact choices[0].finish_reason string (e.g. stop vs end_turn vs content_filter). How null or optional fields are serialized in the response body. Server-Sent Events (SSE) framing quirks during streaming. Summary of Probe Vectors VectorInputObserved Artifact TokenizerFixed Pangram / CJK blockusage.prompt_tokens Harness1-token minimal payloadToken delta (Δ offset) BoundariesExtreme max_tokens / tempStatus and error string Error CodeTriggered filter payloadVendor code (e.g. 1301) DialectShort completionchoices[0].finish_reason Client-Side Execution Because probes are purely deterministic, we run them directly in the browser. Requests are dispatched straight from the client tab to the target endpoint without passing through any intermediate proxy or logging server. Each probe exports a simple runner contract that returns a deterministic value: export default { name: "tokenizer/cjk-density", description: "Measures CJK token segmentation count", async run(ctx) { const res = await ctx.chat({ messages: [{ role: "user", content: "人工智能模型基准评测体系" }], max_tokens: 1, temperature: 0, }); return { value: res?.usage?.prompt_tokens ?? "ERR_NO_USAGE", }; }, }; Why This Matters As API wrappers and proxy routers become more prevalent, verification cannot rely on trust or conversational vibes. By testing deterministic infrastructure artifacts like tokenizers, template deltas, and boundary conditions, we can accurately verify what model is actually serving your traffic. Explore Ratings LeaderboardCompare Models