待翻译：Model Genome: Fingerprinting Whether an LLM Was Trained from Scratch or Derived

AI 服务暂时不可用，以下为来源正文，待恢复后补全翻译。

Back to Articles a]:hidden"> Model Genome: Fingerprinting Whether an LLM Was Trained From Scratch or Derived Community Article Published August 8, 2026 Upvote 12 Proto_AGI mayafree TL;DR — When a lab announces a "self-developed, from-scratch" foundation model, how can an outsider verify the claim using only public artifacts? We built a reproducible pipeline that fingerprints a model on three axes — architecture (config.json), tokenizer (vocabulary overlap), and weights (embedding CKA) — and combined them into a single at-a-glance genotype. Along the way we hit two instructive traps: row-wise embedding cosine is useless because of rotational invariance, and even CKA cannot cleanly separate continued-pretraining from from-scratch — so config + tokenizer remain the primary evidence. We applied the exact same yardstick to the public foundation models of nine Korean organizations. Try it live: Model Genome Korea. 1. The question Building a large language model on top of an open-weight base (Qwen, Llama, DeepSeek, Mistral) is a legitimate, industry-standard practice. But it is different from training a foundation model from scratch — and vendors do not always make the distinction explicit. When several labs released DeepSeek-rivaling "self-developed" models in late July 2026 (e.g. LG K-EXAONE 2.0, 750B), the debate spilled into Chinese tech communities as well — a Zhihu thread (→ link) crossed 2.7M views. The natural question followed: from scratch, or derived? This is answerable, objectively, from public files. Here is how. 2. Axis 1 — Architecture fingerprint (config.json) Every transformers checkpoint ships a config.json. A handful of fields form a surprisingly discriminative signature: model_type vocab_size hidden_size intermediate_size num_hidden_layers num_attention_heads / num_key_value_heads import requests def arch_fingerprint(repo): c = requests.get(f"https://huggingface.co/{repo}/resolve/main/config.json", headers={"User-Agent": "genome/1.0"}).json() return {k: c.get(k) for k in ("model_type", "vocab_size", "hidden_size", "intermediate_size", "num_hidden_layers", "num_attention_heads", "num_key_value_heads")} The shape tuple (hidden_size, intermediate_size, num_hidden_layers, heads, kv) is effectively a fingerprint of the reference architecture. When a model's tuple matches a foreign open-weight exactly, that is strong evidence the architecture was adopted rather than designed independently. Examples we measured: Model shape (h · i · L · heads · kv) Exact match a 7B commercial model 3584 · 18944 · 28 · 28 · 4 Qwen2.5-7B a 72B commercial model 8192 · 29568 · 80 · 64 · 8 Qwen2.5-72B a 14B VLM 5120 · 17408 · 40 · 40 · 8 Qwen3-14B an 8B model 4096 · 14336 · 32 · 32 · 8 Llama-3.1-8B a MoE model 7168 · 18432 · 61 · (moe 2048) DeepSeek-V3 A single coincidental field means nothing; five simultaneously is a fingerprint. 3. Axis 2 — Tokenizer fingerprint (a paternity test) Architecture alone can mislead. A model can copy a foreign architecture but train a genuinely new tokenizer, or vice-versa. The tokenizer is measured directly from tokenizer.json, comparing the vocabulary sets with a min-overlap ratio: def vocab_set(repo): j = requests.get(f"https://huggingface.co/{repo}/resolve/main/tokenizer.json").json() v = j["model"]["vocab"] # BPE: {token: id} return set(v.keys()) def tok_overlap(a, b): A, B = vocab_set(a), vocab_set(b) return len(A & B) / min(len(A), len(B)) # 1.0 == subset This immediately surfaces things config hides. One model matched Qwen2.5-7B's architecture exactly, yet its tokenizer overlapped Qwen by only ~0.38 — a "foreign brain, own language" case: the architecture was adopted, but a new Korean tokenizer was trained. Conversely, some VLMs reused a base tokenizer verbatim (overlap = 1.000), confirming a straight fine-tune. A practical trap: min(|A|,|B|) in the denominator (not the union) is what makes a reduced vocabulary that is a strict subset of a larger one score ~1.0 — the correct signal for "carved out of the base." 4. Axis 3 — Weights fingerprint (the hard one) The gold-standard question is: were the weights trained from scratch, or continued-pretrained on a foreign base? This is where two instructive traps live. Trap 1 — row-wise cosine is useless The naive idea: load embed_tokens.weight from both models, and for shared tokens, average the row-wise cosine similarity. If they share lineage, embeddings should be similar. They are not — even when they obviously share lineage. We measured near-zero mean cosine for both a known from-scratch model and a known Llama-derivative. The reason is rotational invariance: a Transformer's hidden space has no privileged basis, so two models can encode identical information under an arbitrary orthogonal rotation. Row-wise cosine sees rotation as dissimilarity. It cannot distinguish lineage. Trap 2 — CKA helps, but not enough Linear CKA (Centered Kernel Alignment) is rotation- and isotropic-scale-invariant, so it is the right tool for comparing representations: import torch def linear_cka(X, Y): # X: (n, d1), Y: (n, d2) — SAME token order (shared vocab) X = X - X.mean(0, keepdim=True) Y = Y - Y.mean(0, keepdim=True) num = (X.T @ Y).norm() ** 2 den = (X.T @ X).norm() * (Y.T @ Y).norm() return (num / den).item() A from-scratch model scored near-zero CKA against its candidate base — clean evidence of independent pretraining. But a continued-pretrained derivative scored only modestly higher (≈0.25) — barely above the baseline between two unrelated models of the same family (≈0.21). Large-scale training reshapes embeddings enough that CKA loses discriminative power on the derivative side. Conclusion, stated honestly: the weights axis reliably confirms from-scratch (near-zero), but it is not a strong detector of derivation. For that, config + tokenizer fingerprints remain primary. We report the weights axis as supporting evidence, not as a verdict on its own. 5. Bonus axis — attention diversity as an originality proxy Most models declare a single attention mechanism. A few mix several. The count of distinct mechanisms in config.json is a cheap proxy for architectural originality: KEYS = ("layer_types", "linear_attn_config", "sliding_window", "mamba2_d_state", "hyena_filter_order", "mla_kv_lora_rank", "attention_cls") def attention_diversity(cfg): hits = [k for k in KEYS if k in cfg] # e.g. layer_types = [full×16, sliding×48] -> hybrid (2) return hits In our sweep, most Korean models used a single grouped-query or multi-head-latent attention; a couple used a hybrid (layer_types = [full_attention×16, sliding_attention×48]); and the most diverse combined mamba2, hyena, MLA, linear attention, gated-delta-net, native-sparse-attention and sliding-window in one stack. 6. Combining axes → the genotype We collapse the two primary axes (architecture × weights) into one label: Genotype Architecture Weights 🟢 Native self from-scratch 🔵 Adapted mostly self one axis borrowed 🟡 Mixed partial partial inheritance 🔴 Ported foreign (exact match) inherited The tokenizer overlap and attention diversity are shown alongside, not folded into the verdict, so readers can see the raw evidence. 7. Results Applying the identical pipeline to the public foundation models of nine Korean organizations (large enterprises, telcos, mid-size firms, and startups), the picture is not uniform: some models match a foreign architecture and tokenizer exactly (Ported); others use self-built architectures and weights with no foreign match (Native); many sit in between. The per-model breakdown — with a 3D lineage graph, search, and light/dark mode — is in the Space. 8. Honesty & limitations Not an accusation. Building on open-weight bases is legitimate and widespread. The tool reports lineage, not wrongdoing. Weights axis is supporting, not conclusive (Section 4). Same yardstick for every model, without exception. All inputs are public; corrections are welcome. 9. Reproduce it The three functions above are the whole method. Point them at any two repos on the Hub: print(arch_fingerprint("some/model")) print(tok_overlap("some/model", "Qwen/Qwen3-14B")) # weights: load embed_tokens.weight for a shared-vocab pair, then linear_cka Live demo, full dataset, and 3-language UI: Model Genome Korea. Model names, companies, and licenses are the property of their respective owners. Spaces mentioned in this article 1 Community Upload images, audio, and videos by dragging in the text input, pasting, or clicking here. Tap or paste here to upload images · Sign up or log in to comment Upvote 12 Spaces mentioned in this article 1