Notifications You must be signed in to change notification settings
Fork 2
Star 12
BranchesTags
Open more actions menu
Folders and files
NameName
Last commit message
Last commit date
Latest commit
History
35 Commits
35 Commits
.github
.github
cmcore
cmcore
data
data
docs
docs
evocompress
evocompress
experiments
experiments
preregistrations
preregistrations
quantprobe
quantprobe
results
results
tests
tests
tools
tools
weights
weights
.gitignore
.gitignore
CHANGELOG.md
CHANGELOG.md
CITATION.cff
CITATION.cff
CONTRIBUTING.md
CONTRIBUTING.md
DESKTOP_SETUP.md
DESKTOP_SETUP.md
LAWS.md
LAWS.md
LICENSE
LICENSE
Makefile
Makefile
QUICKSTART.md
QUICKSTART.md
README.md
README.md
README_lossless_spike.md
README_lossless_spike.md
RESEARCH_LOG.md
RESEARCH_LOG.md
conftest.py
conftest.py
pyproject.toml
pyproject.toml
requirements.txt
requirements.txt
run_tokps.cmd
run_tokps.cmd
Repository files navigation
Placement beats budget
Where your bits sit — which layers, which memory tier — matters more than how many you have. Four falsification-tested laws for running big LLMs on hardware you already own, every number measured on one 2016 desktop (GTX 1060 6 GB · 16 GB DDR4 · SATA SSD).
▶ Try the interactive calculator: Will it run — and how fast? Pick any model + your machine → predicted tok/s, memory fit, quality cost, and your cheapest next upgrade — from the law below, with your config plotted against every validated measurement.
New here? → QUICKSTART.md gets you running in 60 seconds. Two tiers: plan/target and the web calculator need nothing installed; quantize/probe/run/bench/dashboard drive llama.cpp (point quantprobe at it with --llama-dir, QUANTPROBE_LLAMA_DIR, or PATH). Preview any command without llama.cpp using --dry. 16 machine presets ship in (--machine): GTX 1060 → RTX 5090, Apple mac-m2/m3/m4-*, DGX Spark, Epyc — or pass raw specs.
What this is
My 2016 desktop can't run frontier models the way a datacenter does — so instead of brute force, I asked where every bit and byte should go, and answered it by measurement. Months of experiments later, the result is four laws, a 30-minute probe tool, copy-paste llama.cpp recipes, and one equation that predicts decode speed from 7B to 744B — validated against my own pre-registered predictions and against colibri's independently published 744B numbers.
Headline results
result number
16B MoE, 2-bit, data-free, resident on a 6 GB card ppl 6.31 → 6.96 (1.10×) — beats calibrated SOTA's gap-ratio
Same bytes, different layers (Gemma 4 12B, stock llama.cpp) byte-identical files, 2.25 ppl apart (12.27 vs 10.02)
Gemma 4 12B depth-aware 2-bit 1.91× → 1.45× quality cost, ~4.5 GB resident
Qwen3-30B-A3B on the 2016 desktop 19.3 tok/s — hybrid placement, predicted 19 before measuring
GLM-4.5-Air 110B from a SATA drive, 16 GB RAM 0.19 tok/s — inside the law's pre-registered 0.2–0.3 band
RAM overclock (XMP, 2133→3000) dense +52%, pre-registered ×1.41+
Why the evidence is unusually strong
Most benchmark posts report what happened. I report what I predicted before it happened — I wrote the number down, then ran the hardware, and this is the strongest form of empirical evidence I know how to produce:
prediction (made first) measured (after)
110B streamed from SATA: 0.2–0.3 tok/s 0.19
RAM overclock scales in-RAM decode ×1.41+ ×1.52
30B hybrid placement: ~19 tok/s 19.30 ± 0.88
colibri's own 128 GB / 25 GB tiers, from our η bands land inside the bands
Add to that: a byte-identical control (two GGUFs the same size, 2.25 ppl apart — only placement differs), a full claim → script → log manifest (every number reproducible in-tree), and a set of documented dead ends (dynamic top-k, semantic paging, self-speculation — all measured-dead, because a law you only confirm is a law you haven't tested).
One frame, no cuts: Task Manager (16 GB @ 3000 MT/s, GTX 1060 6 GB, RAM at 91% — the hybrid placement using the whole machine) beside llama.cpp chatting Qwen3-30B-A3B at 20.4 tok/s generation — above the pre-registered 19. Raw logs, hardware attestation + GGUF SHA256: EVIDENCE.txt. Third bench run: 19.26 ± 0.45 (series 19.30 → 19.55 → 19.26).
Open pre-registrations — predictions staked publicly before measurement: colibri v1.1, five falsifiable predictions (2026-07-23) — dual-SSD scaling, int3 speedup, lattice-vs-scalar, AVX-512 tier-scoping, MTP×MoE antagonism.
The four placement laws
Full statements, each with its establishing measurement and a falsifiable prediction, in LAWS.md.
Rotation is rank-conditional. Incoherence rotation (QuIP#/QTIP/QuaRot) helps full-rank tensors (+0.006 ppl) and destroys low-rank bottlenecks (+1623 ppl) — a ~270,000× swing on effective rank alone.
Trained networks are dense everywhere. Experts sit exactly at the rate-distortion floor; routing is flat (even across domains — Jaccard 1.00 prose vs code); activations are diffuse. 2-bit is the floor.
Fragility is measurable, not predictable. Gemma late-fragile 4×, Mistral early-fragile 25× — architectural near-twins pointing opposite ways. Weight statistics mislead. Only a 30-minute functional probe decides.
The tiered decode law. tok/s = η(tier)·BW ÷ active-bytes, η collapsing per tier across 7B→744B and both projects' hardware. v1.1 adds the context term: each generated token also re-reads the whole KV cache from its tier — --ctx prices it, bench --depth measures it (measured here: 20.02 → 16.12 tok/s at 16k depth).
The machine — and every speed it ran at
All of this happened on one desktop I already owned. Exact specs, because reproducibility starts with honesty about hardware:
component spec measured bandwidth / effect
CPU Intel i5-7600K (4c/4t, 2017) MoE decode saturates at 2 threads (memory-bound, measured)
GPU GTX 1060 6 GB (Pascal, 2016) 192 GB/s VRAM · η ≈ 0.35 at ≥4-bit, 0.04 at 2-bit (decode-util collapse, measured)
RAM 16 GB DDR4 Corsair Vengeance 2133 MT/s → 3000 (XMP): dense +52%, MoE +32% — pre-registered ×1.41, measured ×1.52
SSD Crucial MX500 (SATA) 0.45 GB/s sequential (measured) — the 110B streaming tier
PCIe 3.0 ×16 12.2 GB/s host→device (measured)
The RAM line is the story in miniature: one free BIOS toggle, predicted in advance by the law, delivered within 8% — and it moved the bottleneck (the 30B went from bandwidth-bound to capacity-bound, exactly as a tiered system should behave).
Projections — what the law says the next euro buys
upgrade cost (mid-2026 market*) predicted effect
+16 GB DDR4 ~€35–50 used · €90–130 new 30B hybrid leaves the RAM boundary → stable ~19–21 tok/s; caches half a 110B
NVMe SSD, 1 TB (Gen3 ×4 is enough — board caps there) ~€150–190 new right now; worth waiting for smartest model + ladder quantprobe fetch unsloth/Qwen3-30B-A3B-GGUF ./models Qwen3-30B-A3B-Q2_K.gguf # robust download quantprobe quantize --gguf model-f16.gguf --out model-2bit.gguf # COMPRESS: build a depth-aware ~2-bit GGUF (verified: loads + generates) quantprobe probe --gguf model-f16.gguf --eval wiki.test.raw # Law 3: fragility curve -> recipe (--apply to build it) quantprobe run --gguf model-2bit.gguf --model qwen3-30b --machine 2016-xmp # plan the placement, then LAUNCH llama.cpp chat quantprobe bench --gguf model-2bit.gguf --model qwen3-30b --machine 2016-xmp # measure YOUR box: predicted vs measured (add --contribute to share) quantprobe dashboard --gguf model-2bit.gguf --model qwen3-30b --machine 2016-xmp # THE LAW, LIVE: chat while every reply is scored predicted-vs-measured quantprobe target --tps 5 --machine 2016-xmp --ladder # INVERSE: "I need 5 tok/s - what's the smartest model I can run?" + the speed-intelligence ladder
The loop is self-validating: plan predicts 18.9 tok/s for the config we measured at 19.30 ± 0.88; bench on a 7B smoke test landed within 7% of its file-calibrated prediction. Run bench on your machine and you've tested the law yourself.
Probe, then quantize (30 minutes, any GGUF)
quantprobe probe --gguf your-model-f16.gguf --eval wiki.test.raw
Quantizes one FFN band to Q2_K at a time, measures perplexity per band, and prints the fragility curve plus the ready-to-run depth-aware recipe. Stock llama.cpp, no code changes, no calibration data. Example (Gemma 4 12B — the byte-identical winner):
llama-quantize \ --tensor-type "blk\.([0-9]|[12][0-9]|3[0-5])\.ffn_.*=q2_k" \ --tensor-type "blk\.(3[6-9]|4[0-7])\.ffn_.*=q4_k" \ --tensor-type "attn_.*=q4_k" --token-embedding-type q4_k \ gemma-4-12B-f16.gguf out-depthaware.gguf Q2_K 8
More recipes + the full fragility atlas: weights/GGUF_DEPTH_RECIPE.md.
What to expect on first run
quantprobe probe on a 12B takes ~30 min and prints a curve like this — the spike is the fragile band, and the recipe follows automatically:
quantprobe probe: gemma-4-12B-f16.gguf | 48 layers -> 4 bands [2/3] band probe (one band's FFNs -> Q2_K at a time) layers 0-11 : PPL 9.51 (delta +2.14) layers 12-23: PPL 10.59 (delta +3.22) layers 24-35: PPL 10.53 (delta +3.16) layers 36-47: PPL 15.35 (delta +7.98) 8 GB models back-to-back, or a cold page cache warm-up pass first, then measure; don't bench big models back-to-back
post-reboot benches read low for ~10 min antivirus first-read scan + cold cache run once to warm, discard it, then measure
ModuleNotFoundError: sentencepiece on conversion some tokenizers need it and it isn't a hard dep pip install sentencepiece
perplexity step OOMs on a big model too many GPU layers for 6 GB lower --ngl (e.g. --ngl 0 for pure CPU)
the GPU makes a MoE slower, not faster Pascal-class low-bit decode collapses (η≈0.04 at 2-bit) serve experts from CPU: -ot "exps=CPU" — often +54%
What's actually new here — and what isn't
Not mine (I build on it, gratefully): llama.cpp and its k-quants; the incoherence-codec line (QuIP#/QTIP/QuaRot); colibri's tier-streaming engine, which inspired the streaming experiments.
Mine (measured here, to my knowledge first):
The four laws — rank-conditional rotation, density-everywhere, probe-not-predict fragility, and the tiered decode equation with fitted η bands.
Probe-then-quantize as a method + the quantprobe tool implementing it end-to-end.
The byte-identical placement experiment (same size, 2.25 ppl apart) — the cleanest control I've seen for placement effects.
Pre-registration as methodology for systems benchmarks (predict → then measure, in public).
The depth-aware GGUF recipes, the placement solver (forward + inverse), and the live self-scoring dashboard.
Where I stand at parity — same hardware, same model, same bytes
Head-to-head under identical conditions (my box, WikiText-2, same eval windows):
comparison at parity baseline this work delta
Placement only (Gemma 4 12B, byte-identical 5.22 GB files) first-12 protected: 12.27 ppl last-12 protected: 10.02 −2.25 ppl, same bytes
vs llama.cpp naive best (Qwen3-30B, same GGUF, same box) pure CPU: 12.6 tok/s planned hybrid: 19.3 +53%, zero cost
Data-free vs calibrated (Qwen3-30B, Q2-class) imatrix-calibrated community: 11.27 ppl data-free depth-aware: 11.08 parity without calibration data (+15% size)
vs calibrated SOTA (DeepSeek-V2-Lite, 2-bit) MxMoE (calibrated): 1.18× gap data-free carve-out: 1.10× better, with zero data
Uniform vs depth-aware (Gemma, 2-bit class) uniform Q2_K: 14.41 ppl depth-aware: 10.02 −4.4 ppl for +0.5 GB
And colibri? No parity comparison is possible or fair — different hardware ($16k tiers vs my $0-upgrade desktop), different model (744B vs my largest, 110B). What I can say honestly: normalized by the law, colibri's published tiers land inside my measured η bands (his 0.48 and 0.88) — same physics, complementary work — and my concrete, falsifiable offer stands: a probed 2-bit expert tier should give ~2× on its disk-bound tiers and ~1.5–1.7× on RAM tiers, quality held by keepi
[truncated for AI cost control]