Retrieval is a measurement instrument, and nobody reports its coverage

My Agent Answers From 0.6% of Its Corpus and Reports It Like a Full Read Retrieval is not the evidence. It is a measurement instrument, and nobody reports its coverage. My portfolio agent holds 1,003 indexed chunks. When someone asks it a question it retrieves six of them and answers. Six of 1,003 is 0.598%. The answer that comes back carries no trace of that. It reads exactly the same as an answer built from reading all 1,003 — same tone, same citations, same confidence. Nothing in the response, the logs, or the trace says that 997 chunks were never looked at. That is fine when the question is what does he say about X. It is a lie when the question is does he mention X anywhere, and there is nothing in a normal RAG stack that can tell those two apart. The claim retrieval cannot support Take a compliance corpus: 2,431 policy documents, and someone asks whether there is a remote-work reimbursement policy. Retrieval returns eight chunks. None mentions reimbursement. The agent answers: There is no remote-work reimbursement policy in the corpus. That claim requires knowledge of 2,431 documents. It was made from eight. The agent did not lie and it did not hallucinate — it correctly reported what it found, and the shape of the sentence quietly upgraded a statement about eight documents into a statement about the corpus. Absence of evidence in a retrieved fragment is not evidence of absence in the whole. Everyone knows this. No retrieval stack I have used records enough to enforce it, because the number that would enforce it — the denominator — is not carried anywhere near the answer. Where the number should have been This is the same mistake I made in a validator two months ago, one level up. A training run on pure noise returned PASS, with a list of the checks that had cleared it. The checks had not run: every one had hit a divide-by-zero guard and skipped silently, and the tool had no way to say I could not judge this, so the absence of a judgement rendered as a favourable one. Retrieval has the identical hole. An answer built from 0.598% of a corpus and an answer built from all of it are the same object. The coverage is not wrong in the report — it is absent from the report, and an absent value gets read as a good one. Three failures wearing one label The second thing the missing denominator costs you is diagnosis. When a RAG answer is wrong, the post-mortem usually terminates at the model hallucinated. That sentence is hiding at least three separate engineering problems: What actually happenedWhat to fix The relevant chunk was never retrievedThe retriever, the embedding, or the query — not the model It was retrieved, then dropped during context assemblyContext construction — the model never saw it It reached the model, which reasoned past itThe prompt or the model — the only case that is a reasoning failure These need three different people on three different days. They produce one identical symptom. And the distinction is trivially recordable — retrieved, in_context, and whether the answer was right — but almost nobody logs the middle number, so the first two cases are permanently indistinguishable after the fact. What I built An MCP server for notchecked, the coverage-accounting schema I wrote after hitting the silent-skip failure four times in four domains. The library types the gaps a program leaves. The server does it for an agent, which is where the failure moved. The tool that matters is coverage_retrieval. It takes the corpus size, the number of units retrieved, and the number that survived context assembly, and it records what the agent actually had when it answered: coverage_retrieval( target = "portfolio corpus", query = "Rust experience", corpus_size = 1003, retrieved = 6, in_context = 6, ) → scope: 6 of 1003 corpus units (0.598%) exhaustive: false ABSENCE_WARNING: You inspected 6 of 1003 corpus units (0.598%). This supports statements about what you FOUND. It CANNOT support "there is no X in the corpus" — that requires exhaustive coverage, and 997 units were never looked at. retrieved = 0 is not a thin answer, it is a retrieval failure, and it records as one. Everything retrieved and then dropped is a context failure, and it records as that instead. Neither is a gap in the agent's reasoning, and calling them one sends someone to debug a prompt for a day. "But our faithfulness score is 1.0" This is the first objection, and it deserves a straight answer: faithfulness cannot catch this, by construction. Faithfulness asks whether the answer is supported by the retrieved context — did the model invent anything beyond what it was given. The compliance answer above invents nothing. It reports, accurately, that eight retrieved chunks contain no reimbursement policy. It scores a perfect faithfulness and may be false about 2,423 documents. Faithfulness scores answer against context. Coverage scores context against corpus. They are different axes, and a system can be perfect on the first while completely silent on the second. MetricWhat it needsAvailable at answer time? Faithfulnessanswer + contextYes Context precisionanswer + contextYes Context recallground-truth annotationsNo — offline evaluation only Corpus coveragecorpus size + retrieved countYes — and it is not reported Context recall is the metric that would catch it, and it needs labelled ground truth, so it lives in your evaluation harness and not in production. Corpus coverage needs two integers you already have. The control is the shape of the sentence, not a percentage This is the part that took longest to see, and it is why there is no threshold anywhere in the implementation. 0.598% is not a bad number. It is a bad number for one class of sentence.It is entirely adequate for "he mentions Rust" — you need the one chunk you are quoting and nothing else. It cannot support "he never mentions Rust." Same retrieval, same six chunks, opposite verdicts, because the claim changed. A single coverage threshold cannot serve both. Set it low and it licenses the second sentence; set it high and it forbids the first. So the required coverage is decided by what kind of claim is being made: ClaimCoverage required "The policy says X"Any — you need only the units you cite "There is no policy about X"Exhaustive "All policies require X"Exhaustive "The most recent policy is…"Exhaustive — the unread remainder may hold the true maximum "There are three mentions"Exhaustive — a count over a sample is an estimate Every row that needs exhaustive coverage has the same reason: it asserts something about the units that were not read. What this does not do, and will not claim to If your retriever reports corpus_size=100000, retrieved=20, in_context=12, cited=3, this records those numbers and what they can support. It does not know whether the retriever chose the right twenty. It is not a retriever, a vector database, a reranker or a context assembler, and it is not competing with the one you have. It sits above that infrastructure and records what the system observed. If your stack can provide stronger provenance, it consumes that. If it cannot, the limitation stays visible instead of quietly becoming full coverage. That is the entire offer, and it is deliberately smaller than "we validate your RAG." No threshold, at any size 999,999 units of 1,000,000 is 99.9999% coverage and still cannot establish absence. The one document you did not read is the one the question was about, or it is not, and a percentage cannot tell you which. A threshold here would be a lie with a decimal point on it, so there isn't one: exhaustive is true when the count reaches the corpus and false at every other value. That is the whole rule. The bug this found in its own implementation I wrote a suite that replays six investigations of my own that produced wrong claims — a page judged from 3,000 of its 10,828 words, a search that had stripped the HTML so anything named only in an href was invisible, three different counts from three broken filesystem walks. On its first run it failed, and it failed on my code rather than on the cases. exhaustive was a bare assertion: an agent could pass exhaustive: true alongside scope: "3,000 of 10,828 words", and the absence warning was dropped. The tool committed the exact failure it exists to prevent, one layer above the schema it protects. Where the scope carries "N of M", the contradiction is machine-visible and is now refused. That is the third time this idea has caught its own implementation. I have stopped finding it funny and started treating it as the strongest evidence that the shape is real. What this does not do It cannot force honesty. An agent can decline to call the tools, describe its method inaccurately, or report a corpus size it invented. Three limits are recorded as passing tests rather than left out of the README: An exhaustive search of the wrong instrument is still exhaustive. Nothing here knows that a substring search is not a robots-tag check. The target list is self-declared. No protocol can know what the caller failed to think of. The retrieval counts are self-reported. Only internal consistency is enforced — in_context may not exceed retrieved. What it removes is the silence. The gap stops being invisible to whoever reads the answer. That is a smaller claim than "this makes agents honest," and it is the one the evidence supports. I also want to be precise about scope, since that is the entire subject: I have measured this in one live system, my own. I have not established what other RAG deployments report, and I am not going to claim it from a sample of one. pip install notchecked · github.com/Mormolykos/notchecked — MIT, zero runtime dependencies, 112 tests, MCP over stdio written from the JSON-RPC wire format. The eight coverage states were reviewed publicly by Boris Teplitsky, an IBM Certified Expert IT Architect who hit the same shape in infrastructure compliance and gave three corrections that changed the schema. They are frozen. Two later proposals to add states were rejected on his ground: a context truncation is a checker that could not observe, not a ninth kind of gap. If you run RAG in production and you do record retrieval coverage alongside answers, I would genuinely like to know — that would make this a solved problem I had not found the solution to, which is a better outcome than being right.