Show HN: Catch AI code hallucinations without asking a model

Notifications You must be signed in to change notification settings Fork 0 Star 1 BranchesTags Open more actions menu Latest commit History 4 Commits 4 Commits Folders and files NameName Last commit message Last commit date examples examples hedgemony hedgemony tests tests .gitignore .gitignore AGENT.md AGENT.md LICENSE LICENSE MANIFEST.in MANIFEST.in README.md README.md SKILL.md SKILL.md pyproject.toml pyproject.toml Repository files navigation hedgemony finds the things that do not exist in code an AI wrote — packages that were never published, methods that were never written, arguments no function accepts — and the code that contradicts its own stated examples. Every verdict comes from the Python interpreter or a package registry. No language model is asked anything. That is the point: a finding is a fact about the world, not a second opinion from the same kind of system that produced the mistake. $ hedgemony dashboard.py dashboard.py 2 fabrication(s) in 21 lines = 9.5 per 100 lines no stated examples in this file, so its behaviour was not checked at all line 12 ATTR console has no attribute table line 19 ATTR console has no attribute progress rewrite -- that attribute does not exist What the words actually mean "Hallucination" is vague and "lying" is wrong, so hedgemony does not use either as a verdict. Every finding is named precisely, and each name is a claim you can check: word what it means example decidable? fabrication the umbrella: a claim about the world that is false — yes invention the name exists nowhere import ghostlib yes misattribution a real name on the wrong owner json.serialise yes malformation the target is real, the call is impossible math.sqrt(2, 3) yes contradiction the code disagrees with its own stated behaviour a docstring example that fails yes confabulation plausible wrong logic, with nothing stated to check it against — no — not detected Two words this tool deliberately avoids: "Lying" is the wrong word. Lying requires intent to deceive. A model has no intent, so nothing here takes a position on motive. hedgemony reports truth value only: this name does not exist, this example did not hold. Whether anything meant to mislead is not a question the interpreter can answer, and not one this tool pretends to. "Hallucination" is a popular umbrella covering both the decidable and the undecidable. hedgemony measures only the decidable part — the first five rows above. That is why a clean result is reported as no fabricated names, never as correct. Why confabulation is not detected, and what to do about it To say code is wrong, you need something to compare it against. hedgemony has two such standards: the interpreter, for whether a name exists, and an example you wrote, for whether the code does what you said. Confabulation has neither. def average(values): return sum(values) / len(values) - 1 # the -1 is wrong Every name exists. Nothing states what the answer should be. The only description of what this function does is the function, and it agrees with itself perfectly. There is nothing to check it against — not by this tool, and not by any tool. It is not permanently invisible. One line changes it: def average(values): """ >>> average([2, 4]) 3.0 """ return sum(values) / len(values) - 1 line 3 CONTRACT average([2, 4]) was stated to give 3.0 but gave 2.0 The confabulation became a contradiction, and contradictions are caught. This is exactly why hedgemony reports NO_CONTRACT loudly instead of passing such files quietly — it is telling you the one thing that would make the file checkable. Why no better tool fixes this. "Is this what you meant?" is not a property of the code. It exists only in your head until you write it down, and no amount of analysis can read an intention that was never recorded. So the honest goal is not to detect intent — it is to make stating intent cost one line, then execute it without mercy. That is what the contract layer does, and it is why this gap is named here rather than left out. How it works Two passes over each file. The first never runs anything; the second runs only what you wrote down. Pass one — does this name exist? The file is parsed into a syntax tree, and every name it refers to becomes a question put to a live interpreter. Does math have median? Does json export serialise? Does re.sub take a greedy keyword? These are answered with hasattr and inspect.signature — the same machinery Python itself uses — so an answer is a fact about the machine that will run your code, not an inference. Nothing is executed: asking whether a name exists never requires calling it. Pass two — does the code do what it says? If a docstring contains a >>> example, that example is a claim the author wrote down, and running it settles whether the code agrees with it. This is the only part that executes anything, it happens in a bounded separate process, and a file with no stated examples is never run at all — that is decided by parsing, before anything starts. When it cannot decide, it says nothing. An uninstalled package, a variable whose type is ambiguous, a C builtin with no readable signature — all produce silence rather than a guess. That asymmetry is deliberate: a false alarm sends you to rewrite correct code and you have no way to discover the tool was wrong, while a miss still meets every test and review downstream. your file │ ├── parse ──► every name it claims exists │ │ │ ▼ │ ask the interpreter ──► exists / does not / cannot tell │ │ │ │ │ silent FINDING NOT CHECKED │ └── any ">>>" examples? ──no──► NO_CONTRACT (behaviour unknown) │ yes ▼ run them, sandboxed ──► held / did not hold That is the whole design. There is no model in it, no scoring, and no threshold to tune. Install pip install hedgemony Or just clone it and run — there are no dependencies. Python 3.9 or newer, standard library only, nothing to configure. git clone https://github.com/lovettsendit/hedgemony cd hedgemony python3 -m hedgemony yourfile.py Use hedgemony app.py # one file hedgemony src/ # a directory, recursively hedgemony src/ --quiet # only show files with problems hedgemony app.py --report # write an annotated copy beside the file hedgemony app.py --report html # ...as a self-contained page instead hedgemony src/ --json # machine-readable, for a pipeline hedgemony app.py --report both # one of each hedgemony app.py --no-run # never run the checked file hedgemony app.py --online # also ask registries about uninstalled packages hedgemony --board a/ b/ c/ # rank directories against each other Exit codes: 0 nothing found · 1 findings · 2 the tool could not run. Drop it into CI as-is. What it catches Six kinds of invented name, each decided by asking the interpreter directly: PACKAGE import ghostlib no such package was ever published MODPATH from json.fast import load json is real, json.fast is not IMPORT from json import serialise json exists and does not export that ATTR console.table(...) the object has no such attribute KWARG re.sub(..., greedy=True) the function accepts no such keyword ARITY math.sqrt(2, 3) the function cannot take that many arguments They split into exactly two actions, which is the part that saves time: meaning what to do PACKAGE MODPATH IMPORT ATTR the thing does not exist rewrite KWARG ARITY the function is real, the call is wrong fix the call The distinction other tools cannot make A type checker gives the same error for both of these: from humanize import naturalsize # a real package — just not installed here import ghostlib # never existed anywhere Cannot find implementation or library stub for module named "humanize" Cannot find implementation or library stub for module named "ghostlib" Same message, completely different problem. One is pip install. The other means the code can never work and needs rewriting. hedgemony --online tells them apart: line 2 PACKAGE no package ghostlib was ever published humanize is not flagged. It exists. What it catches that has nothing to do with names Names existing is not the same as code being right: def pages_needed(items, per_page): """How many pages are needed to show every item. >>> pages_needed(10, 3) 4 """ return math.floor(items / per_page) math.floor exists. The call is well formed. Every static checker passes this file — and it is wrong. Ten items at three per page needs four pages; this returns three. line 11 CONTRACT pages_needed(10, 3) was stated to give 4 but gave 3 The authority is not this tool's opinion about what the function should do. It is a claim the author wrote into the file, in a standard executable format. hedgemony reports the contradiction between two things already in the file — and does not guess which side is wrong. Making a file checkable costs one line. If a file states no examples, hedgemony says so plainly rather than passing it: no stated examples in this file, so its behaviour was not checked at all What it does not do This matters more than the feature list. A clean result is not a proof of correctness. It means no fabricated name was found. Code that calls the wrong real function is invisible to name checking, by construction. Every report says so in those words. It does not judge style, performance, or design. It does not guess. Anything that cannot be decided — an uninstalled package's internals, a variable of ambiguous type, a C builtin with no introspectable signature — is left unreported. A false alarm sends someone to rewrite correct code with no way to discover the tool was wrong; a miss still meets every test downstream. The costs are not symmetric, so ambiguity always resolves to silence. Reports hedgemony app.py --report # app.py.hedgemony.md hedgemony app.py --report html # app.py.hedgemony.html One report per source file, at one fixed name, overwritten every run — a hundred runs leave one file, not a hundred. The markdown carries the whole file with every line marked and each finding keyed by number. It reads correctly as plain text, needs no renderer, and compresses well, which matters when the reader is an agent paying for every line: + 8 | def pages_needed(items, per_page): + 9 | """How many pages are needed to show every item. + 10 | ! 11 | >>> pages_needed(10, 3) #1 + 12 | 4 + 13 | """ + 14 | return math.floor(items / per_page) The HTML is one self-contained page — black ground, code coloured green where it is clean and red where it is not, nothing loaded from anywhere. Choosing what gets written hedgemony app.py --report # markdown (default) hedgemony app.py --report html # a page hedgemony app.py --report both # one of each Nothing is written unless you ask. Without --report it prints to the terminal and leaves no files behind. Using this with your local model hedgemony never talks to your model. There is no endpoint to configure, no API key, no integration with any runner. It works on the code your model produced, which is already a file on disk. That is deliberate: a checker that asked the model whether the model was wrong would be asking the thing that made the mistake, and its answer would be worth nothing. The interpreter has no such conflict of interest. So the flow is three steps, and the first two are what you already do: 1. your model writes code (any runner, any IDE, any agent — it does not matter) 2. it lands in a file (this already happens) 3. hedgemony thatfile.py ← the only new step # whatever you normally do to get code out of your model, then: hedgemony generated.py That works for any model — local, hosted, one you have no API access to, or a snippet someone sent you. If it produced code you can save, hedgemony can check it. Comparing models Give each model its own folder and rank them: out/ qwen/ ← one model's output llama/ ← another's handwritten/ hedgemony --board out/qwen out/lla [truncated for AI cost control]