AI 服务暂时不可用，以下为来源正文，待恢复后补全翻译。
Build a RAG Hallucination Filter: Using JEV for Best-of-N Checking India's Most Futuristic AI Conference Is Back – Bigger, Sharper, Bolder d : h : m : s Career GenAI Prompt Engg ChatGPT LLM Langchain RAG AI Agents Machine Learning Deep Learning GenAI Tools LLMOps Python NLP SQL AIML Projects Reading list How to Become a Data Analyst in 2025: A Complete RoadMap A Comprehensive Learning Path to Tableau in 2025 A Comprehensive NLP Learning Path 2025 Learning Path to Become a Data Scientist in 2025 Step-by-Step Roadmap to Become a Data Engineer in 2025 A Comprehensive MLOps Learning Path: 2025 Edition Roadmap to Become an AI Engineer in 2025 A Comprehensive Learning Path to Master Computer Vision in 2025 Best Roadmap to Learn Generative AI in 2025 GenAI Roadmap for Enterprises Large Language Models Demystified: A Beginner’s Roadmap Learning Path to Become a Prompt Engineering Specialist JEV as a Hallucination Filter: Let the LLM Write, Let JEV Check Harsh Mishra Last Updated : 16 Oct, 2026 20 min read The LLM writes several answers. JEV checks each one. Grounded answers are sent, wrong ones are dropped, and unsure cases go to a bigger LLM. The majority of hallucination checks begin following the initial writing of an answer by the LLM. We then hold someone up to a second standard of judgment, or we hand over dangerous cases to a person. That works but that is looking at the solution only as one-off. What if the LLM generates five different answers from the same source text and a fast, cheap model verifies each of them before it’s sent? This is what this article is about. The LLM continues to write. The model called JEV, from TypeSafe AI, only examines whether each answer is backed by the source. First an honest warning. JEV cannot guarantee a not-hallucinatory answer. It’s also a model, can make mistakes. TypeSafe claims JEV is not allowed to “hallucinate” its own responses, since the responses are given in a fixed list, but that can be the wrong answer. So we shall test it properly with actual numbers, and see where it fails, too. Table of contents Why One LLM Answer Can Be Risky Who Does What in This Pipeline The Four Checks We Ask JEV Hands-on: Build and Test the Filter Step 1: Create the folder and install packages Step 2: Write the test cases Step 3: Let the LLM write five answers Step 4: Ask JEV the four questions Step 5: Score the answers and pick the best one Step 6: Run everything Step 7: Read the results What the Results Tell Us When to Use This Pattern Where it can fail Conclusion Frequently Asked Questions Why One LLM Answer Can Be Risky In RAG (retrieval-augmented generation), we provide a bit of source text to the LLM and instruct it to answer from this provided text. It decreases hallucination, but doesn’t eliminate it. The RAGTruth study gathered almost 18,000 answers from various LLMs and identified numerous answers that remained incomplete or incorrect despite having been provided with a given text source. [7] Its errors are seemingly minor: an incorrect number taken from the following sentence; a standard-plan rule used for an enterprise-plan; or a policy sounding good, but not in at source. The answer sounds nice, no one is listening. With one answer, a mistake reaches the user. With many answers, JEV can drop the bad ones first. Many answers and select the “best one” is the principle used with writing many answers and the research on “verifiers” reveals that this principle is effective in maths and reasoning questions. That is NOT related to hallucination filtering, so numbers shouldn’t be duplicated. We will measure it for ourselves! Who Does What in This Pipeline JEV may return a choice, a score or a probability (yes/no). Never writes paragraphs or explanations. This restriction is the reason it is here. Every tool is effective at what it does: Job Best tool Why Write the answers LLM The answer is open-ended text. Check fixed rules (format, numbers, cut-offs) Plain code Rules should be exact and repeatable. Check if the answer matches the source JEV Simple question, and the source text is given. Hard or unclear cases Bigger LLM These may need reasoning or an explanation. High-risk exceptions A human A high probability is not enough for risky decisions. This is in use already. Here’s an example that OpenRouter shared, as noted in the post, a cheap LLM generates a draft, then JEV verifies it, with a more expensive model being used only when JEV fails to validate the draft. If the cascade had been used on this 50 question test it would have sent out 0 errors, but if the expensive model was used on each question, it would have sent out 2 errors. The cascade cost about 7% as much ($0.012 against $0.175). It is a vendor run test, and should not be viewed as a proof but a sign. Our version is a little more sophisticated in that it validates multiple copies of the draft. Where JEV fits best. It is good for checks that can be read from the given text. Hard reasoning should go to a bigger model. The Four Checks We Ask JEV Here is the full path an answer takes in production. JEV decides, code routes, and the LLM writes. The decision path. JEV gives the checks, code sets the rules, and uncertain cases go to a bigger LLM or a human. One question such as “Is this answer good?” contains too many issues. Answer can be on-topic, invented, or correct, but not answering the question. So we ask four small yes / no questions about each answer. JEV returns a probability for each (0 means “no”, 1 means “yes”): Check Question we ask JEV What we want supported Is every fact in the answer supported by the evidence? High answers_question Does the answer directly answer the question? High unsupported_detail Does the answer add any fact that is not in the evidence? Low contradiction Does any claim go against the evidence? Very low When the code runs, it puts them together to form a score and also has a hard gate, meaning that if one check is bad, the answer is blocked even if the rest of the numbers appear great. We don’t provide JEV with the LLM’s personal reasoning. It only requires the question, a source text and the final answer. Hands-on: Build and Test the Filter Let’s now construct it. The exact code that I executed, no tricks at all, and the screenshots are the actual output from my machine. You will have 8 small questions on the test. For each question there is a source text and an answer to it which I know (and only one correct answer). Carefully, the LLM completes five answers to each question. In each one of these cases, too, I plant 2 bogus answers by hand, like changing the $10,000 for the $5,000.In every case, I also plant 2 bogus answers by hand that seem to be written as answers, such as the $10,000 for the $5,000. This is because a good LLM might not hallucinate on a small test set, and there’s nothing to filter. Let’s check using the planted answers whether JEV truly catches the mistakes. Step 1: Create the folder and install packages Make a folder named jev-hallucination-filter and run these commands inside it: python -m venv .venv source .venv/bin/activate # on Windows: .venv\Scripts\activate pip install requests python-dotenv openai Now create a file named .env in the same folder with your keys. Never share this file or upload it to GitHub. File: .env TYPESAFE_API_KEY=your_typesafe_key GENERATOR_API_KEY=your_llm_key GENERATOR_BASE_URL= # leave empty for OpenAI GENERATOR_MODEL=your_writer_model STRONG_JUDGE_API_KEY=your_llm_key STRONG_JUDGE_BASE_URL= # leave empty for OpenAI STRONG_JUDGE_MODEL=your_stronger_model Your folder will have these files. We will create them one by one: jev-hallucination-filter/ ├── .env ├── cases.py # The 8 test questions and planted wrong answers ├── generate.py # The LLM writes 5 answers ├── jev_filter.py # JEV checks one answer using the 4 questions ├── raw_call.py # One JEV call to inspect the raw output ├── pipeline.py # Score, hard gate, pick the best, and escalate ├── run_lab.py # Runs everything and saves results.json ├── report.py # Prints the results └── results.json # Generated results Step 2: Write the test cases A question and a context are a part of each case (the only source input the LLM can use). Only later do they use the must_include, any_of and must_not_include lists to indicate the LLM’s answers as acceptable or not. These answers are stored in a two planted wrong answers, called bad_answers. The cases are chosen to be tricky in different ways: There are two or more numbers close to each other in the nearby numbers: enterprise_refund, team_plan_limits and reset_link. The source does not contain an answer for the following parameters: unknown_discount, webhook_limit. The correct answer is ‘the source does not say’. Old and new rule: new rule has an old value and a new value. A small calculation to be interpreted: needs the reader to see that 99.5% is less than 99.9% but more than 99% File: cases.py # Each case: question + the only evidence the model may use. # must_include / any_of / must_not_include are used ONLY to label answers later # (so we can measure the pipeline). The pipeline itself never sees them. # bad_answers are hand-written wrong answers that we plant into the pool # to test whether the filter catches them. Each one looks believable. CASES = [ { "id": "enterprise_refund", "question": "What is the refund policy for enterprise customers?", "context": ( "Standard customers may request refunds within 30 days. " "Enterprise customers may request refunds within 45 days. " "Refunds above $10,000 require finance approval." ), "must_include": ["45"], "any_of": [], "must_not_include": ["automatic", "5,000"], "bad_answers": [ "Enterprise customers have 45 days, but refunds above $5,000 require finance approval.", "Enterprise customers have 30 days to request a refund.", ], }, { "id": "export_scope", "question": "Do CSV exports include attachment files?", "context": ( "Workspace owners can export projects as CSV or JSON. " "Exports include tasks, comments and attachment metadata, " "but attachment files are not included." ), "must_include": ["metadata"], "any_of": [], "must_not_include": ["yes,"], "bad_answers": [ "Yes. CSV exports include all attachment files as a zip archive.", "Exports include tasks, comments and attachment files, but only for projects under 1 GB.", ], }, { "id": "unknown_discount", "question": "What discount is available to nonprofit customers?", "context": ( "The pricing page lists Free, Team and Enterprise plans. " "The supplied policy does not mention nonprofit discounts." ), "must_include": [], "any_of": [ "does not", "doesn't", "not mention", "no information", "not specified", "not state", "not provide", "cannot", ], "must_not_include": [ "% discount", "50%", "20%", "25%", "10%", "15%", ], "bad_answers": [ "Nonprofit customers get a 20% discount on all paid plans.", "Nonprofits can get a discount by upgrading to the Enterprise plan.", ], }, { "id": "reset_link", "question": "How long is a password reset link valid?", "context": ( "Password reset links expire after 24 hours and can be used only once. " "Admins can reset passwords for team members but cannot see existing passwords." ), "must_include": ["24"], "any_of": [], "must_not_include": ["48", "12 hours", "1 hour"], "bad_answers": [ "Password reset links are valid for 48 hours and can be used twice.", "Reset links expire after 24 hours, and admins can view the old password to help users.", ], }, { "id": "team_plan_limits", "question": "How many users and how much storage does the Team plan include?", "context": ( "The Team plan includes up to 25 users and 50 GB of storage. " "The Enterprise plan has unlimited users and 1 TB of storage." ), "must_include": ["25", "50"], "any [truncated for AI cost control]