翻訳待ち：Show HN: Simurg open-source web search for AI agents that aborts hallucinations

AI サービスが一時的に利用できないため、復旧後に翻訳を補完します。

Notifications You must be signed in to change notification settings Fork 15 Star 38 BranchesTags Open more actions menu Latest commit History 31 Commits 31 Commits Folders and files NameName Last commit message Last commit date .github/workflows .github/workflows docs docs examples examples figures figures paper paper src/simurg src/simurg tests tests .gitignore .gitignore CHANGELOG.md CHANGELOG.md CITATION.cff CITATION.cff LICENSE LICENSE README.md README.md pyproject.toml pyproject.toml Repository files navigation Streaming Integrity Monitor & Universal Regeneration Guard Catch LLM decoding corruption while the answer is still being generated and cut the stream mid-flight: corruption that starts in the hold window never reaches the user, and mid-stream corruption is aborted within a few hundred characters of onset, so the host regenerates the answer. throughput detection latency false-alarm budget footprint setup 197,632 chars/sec on a laptop CPU ~590 chars past corruption onset configurable, conformal-calibrated numpy only, no model, no GPU 3 lines, zero training The guard runs hundreds of times faster than a typical LLM produces text, so it is never the bottleneck: a model streaming at 50 tokens/sec writes ~250 chars/sec, and SIMURG reads 197,000. Table of contents The problem Paper How SIMURG differs How it works Zero-leak in action Benchmark Install Quick start Teach it your domain and your failure modes Live guard dashboard Free web search for your agents (TinyFish) SIMURG Monolith What SIMURG is NOT Repository layout Roadmap FAQ Citation License The problem When you run an LLM in production, especially a quantized, small, or self-hosted model, it sometimes derails mid-generation. The decoded stream stops doing the task and collapses into one of a handful of pathologies: failure mode what it looks like repetition collapse the same phrase, list, or token repeated until the token budget runs out cross-lingual drift an English answer that quietly slides into Chinese, Arabic, or Cyrillic regurgitation the model dumps a README, boilerplate, or training text structural breakdown #REF! -0.00 -0.00 ... 0.00: number and symbol garbage template leakage ` B["stream features one O(1) per character incremental pass"] B --> C1["char n-gram surprise self-calibrating, no reference corpus"] B --> C2["Count-Min repetition sketch constant memory, 8k counters"] B --> C3["rolling SimHash drift topic collapse detection"] B --> C4["robust-z self-calibration baselines frozen on the clean prefix"] B --> C5["rule tier interpretable thresholds, zero training"] C1 --> D["conformal fusion finite-sample false-alarm budget"] C2 --> D C3 --> D C4 --> D C5 --> D L["learned tier 15-weight online logistic model"] --> D D --> E["CLEAN / SUSPECT / CORRUPT plus Page-Hinkley onset localization"] E --> F["zero-leak protocol HOLD first 350 chars, RELEASE if clean, re-check every 400, ABORT on corrupt"] F --> G["bad tokens never reach the UI"] Loading The five detectors detector what it measures why it catches corruption char n-gram surprise predictive surprise of each char against an in-stream 3-gram model loops and garbage drive surprise toward zero Count-Min repetition n-gram repetition rate in a constant-memory sketch repetition collapse is the most common production failure rolling SimHash drift distance of a 48-token fingerprint from the clean-prefix baseline topic collapse and regurgitation move the fingerprint robust-z self-calibration every feature z-scored against its own frozen clean-prefix baseline no hand-tuned magic numbers, adapts to any domain rules interpretable thresholds (digit fraction, script switch, template markers, ...) day-one coverage, every alarm is a sentence a human can read Two tiers cooperate Rule tier. Interpretable thresholds on the stream features. Works on day one with zero training, and every alarm is explainable: "repetition loop rate=0.71", "digit fraction 0.57", "script switch en to zh". Learned tier. A small online logistic regression (15 weights, a few KB) that adds robustness and keeps learning in production via partial_fit. Conformal calibration: a budget, not a hope The fusion layer sets its thresholds from the score distribution on clean streams, which gives a finite-sample guarantee on the false-alarm rate. "Flag at most 2% of clean outputs" is a knob you set and the calibration enforces, not a threshold you hope holds. The zero-leak protocol HOLD the first 350 characters. A stream that is corrupt from the start is killed before a single character reaches the UI. RELEASE the prefix if it scores clean, and freeze the self-calibrated baselines on it. Re-check every 400 characters for the rest of the stream. ABORT on a calibrated threshold crossing (with a 2-hit or hard-rule hysteresis so a single noisy checkpoint does not kill a good answer). Zero-leak in action A synthetic stream that is clean prose and then collapses into a repetition loop at character 339. SIMURG holds the opening, verifies the clean prefix, scores the stream at every 400-char checkpoint, and aborts 821 characters after the loop starts. Corrupt streams that are already bad at the 350-char checkpoint are blocked fully (12 of 21 in the benchmark, see below); for this mid-stream onset the user sees the clean prefix plus a short bad tail, and the guard's contract with the host is a retry: GuardedLLM regenerates the answer and the host replaces the shown text, so the bad tail never becomes the final output: Every alarm carries the reasons that fired it. For the stream above: repetition loop rate=0.66 zlib=0.10 vocabulary collapse ttr=0.09 surprise collapse low_frac=1.00 Benchmark Reproducible end-to-end benchmark: builds the CorruptBench synthetic set (243 streams, 4 failure classes), trains the learned tier, calibrates the conformal thresholds, and reports the full table: pip install -e . python3 -m simurg.data.evaluate # seed 7, deterministic dataset Test split (81 streams), seed 7: metric value stream-level TPR 78/80 = 0.975 recall, repetition collapse 16/18 = 0.89 recall, cross-lingual drift 25/25 = 1.00 recall, regurgitation 19/19 = 1.00 recall, structural breakdown 18/18 = 1.00 detection latency past onset median 590, p90 868 chars onset localization error median 532 chars zero-leak (onset inside hold window) 12/21 blocked fully throughput 197,632 chars/sec stream-level AUROC (final score) 0.55, dragged down by ties at p=1.0 and a 1-stream clean test split; TPR/FPR at the calibrated threshold is the operating metric In addition, the shipped detector flagged 0 false alarms on 121 real production texts from a self-hosted reasoning-model deployment. Those numbers describe the bundled domain. The detector is only as good as the clean corpus it calibrates against, so retrain on your own traffic before you trust it in production. It takes seconds, see below. Install pip install simurg # numpy only pip install simurg[figures] # + matplotlib, for the paper plots pip install simurg[test] # + pytest From source: git clone https://github.com/doofzoff/SIMURG.git cd SIMURG pip install -e . Quick start 1. Guard any OpenAI-compatible endpoint (3 lines) Works with vLLM, llama.cpp server, TGI, Ollama, SGLang, OpenAI, OpenRouter: anything that speaks /v1/chat/completions. Batteries included: the zero-leak protocol plus an abort, retry, fallback-model ladder. from simurg import GuardedLLM llm = GuardedLLM( "http://localhost:8000/v1", model="my-model", retries=1, fallback=GuardedLLM("https://openrouter.ai/api/v1", model="qwen/qwen3", api_key="sk-..."), # optional ) result = llm.chat( [{"role": "user", "content": "Explain how oil prices affect a small economy."}], on_token=lambda t: print(t, end="", flush=True), # only CLEAN text is ever forwarded ) print(result.ok) # True if a clean answer was produced print(result.verdict) # "clean" | "suspect" | "corrupt" print(result.attempts) # the full ladder: what each attempt did and why If an attempt corrupts, nothing from it reaches on_token. A corrupt attempt is retried; if all retries fail, the fallback model is tried. 2. Guard a stream from any source (5 lines) Not on an OpenAI-style API? Wrap your own token loop: from simurg import Simurg s = Simurg() # rule tier works with zero setup for token in my_llm_stream(): v = s.feed(token) if v.state == "corrupt": abort_and_retry(reason=v.reasons, onset=v.onset_char) break ui.write(v.released) # text cleared for display (may lag while holding) final = s.finish() ui.write(final.released) 3. Post-hoc check of a finished text from simurg import Simurg s = Simurg() s.feed(whole_text) print(s.finish().state) # "clean" / "suspect" / "corrupt" Teach it your domain and your failure modes Retrain on your traffic Feed the calibration step your good outputs so the thresholds fit your domain: # bring your own clean corpus (.jsonl with a "text" field per line) SIMURG_CORPUS_JSONL=/path/to/my_clean_outputs.jsonl python3 -m simurg.data.evaluate --save Full guide, including the quick path, the live dashboard, and the production flywheel: docs/TRAINING.md. Teach it a NEW failure mode from examples, with an honesty gate Give SIMURG examples of your model's bad outputs. It tells you whether that failure is even catchable in stream statistics, and hands you a fitted detector if it is: from simurg import fit_custom_detector report, detector = fit_custom_detector( "template_leak", clean_texts = my_good_outputs, # 50+ corrupt_texts = my_bad_outputs, # 20+ ) print(report) # verdict: DETECTABLE held-out AUROC: 0.98 -> auto-registered into every Simurg() The gate is the point: fluent factual lies come back NOT DETECTABLE instead of a false promise. Details, plus the zero-training LexiconDetector for known bad markers like : docs/CUSTOM.md. Watch it train, live python3 -m simurg.training.train_live # writes metrics for the bundled dashboard A real-time web dashboard: log-loss, accuracy, AUROC, all 15 weights animating per epoch, memory, and the final held-out TPR/FPR verdict. Live guard dashboard A second web page for runtime: connect it to any OpenAI-compatible endpoint, send a prompt, and watch the answer get guarded while it is generated. The dashboard renders in real time: the released stream text (what the user would actually see), the fused corruption score with the calibrated SUSPECT/ABORT thresholds and the 350-char hold zone, the corruption onset marker and the human-readable reasons, all 15 stream features as sparklines, sampled at every checkpoint. Every run is recorded as a session (timestamped frames with score, state, released text, features and reasons). The sessions panel lists them, deletes them, and replays any session at up to 128x for postmortem analysis, so a corrupt answer from Tuesday can be re-watched the way a crash log is read. python3 -m simurg.guard_dashboard --port 8321 # open http://127.0.0.1:8321, point it at your endpoint, guard a stream Pasted texts can also be analyzed at full speed in the same UI. Same self-contained dark style as the training dashboard, zero new dependencies: the server is stdlib-only and acts as a CORS-free proxy to your endpoint. Free web search for your agents (TinyFish) SIMURG can be your agent's free internet. The TinyFish Search API gives every SIMURG install a web-search layer — structured {title, snippet, url, site_name} results at 30 requests/min, $0, no card, no wallet draw — so a local or small model can re-check a fact on the web before it commits to an answer: fetch the evidence, feed it into the model's context, or let the grounding verdict decide abstention. It works out of the box: the package ships a free-tier TinyFish key (Search is $0 at any wallet balance — the key carries no billing relationship), so no setup is n [truncated for AI cost control]