翻訳待ち：Muninn: Code localization model in the world fits on your phone

AI サービスが一時的に利用できないため、復旧後に翻訳を補完します。

Introducing Muninn and Quarry Muninn is a state-of-the-art code localization model (English to function-level code chunks), and Quarry is a new benchmark measuring code localization for agentic search. Muninn clocks in at just 346M parameters, small enough to be fast even on a weak GPU, and Muninn-small (47M) is small enough to run on CPU alone. All are Apache-licensed. (And yes, the bar is low, OpenAI's text-embedding-3 is two and a half years old now, but people still use it, and tiny Muninn-small beats it.) Muninn and Muninn-small are built into the Bifrost code analysis tool, where they’re configured to use 512- and 384- dimension vectors, respectively, compressed to 8bit using fastrq. Add Muninn semantic search to your harness of choice with a quick uv tool install brokk-bifrost or npm install -g @brokkai/bifrost. Vector Search for Agents In the Old Days (~2024), everyone wanted to one-shot context retrieval because models were dumb and modern harnesses didn't really exist. So all the code search models from that era are trained + evaluated on mapping (initially) docstrings and (later) full issue text to code fragments. Both of these have the virtue of being easily available, but they come with the minor downside of just not being very good at modeling the kinds of queries that people (and agents) actually need to make. Modern harnesses are designed to allow the model to make tool calls incrementally as part of task execution, modifying its plan based on the results. So we need a search model that focuses less on “here’s the ticket, find me everything related” (which includes lots of really bad issue descriptions as well as things like stacktraces that really don’t need vector search at all) and more on “Find me the code that does X.” So we had DeepSeek (for Muninn) and Luna (for Quarry) first generate task descriptions from commits, then generate natural language queries from the tasks and map them to the most-relevant functions in the original commit’s changed files as well as related files (Bifrost most_relevant_files, based on co-edits and import analysis). We used the same design for both training dataset and for Quarry, from disjoint repos. (We switched models because the Luna price cuts made it a no-brainer, with the happy byproduct that we’re not just measuring “the model memorized DeepSeek’s idiosyncratic query generation.) Muninn is trained on 11 languages: C, C++, C#, Go, Java, JavaScript, PHP, Python, Rust, Scala, and TypeScript. (Since the Muninn training, Bifrost has also added support for Kotlin and Ruby.) Quarry: A new Benchmark A good benchmark for search as used by modern agents should: Be based on real repos with the entire repo’s codebase as search candidates, not just short excerpts. Measure natural language queries, not docstrings, pseudocode, or issue texts. Include tasks from all common languages, not just Python. Use the most relevant metric in the headline. For agentic search, that’s micro recall (how many of the gold functions were retrieved), not MRR (how high the first one ranked), because the former measures what the agent sees; within reason, it is better to include false positives (the agent can ignore noise) than to leave out true positives (the agent never sees them). Provide enough tasks to be able to meaningfully differentiate between “pretty good” and “best in the world”. We measured Muninn on the close-if-you-squint evals, but it was clear that if we wanted an actually useful measurement we needed to build it ourselves. APPS: algorithmic text → standalone code. Competitive-programming problem statements as queries, correct solution programs as documents. Python only. CodeSearchNet: docstring → code. The only multilingual dataset (Python, Java, JavaScript, PHP, Ruby, Go), but very small, just 1k functions each. (One of the reasons it’s so saturated.) CosQA: short English → code. Real web search queries ("python how to sort dict by value") matched to functions. Python only. SWE-Bench-Lite localization: issue → edited-code. Full GitHub issue as the query, retrieving the functions edited by the fixing PR. Python only. LocBench: issue → edited-code on a newer set of PRs set built to dodge SWE-bench contamination and broaden beyond bug fixes. Python only. Quarry construction To balance between eval expense (dominated by repo count, since you can substantially reuse a repo’s embeddings across tasks from that repo) and coverage/diversity, we selected 5 large repos with deep history from each of our 11 languages and generated 3,411 tasks and 6,525 queries across them. These repos were distinct from the repos that Muninn’s training data was generated from. Construction of the Quarry dataset was as follows: Identify “interesting” commits. (Primarily code changes, not merge commits, etc.) Reverse-engineer the commit into task instructions that could be applied to the preimage, and filter out tasks too prescriptive to yield a non-leaking query (“Update Foo::bar to change its behavior as follows…”). Generate up to five queries covering different aspects of the task, based only on the task and minimal repo information. Specifically, the generator does not see the commit diff. For each query, provide the full set of changed source files (excluding tests) supplemented with Bifrost most_relevant_files to a judge model and select the most relevant functions. (The judge does not see either the task text or the commit diff.) Each task is then pinned to a retrieval revision: the preimage by default, the postimage when the commit primarily adds new code. Retain queries that cover the most not-yet-covered gold units from the original commit, until gold is covered or no remaining query adds new coverage. Generated rows look like this: query: "Where is SA1503's braces-required analysis implemented for a using statement with a non-block direct child?" repo: DotNetAnalyzers/StyleCopAnalyzers @ b1c46c34 gold: LayoutRules/SA1503BracesMustNotBeOmitted.cs :: Initialize (lines 78–91) :: CheckChildStatement (lines 127–145) We used a mix of DeepSeek v4 Pro Preview, DeepSeek v4 Flash July Update, and GPT 5.6 Luna for inference; v4 Pro Preview was the least expensive model smart enough to be useful when we started, but by the time we were done both the updated v4 Flash and Luna were smarter and cheaper. We validated the results of each stage with a GPT 5.6 Sol judge, with an additional Kimi K3 cross-check of one tranche. The judge is blind and adversarially checked (~10% negative controls it must reject, ~5% hidden duplicates, failed controls void the audit); the first full audit found 21% invalid and triggered adjustments to the pipeline and a rebuild; the final set passed with 100% of sampled queries mapped to an essential gold unit and 95% of units essential-or-related, cross-tranche ranking ρ=0.99. A note on Bifrost We used Bifrost pervasively in constructing Quarry (and the Muninn training set). With Bifrost providing a common API, this meant that training against 11 languages was no harder than against a single one, with no janky regular expressions to maintain. Bifrost provided: Code-unit extraction, of course, and canonical FQMN resolution for when models would take shortcuts most_relevant_files to expand the candidate set get_summaries to perform a cheap inference prefilter across candidate files (reduce any noise introduced by most_relevant_files) classify_test_files to determine when a file primarily represents test code analyze_diff to extract gold functions from patches, and to identify qualified names for mechanical leak analysis Training Muninn We fine-tuned voyage-4-nano and granite-embedding-small-english-r2—the strongest models in their respective weight classes—for Muninn and Muninn-small. The most similar prior art to Muninn is SweRank (which trained on GitHub issue descriptions). Briefly, SweRank is described as using plain InfoNCE, with in-batch and hard negatives, discarding positives out of the top 20, with a second iteration of negative mining performed by the first-gen fine tune. We started with the same design, but ended up diverging as we learned more about the problem domain. In rough order of impact: Margin-MSE Distillation Having already scored every query-document pair during negative filtering ("Filter the negatives," below), we wired up the teacher’s score gap (the margin) for each positive/negative pair to train Muninn’s score gap to match. This delivered the largest single training-side gain in the program by adding margin-MSE's term to InfoNCE's binary good/bad training signal. (Student and teacher margins are standarized before MSE, so the student matches the teacher's margin structure without inheriting its score calibration.) Multi-positive rows SweRank turns a PR that modifies N functions into N separate single-positive training rows. With in-batch negatives, that means that sibling gold functions from the same query can show up in each other's denominators as negatives when they happen to get scheduled in the same batch. This may be negligible for SweRank, but for Muninn it is not, since we generated many tasks and queries from a relatively small number of repos. We first tried a band-aid with a ConflictAwareBatchSampler, but we realized that by representing the task “correctly” in a single row, where all the gold positives are equally part of the answer, we could solve the problem by construction: every positive competes only against vetted negatives—its own row's and, for a bigger denominator, every other row's in the batch—but never against another query's gold. We capped data generation at 8 positives per query which was enough for over 99% of queries as judged by our pipeline models. Judge-driven relevance SweRank's positives are definitionally "whatever the PR edited," which is defensible when your query is the raw issue text. Our synthetic queries are messier; they will typically overlap the goal patch chunks but also have relevant hits outside of gold, and not all of the gold will be relevant to every query. So we have to rely on an LLM judge to review candidates against each query and assign relevance categories; only direct-implementation and same-feature-support functions become positives, while generic plumbing and lookalike matches are excluded even when the commit touched them. Filter the negatives SweRank “mines” negatives as “functions that rank highly in the baseline model, that were not in the gold patch.” This often includes functions that are in fact relevant to the query, but didn’t happen to be edited in that particular commit. So after the mining, we filter negatives using voyage rerank-2.5-lite first to make sure we don’t accidentally train it to derank relevant-but-not-edited functions. We adapt the filtering threshold by eliminating “negatives” that outrank all the gold positives when scored by the same reranker. (We feed these eliminated negative candidates to the relevance judge as well, where they may be promoted to positives.) We also found that global mining only rarely surfaces same-file negatives, and going from finding the right file to finding the right function in that file was a problem for our student. So we added two random same-file negative functions to the 20 globally mined ones per row. (We found that the value from additional negatives flattened after 20.) Don’t filter the queries SweRank filters out queries whose gold positives are outside the top 20 of the base model as too far out-of-domain. This could be due to two factors: either the student model is wrong (the things it preferred over gold are genuinely irrelevant) or our row construction is wrong (the student found relevant functions in the codebase that we didn't consider at construction time). We found a modest improvement from allowing these queries once every negative that outranked its gold is validated as a true negative by [truncated for AI cost control]