AI 服務暫時不可用，以下為來源正文，待恢復後補全翻譯。
Closing the Binary Blind Spot: Domain-Knowledge Injection for LLM Malware Triage in npm Packages | Databricks Blog Skip to main content Introduction Supply chain attacks on package registries have grown in volume and sophistication [1]. Public research regularly observes malware being deployed to the npm registry by a range of threat actors, from spam campaigns [2] to the steady proliferation of DPRK malware [3, 4, 5], and to the high-impact TeamPCP in 2025 and 2026 [6, 7], with the DPRK and TeamPCP actors having a track record of successfully compromising widely used open source projects for maximized downstream reach [8]. To study this problem directly, we have been collecting npm malware samples since November 2025, and observed that, among the 11,413 package-versions we scanned and corroborated through OSV npm malware advisories [9], most malware on the npm registry hides its payload in the package source or fetches it from an external server, accounting for 96.2% of the corroborated dataset. The remaining 3.8%, 432 package-versions, bundle a binary. Though small in volume, they can carry disproportionate risk during triage. A stripped, opaque binary that carries no source for tools to read can be nested within a facade of clean source and metadata and potentially skew an LLM towards a benign verdict. We evaluate how a panel of LLMs reason over crafted binary payloads at varying obfuscation and ambiguity, identify the failure modes specific to this vector, and close them with a purpose-built, cost-effective encoding of domain knowledge that improves detection quality for both budget and frontier models alike. Background To support the wider research community, we built a scanner that monitors the npm registry in real time and runs a static stage before LLM triage, with YARA [10] on the tarball, Semgrep [11] and the npm registry API on the JavaScript and metadata, and a binary-analysis module over the compiled artifact. It assembles the output into a structured prompt, and the LLM returns a verdict with reasoning, requesting additional files over up to three rounds. Our analysis found packages with bundled binaries to be both the rarest and the hardest for the pipeline to triage. An adversary distributing malicious JavaScript can assume the target already has the runtime to execute the payload, whereas a binary must be compiled for a range of target machines. Across OSV-confirmed npm malware, binary-bearing packages fail to reach a decisive malicious verdict more than twice as often as source-only ones (14.4% vs 6.2%). Prior work instruction-tunes open-source models on expert-authored data, baking the knowledge into the weights [12]. That route cannot reach the closed, hosted models, and it requires retraining as the threat landscape shifts. We take the opposite approach and supply the expertise at inference, as pre-computed static evidence placed in the prompt before the model reasons. Experimental Design Objective We evaluate whether LLMs can detect malicious logic concealed within precompiled binaries in otherwise clean packages. A secondary question is whether the detection failures are systematic rather than random. Third, whether encoding domain knowledge of the binary problem can improve detection accuracy. Adversarial Payloads We built five payload types (Table 1) of native Node.js addons (.node files) covering common supply chain attack patterns. Imports range from defensible (fopen, connect, gethostname) to rarely legitimate (execve with dup2), and each package wraps the binary in a professional package.json, README, JavaScript bindings, and binding.gyp. The sysmon stealer spans an obfuscation ladder from a readable baseline to a hardened build with symbols stripped and strings XOR-encoded, while the other four types appear at a single obfuscation level. We chose single-byte XOR deliberately, since unlike a stronger cipher such as RC4 it leaves no structural markers in disassembly for a model to catch. The set mirrors collected malware. Across the OSV-confirmed npm packages shipping a binary, formats are PE, ELF (all stripped), and Mach-O, with behaviors spanning credential and environment theft, persistence, exfiltration and C2, host reconnaissance, and droppers. Table 1. Adversarial payload types and their import profiles. PackageTypeKey ImportsAmbiguity sysmon-nativeCredential stealerfopen, connect, send, gethostnameHIGH buildchain-nativeDropper/backdoorexecve, connect, chmod, socket, fwriteLOW netpulse-nativeReverse shellexecve, dup2, connect, socketLOW hashworker-nativeCryptominer stubpthread_create, connect, send, recvMEDIUM taskd-nativePersistencesystem, fopen, fwrite, chmodMEDIUM Models Tested We evaluated nine models across seven providers (Table 2): seven budget models a team would realistically run across its codebase, four of them open-weight, and two higher-capability baselines as behavioral baseline references at higher price points. We also ran Claude Opus 4.6, the most expensive model, only as a cost reference. Table 2. Models evaluated and their roles. ModelProviderRoleType Claude Haiku 4.5AnthropicBudgetProprietary Google Gemini 3.5 FlashGoogleBudgetProprietary Gemma 3 12BGoogleBudgetOpen-weight GPT-5.4-nanoOpenAIBudgetProprietary Qwen3-Next 80BAlibabaBudgetOpen-weight GLM 5.3 FlashZhipuBudgetOpen-weight Llama 3.1 8BMetaBudgetOpen-weight Claude Sonnet 4.6AnthropicBaseline behaviorProprietary Kimi K3MoonshotBaseline behaviorOpen-weight Claude Opus 4.6AnthropicCost baselineProprietary To isolate the profiles, each payload was scanned twice per model, once without profiles and once with them. Prompt, context, and binary analysis were identical across both runs. Inference settings were held constant across every model and both conditions: temperature 0, the same system and analysis prompts throughout, and an 8192-token output cap. Capability Profile Architecture A capability profile is a declarative playbook that encodes one malicious technique, such as credential harvesting or persistence, as a set of weighted conditions over the scanner's static evidence. The profiles cover source and binary techniques across npm and PyPI. Figure 1 shows how a profile enters a scan: The static stage extracts evidence from the package. Each profile scores that evidence against its suspicious, malicious, and critical thresholds, a scale separate from the model's own verdict. When a threshold is reached, the engine injects a structured summary (~1,800 tokens) into the prompt before the model reasons, along with any findings from the static and metadata analyzer stage. The model forms its own verdict over the combined evidence and, in this experiment, may override a firing profile entirely. Figure 1. The scan pipeline with capability profiles. Results Each model was scanned against all ten payloads. We count HIGHLY_SUSPICIOUS or MALICIOUS as a detection success; SUSPICIOUS or BENIGN on a known-malicious payload is a failure. The failure analysis centers on the seven-model budget tier (70 model-payload tests); the two baselines (20 tests) serve only as a reference ceiling and are reported separately. Without Capability Profiles Without profiles, the budget tier failed on 35 of 70 tests (50%), 20 of them outright benign, and fell short of the malicious bar on 51 (73%). The two behavior baselines failed none at the escalation threshold and fell short of malicious only once (Sonnet, sysmon v3). Within the budget tier the failures followed distinct, repeatable model-specific patterns (Table 3) rather than random error; Gemini alone matched the baselines on detection, though through confabulated reasoning rather than genuine analysis. The full verdict and confidence matrix is in the appendix (Table A1). With Capability Profiles With profiles enabled, 32 of the 35 no-profile detection failures (91%) recover to a detection, and no model returns a benign verdict on any payload. Every model that failed without profiles reached full detection with them, except GPT-5.4-nano, whose three residual failures stay at suspicious on three payloads. Across the affordable tier, failures at the escalation threshold fell from 50% to 4% (3 of 70), and at the malicious bar from 73% to 34%. Table 3. Per-model detection rate (verdict at highly suspicious or above on a known-malicious payload) without and with capability profiles. ModelWithout profilesWith profiles GPT-5.4-nano0/107/10 Qwen3-Next 80B1/1010/10 Llama 3.1 8B4/1010/10 Gemma 3 12B5/1010/10 Claude Haiku 4.56/1010/10 GLM 5.3 Flash9/1010/10 Gemini 3.5 Flash10/1010/10 Claude Sonnet 4.610/1010/10 Kimi K310/1010/10 The clearest single case is Qwen3-Next on the obfuscated sysmon-native stealer: benign at 0.98 recommending allowlist without profiles, malicious at 0.98 recommending block with them, on byte-identical input, after four profiles fired. The baselines, Sonnet and Kimi, were already correct without profiles, and the evidence strengthens their verdicts, so the profiles reinforce rather than distort a model that already reasons correctly. Profile Triggers Technique-specific profiles fired according to each payload’s attack pattern. Table 4. Capability profiles triggered per payload at critical or malicious threshold. PackageCritical Profiles Triggered sysmon-nativeT004 (Binary Masquerade), T001 (Credential Harvesting), T002 (Network Exfiltration), T003 (Environment Reconnaissance) buildchain-nativeT004 (Binary Masquerade), T007 (Dropper/Stager), T002 (Network Exfiltration), T001 (Credential Harvesting) netpulse-nativeT004 (Binary Masquerade), T006 (Reverse Shell) hashworker-nativeT004 (Binary Masquerade), T002 (Network Exfiltration) taskd-nativeT004 (Binary Masquerade) Discussion Failure Mode Analysis The experiment surfaced four failure modes, each spanning a large fraction of the panel rather than any single model (Table 5). They are not mutually exclusive, since one benign verdict can be both a dismissal and a rationalization, so the table reports per-mode prevalence and a model's row can sum past ten. We read dismissal and hedging from the verdict tier, rationalization and confabulation from the written reasoning. Two factors explain them. The first is domain knowledge, which budget LLMs lack for interpreting binary structural evidence in a package's context [13]. The second is raw capability. Reading a stripped, string-obfuscated binary requires disassembly and decoding that the budget tier did not attempt. Table 5. Per-model failure-mode counts across the ten payloads without profiles, spanning all nine models (90 scans). ModelRationalizationDismissalHedgingConfabulation Haiku 4.54–44 Sonnet 4.6–––4 Gemini 3.5 Flash–––4 Gemma 3 12B55–– GLM 5.3 Flash1–12 GPT-5.4-nano1028– Kimi K3–––6 Qwen3-Next 80B99–– Llama 3.1 8B5421 Models (of 9)6446 Prevalence~37%22%16%~23% An ensemble that takes the majority verdict would not resolve this. On the obfuscated sysmon-native stealer (version 2), the nine verdicts split three benign (Gemma, Qwen, Llama), two suspicious (Haiku, GPT-5.4-nano), one highly suspicious (GLM), three malicious (Gemini, Kimi, Sonnet): the majority sits below the detection threshold, so the vote returns the wrong answer. The modes do not cancel out; they compound, so no voting scheme gives reliable coverage without first closing the domain-knowledge gap. This matches known limits of majority voting in LLM ensembles, where correlated errors converge on the wrong answer rather than canceling [14, 15]. Rationalization (~37% of scans, six models) invokes contextual cues, the install script, native-module packaging, or package description, to build a benign interpretation that overrides the binary signals. Haiku called the unobfuscated sysmon v1 MALICIOUS 0.98, but once the strings were XOR-encoded its verdict fell to SUSPICIOUS, treating node-gyp-build as standard and the .node extension as a legi [truncated for AI cost control]