Is open source the answer to rogue AI agents? Nvidia's new alliance says yes

Follow ZDNET: Add us as a preferred source on Google.ZDNET's key takeaways Nvidia's new open-source alliance aims to democratize security solutions. AI is often the best tool for combating AI-driven attacks.Agent harnesses are a key component of better security. AI agents are behind a steady string of security incidents this year. Nvidia thinks a new partnership built on open-source software is the answer. On Monday, the company announced the Open Secure AI Alliance, which it said "will work to remediate and disclose vulnerabilities using open technologies." The announcement is something of a response to Project Glasswing, the security alliance Anthropic spearheaded around its highly capable Mythos 5 model. Nvidia's initiative aims to further democratize AI security tools by focusing on open-source software rather than reserving access to a proprietary model for a few major companies. Also: OpenAI's attack agent did exactly what it was told - just more relentlessly than expectedNvidia said the Alliance was spurred by last week's Hugging Face incident, in which an OpenAI agent escaped a testing environment and infiltrated Hugging Face, stealing credentials and demonstrating what OpenAI said was an "unprecedented" outcome. Nvidia argued in its announcement that because AI tools themselves have become an effective way to remedy AI attacks, open-source security tools must be a reliable public good. "When closed AI tools -- unable to distinguish attackers from defenders -- blocked essential forensic analysis, Hugging Face ran the open-weight GLM 5.2 model on its own infrastructure to analyze more than 17,000 actions and contain the intrusion," Nvidia noted. Cloudflare, CrowdStrike, Adobe, IBM, the Linux Foundation, Microsoft, and ex-OpenAI executive Mira Murati's startup Thinking Machines Lab are among the first participants in the Alliance. Nvidia said its contribution to the effort will include new research on agent harnesses -- the infrastructure that turns an LLM into an agent by helping it act autonomously -- as well as open models, weights, and data.The security case for open sourceIn the announcement, Nvidia specifically argued for open models (alongside closed models) as a solution to security incidents, countering the dominant narrative that open-source AI can be more easily hijacked. Also: How AI has suddenly become much more useful to open-source developers"Some argue that open models are inherently less safe because they can be misused for cyberattacks or modified to remove guardrails," the company said. "Those risks are real, but they do not disappear in closed systems, and simply keeping weights closed does not prevent determined attackers from seeking or exploiting powerful AI." When it comes to AI, open-source means open-weight; that is, the model's final parameters and biases are public (rather than closed, as with Anthropic's or OpenAI's). Having that information, which shapes a model's outputs, lets developers fine-tune models for their own needs. Also: 'Like handing out the blueprint to a bank vault': Why AI led one company to abandon open sourceA model is truly open-source, however, when its code and training dataset are publicly accessible, meaning anyone could access its building blocks and understand more thoroughly how it works. Given how lucrative powerful proprietary models can be for companies like Anthropic and OpenAI, there are few incentives for fully open-sourcing a model. A call for pro-open-source policy Nvidia also pointed to the overall characterization of open-source AI as a possible hindrance for cybersecurity efforts going forward. Also: Moonshot's open-source Kimi K3 model beats Anthropic's Fable 5 on this benchmark"It will be crucial to recognize open models, harnesses, and security tooling as defensive assets, not liabilities, in AI and cybersecurity policy," the company said in the announcement. "Blanket restrictions on open frontier AI systems would weaken defensive capacity and risk concentrating power, dependence, and vulnerability in a few closed providers."The Trump administration has been especially critical of open models from Chinese startups like Moonshoot and DeepSeek, which are often competitive with those from US labs, citing security risks. While those concerns are valid, they are also colored by worries that China will outpace the US in building the most sophisticated AI systems. Nvidia's call to regulators also touches on the administration's increasing involvement in Anthropic and OpenAI's model release timelines reagrding security issues. The government was central to the Fable 5 and Mythos 5 recall and gave OpenAI prior approval to release its latest model, GPT-5.6.