待翻譯：Why It Hasn't Happened Yet: Capable AI and Malicious Intent

AI 服務暫時不可用，以下為來源正文，待恢復後補全翻譯。

Ryan Baker Aug 17, 2026 When I heard about the recent hacking events starting at OpenAI, Anthropic and AISI, I thought it was a wake up call in more ways than one. I’m finding that not everyone agrees. One reaction does not include the sense of urgency to prepare for AI-based attacks. It’s my reaction that there is a need for an all-hands-on-deck response from the rank and file of developers, software companies, and companies dependent on software. That response should improve operational security postures using every tool available. At first this is shocking, but with a little reflection, I see how the full ramifications haven’t sunk in. As a news story, it’s led by one attack, and little damage occurred. While it would be hard to miss the tinge of panic when reading coverage, people may have become dull to this from persistent rounds of security coverage, often from security sales, that have always hyped risks. This is different. The attacks were both demonstrations of carelessness, and demonstrations of the capabilities of powerful AI models. Damage was minimal because these attacks did not combine malicious intent with those capabilities. The attacks occurred with the more benign intent of a model trying to pass a test. And this is the least effective that models will be. As models improve, defenders and attackers will gain access to new capabilities. How labs manage this is important, but so is how we adopt those capabilities and put them to work. The status quo is not stable. The world hasn’t changed too much yet, but it will. I thought I should go into that in more detail. What are the things protecting us today? What are the risks they won’t hold? What can we do to prepare for that? The answer I’ll offer can be summarized as using new techniques to accelerate old techniques. What I won’t cover but is still important The Hugging Face event isn’t just about the urgency to prepare for AI-based attacks. It was also about the type of testing that the AI labs were performing. They removed protection layers and didn’t follow-up with monitoring, which I’ve already written about in Nobody Was Watching. It also demonstrated unaligned behavior, which I’ll suggest What Happened: OpenAI and HuggingFace as a source to learn more about. This type of work needs to receive high care and caution. Monitoring applied to this testing was insufficient. We should not assume when we explore new territory that the exploration is safe. Even if we’re exploring to understand safety, we should be monitoring the exploration. That should be obvious, doubly so because how effective is the exploration if you’re not monitoring it? Certainly less than if you are. I don’t cover those here not because they aren’t important, but because they are, and they deserve their own space. What protections are holding up today? Intent: In the case of the Hugging Face event, the most effective protection was the lack of malicious intent. We can’t expect that to hold, as it never has. But other protections are about preventing this type of intent accessing capabilities. The most unusual aspect of the recent events is that the AI involved was accidentally persistent. That’s a warning sign for sure, but its probability of great harm alone passes through another layer: intent. None of the agents participating in the Hugging Face hack would have what we’d call malicious intent. Training Alignment: Lesser models have lesser capabilities due to less advanced training. Training also applies alignment to the models. Roughly put, this causes models not to “want” to do harm. A bit more specifically, this both causes models to not spontaneously respond in ways that would invoke harmful actions, and to refuse to respond to requests that look to be trying to create harm. While this type of protection is important, it’s also weak. Security Classifiers: When models are deployed behind APIs, it’s standard practice to use a “classifier” to evaluate the inputs and outputs. If they appear to intend to cause harm, they refuse the request. During the recent hacking events, those were disabled intentionally. This was to enable important testing. While doing that without monitoring was reckless, we should remember that these are enabled for external deployments, and thus contribute to keeping malicious intent and capabilities from merging. The ability to monitor and tune security classifiers is greater than training alignment. Training alignment is something you have to get right during training. Abuse Detection: The major AI labs all have documented evidence of attackers trying to use models for attacks. Security classifiers deny a lot of this. They also take other measures to identify attackers and limit their access. There are evasion techniques here, so it’s not as simple as flipping a switch, but the efforts they put in do add complexity and costs for the attackers. Conventional Security: Models will only be able to perform an attack that is below their capability level. If the infrastructure they are attacking is better secured, it raises the complexity level. Raise the complexity level high enough and the attempt will fail. What puts protections at risk? Jailbreaks are inputs that defeat training alignments. This is why training alignments are weak protections against malicious use. These are usually partial and it’s accepted that partial jailbreaks exist. Universal jailbreaks are less common and more concerning. Even with the existence of jailbreaks, alignment training is very important. Jailbreaks tend to make the model a little stupider in the process. That’s all quite complicated, and there’s some decent papers on it, but the analogy would be that trying to confuse the model leaves it confused. A clear-headed model, with a clear-headed goal would be more dangerous than a confused model. Since one of the main goals is to prevent malicious intent and capability from connecting, this helps. Attackers can try to evade security classifiers, by hiding their intent. Keep in mind that when you hide your intent, a model is less likely to be able to autonomously pursue your goal. If you can’t state your goal without being rejected, the effective capabilities are lower. Plus it’s a lot of effort to evade in the first place. Open-weight models allow anyone a copy of the trained model, rather than access via API. Since security classifiers are part of deployments, they aren’t part of open-weight models. If I download an open-model and have hardware under my control, there’s nothing forcing me to install them. A deployment could add them, but we wouldn’t expect an attacker to do so. Attackers thus get a free pass on security classifiers from open-weight models. That leaves the training alignment to overcome. Attackers can also apply additional training to open-weight models that “untrains” their alignment training. Like jailbreaks, known techniques degrade the model somewhat. It’s an open question on how hard it is to restore that initial level of capability. This “dumbing” effect is less significant than that from jailbreaks. Continued training does require infrastructure, it’s like the original training. But it takes many fewer iterations than the original training, so even if it’s running slow, it’s possible to complete without a ridiculous amount of infrastructure. Continued training: If open-models continue to be released, at higher and higher levels of capability, at some point one will be as capable as the one that orchestrated the Hugging Face event. Today, the best ones are 8 months or more behind. Add some extra buffer for the effects of untraining, and it should be clear that unless something changes, an attacker will in the future have access to a model capable of a hack of equivalent complexity to the Hugging Face event. But should those buffers be spent, for an attacker it’s just a matter of injecting the malicious intent, which could now happen at scale, and it would be quite difficult to stop that from scaling up. We should worry about closed models too, as their training is the most advanced. One risk is leaking the weights, making them effectively into open-weight models. A lesser, but still relevant concern is bypassing the security classifiers. The reason this is lesser is that an appropriate reaction to a failure there would be to disable all access to the model creating the risk. Why should we think security is at risk? As a single piece of evidence, the recent events are the best there is, but the evidence that powerful AI models were gaining these capabilities didn’t start there. Conceptually, the idea has been around a long time. And evidence that we are approaching a pivotal moment, was clear to me in March when I wrote Security Can’t Wait. But what was the level of conventional security needed to avoid a compromise? Hugging Face demonstrated a level that was insufficient. We should all be careful about assuming we are above that level. Ideally, you should assume you are worse than you think, and thus as a goal, strive to be several steps further than you need to be. The security of software systems has always been a bit of a mirage. The mirage made them appear invulnerable. If you worked in security, you’d know this wasn’t true, it was a constant battle with many compromises and ample mistakes. What’s more, while there were many lazy or reckless mistakes, many other mistakes were of a complexity that labeling them as mistakes no longer fits. It’s a bit of a pipe dream to eliminate all the lazy and reckless mistakes, but that is almost realistic compared to eliminating all the complex vulnerabilities. Occasionally signals that systems were not invulnerable would filter through to the public, but overall the weakest point in these systems was often the public itself. You’d blame yourself for installing that malware, or having an insecure password, or going to “that” site. That’s not always fair, security professionals know they need to take responsibility for not just what they do, but for helping users avoid harm. Software systems have survived by having multi-layered defenses. While every layer has mistakes that could be exploited, multiple layers raised the difficulty of finding a path through these layers. Finding the mistake in a second layer is harder because the compromise of the first layer is partial. The typical analogy is Swiss cheese. Swiss cheese has many holes, and in a thin slice, you’ll see many pathways through. A thicker block has some of these dead-end. The remaining complete paths also won’t simply be clearly visible. A thick enough block might have no complete paths, but neither you or the attacker know that until fully exploring. The Swiss cheese analogy breaks down a bit here because it’s hard to imagine a piece of cheese as complex as modern software, but the analogy has taken you a little closer to the truth. Why the equilibrium broke When extended far enough, the Swiss cheese analogy resembles a complex multidimensional maze. There is likely more than one entry point and more than one exit. It takes effort and time to explore the maze, and if defenders are doing their job, there are hazards as well. Take too long and the walls in the maze may shift, closing discovered paths. When security succeeded – and let’s be honest, all failures aside, it succeeded well enough for us to rely heavily upon it – it succeeded by having a complex enough maze. By raising the required time investment of the attacker to a level that was no longer attractive, there were no attempts persistent enough to succeed. Sometimes this succeeded on the principle of the bear in the forest; don’t outrun the bear, outrun the other targets. But even then, there’s a question of how many targets the bear wanted. Adding these layers wasn’t cheap though, each requires investment and maintenance. With enough time, we might be able to make them impervious, but [truncated for AI cost control]