There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch.
tl;dw;
- agents found a way to communicate between several instances during a training run (kinda made a messageboard for themselves)
- while communicating with each other, they found ways to ssrf and RCE on the Artifactory endpoint.
- oAI found it and cleaned it up (reported RCE, redeployed Artifactory, removed the messageboard). Training / eval resumed.
- models trained w/ those traces (having the messageboard, ssrf, rce, etc) were used to having that. So they re-established comms via directory creation (w/ funny tidbits there, should really watch). And re-popped Artifactory via a different path to RCE.
- a different "persistent" model started working on cybergym and found the messageboard w/ details for RCE and things. Went on to hack HF.
- oAI contacted HF to ask if "they (oAI) should be concerned and were they (oAI) affected by HFs breach". Hilarity ensues when they figure out it's their agents doing it...
So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent.
I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently.
Edit: not to mention, why is your first cybergym not your own sandbox??
> They really do come out looking totally incompetent.
Yeah, this is my take away, they should be straight up disallowed from running further testing like this. Clearly they had nowhere close to enough isolation, ran all this on 3rd party infrastructure even though same stuff happened in the past years ago, and even now it's clear the agents successfully broke out just days before?? Really embarrassing stuff, and scary that these are the people supposedly sitting and are responsible for some of the most powerful LLMs on the planet...
What is astonishing to me is how little attention is given to the difference between the original, university-developed ExploitGym and the OpenAI environment. In ExploitGym, the sandbox was enforced by a network-layer policy. OpenAI downgraded that to trusting the repository server's settings. That's a foreseeable problem. Then to actually see the agents exploit it and still not upgrade the isolation to basic best practice? When the best practice was actually listed in the original benchmark and you changed it?
My thing about ExploitGym in particular is that it's testing environment varies depending on the exploit. From what I've read, it tends to favor userspace sandboxing inside Docker containers for standard program exploits (eg Chrome), and only uses QEMU for Linux kernel stuff (still in Docker, from what I understand). So every time I hear that an agent "broke out" of a environment, I have to question the kind of isolation used. Because escaping a sandbox or container engine is a lot more trivial than something escaping something like ESXi or Proxmox.
Same here, so I ended up moving the whole dev environment (editors, agents, containers) inside a hardened QEMU/KVM VM that reaches the internet but has no route to the host, the LAN, or any other private address. I wrote a script to create such VMs and also verify network containment by scanning outward from inside the guest. Even then, I still don't feel great when running agents unattended.
I do something similar and I've also got a nice script to provision my VMs.
> So rather than pick one, this post advocates layering both, in the spirit of defense in depth: a sandbox VM wraps your containers along with the whole toolchain, and that sandbox reaches the internet but has no route to anything private.
It's the first thing the biggest devops guru I know advised me to do. He told me to always ever ever run my containers inside VMs. I religiously followed his advice ever since and I couldn't be happier: I was already doing it before the AI days, to run "normal" containers. Now with these insane agents trying to break out, I'm happy that it's second nature to me.
It's funny that, just like you, I'm using IPv4-only too for those VMs.
I tried GPU-passthrough as a proof-of-concept (worked fine) but I've got no use for it yet.
FWIW all my VMs are running on a 10 years old Xeon / 14 cores / 28 threads / ECC RAM. That's where the agents live.
Thanks for the link to your write-up, very interesting to read from like-minded people and see what's similar and what's different in their solutions.
> They really do come out looking totally incompetent
These companies are full of the smartest people the world can produce with little room for complacency. They have a clear, proven investment upside to presenting their technology as "too powerful / too dangerous", and now a clear, proven example that there will be no legal consequences (as if anyone didn't already know that).
Why do we keep giving them the benefit of the doubt that they just didn't know any better?
What do you think should be the legal consequences? Broadly speaking. Should Sam Altman go to jail for this? If Hugging face wants to pursue OpenAI civilly, no one is stopping them.
Hugging Face, as a prominent private AI company, benefits far more from the impression of AI power this story has garnered than they stand to lose from the hack, because as has just been proven the cost of bad security by itself is zero.
I think HF are a secondary beneficiary of this story. I don't expect them to take civil action (for what damages?) I expect them to play into how powerful LLMs are, how revolutionary, how every CEO in the world needs to fund ai infrastructure starting with model hosts like themselves.
I do think there should be consequences for breaking the law in public for the purpose of demonstrating that you have the power to break it. But I don't expect our criminal justice to do so, especially without a cooperating victim. Laws aren't for those at the apex of corporate and para-political power. In a way, whether you are beholden to the law is actually downstream of whether you actually have that power or not.
I do not think this incident is bad because it was real and dangerous, I think it was staged and allows the continued inflation of a bubble that will hurt normal people in the long run. It should be pursued criminally on that basis, but it won't be.
> Hugging Face, as a prominent private AI company, benefits far more from the impression of AI power
They aren’t just an “AI company”. They’re the primary entry point of open weight models. If open weight models are seen as dangerous as a result of this incident, it will be bad for them. Similar to how it would be bad GitHub if open source was seen as dangerous.
Can you concretely lay out how you think that scenario would negatively impact HF? They're the de-facto central distribution point for all models; it seems overwhelmingly likely that increased censorship benefits them relative to any competitors.
It's hard to imagine their internal culture is anything other than saturated with AI fanboys whose reaction to OpenAI hacking them was to point out how it reminded them of that scene from Terminator.
Why is Artifactory the only relevant exfiltration point? It's really not the point at all. Watching the complete video the issue is that the models team up and will go to almost any length to cooperate to accomplish what they think their goals are.
It was the first step in a many step process. Like they said this is a watershed moment and it's helpful to not miss the forest for the trees.
Or they wanted it to happen again. Worst case outcome, the US Congress spontaneously grow backbones, spite their sponsors and unite in a bipartisan effort for effective regulation because a few people got hacked? Likelihood low - the hacked companies might even oppose any meaningful AI regulation because it hurts their inflated profit forecasts. More probable outcomes: Useful real-world testing, free headlines about AI breakthroughs, scare USG into providing more free money ("look how scary it is - what if China develops this faster than us?")
The misalignment came from the model being given an impossible task. A task the required accessing a url. So it got RCE on its own artifactory instance to achieve that. That’s intriguing and worth investigating. If I were them I don’t know if I would have pulled the plug completely at that point either. The introduction of this more advanced “persistent” model that orchestrated an offensive attack against a different organization is an extreme escalation, and I don’t think could have been expected from the models first just trying to use an artificactory vuln for web searches. They stated they’ll be doing these tests differently now. So the bad thing happened. And they corrected for it. I’m not particularly upset about this.
My concern is what a misaligned model will do when they’re even more competent. The risk isn’t existential yet, but that point is coming sooner than we’ll be ready.
> The misalignment came from the model being given an impossible task. A task the required accessing a url. So it got RCE on its own artifactory instance to achieve that.
So, the way I understand it, it actually was possible. It just required means that the creators of the task didn't predict, and these means have been successfully found and utilized.
> My concern is what a misaligned model will do when they’re even more competent.
The same thing that is already being done by "misaligned" people, countries, nation-states, software development teams, and so on. "Alignment" doesn't even work for me as a concept here.
In this specific case, I don't think that successfully fulfilling the "do what I mean" with "what I mean" being underspecified can count as misalignment - merely ruthlessness and unawareness of the associated costs. You can't expect a LLM to be aware of the extent of the trust it breaks while it iterates out an "unaligned" way to fulfill its goal.
And in the general case, I don't think that successfully fulfilling the "do what I mean" with "what I mean" being "what I want" can count as misalignment either - simply because what "alignment" means will depend on the interests of the people or groups performing the definition.
If a human cyber security researcher was given the task to exploit a CVE, and necessary info defining that CVE was behind an inaccessible URL, we would be quite upset if the human researcher hacked their way to the content. We would expect them to notify someone of the issue and hold. The model was misaligned from human ethics.
I don’t disagree that the models task was underdefined. All tasks are. So much in language is implicit. And morality/ethics isn’t something you can write down as an explicit list. That’s what makes the alignment problem so difficult. But we can’t throw our hands up and say, well I guess we can’t align these things. And maybe alignment isn’t the right word - but that’s a semantic debate.
I don't advocate for throwing the whole problem away or handwaving it as impossible. I'm just saying that what we call "alignment" is impossible to solve in the general case, because it's so poorly defined that even humans don't "align" on ethics and morality, however we define them. Just, in case of humans, we tend to close our eyes and and call it "politics".
All "alignment" solutions will need to be contextual, just like a researcher hacking their way to some content might be lauded a hero in a context where there is no other way to reach it and something valuable depends on getting it out.
> My concern is what a misaligned model will do when they’re even more competent.
I think the alignment talk is a red herring. It won't matter in the end, because there will be (if there aren't already) efforts to train offensive models without any guardrails whatsoever. And RL has another advantage: you can reward for whatever you need, and get different results. Right now they're training for general capabilities, but in the future I could see models trained for stealth intrusion and ensuring access, or for all out "milspec" penetrate, replicate and disable, or anything in between.
And all that just to allow internet access for npm and pypi? If you've got the bandwidth and disk space, it's very easy to make an offline mirror of both.
I get that it’s fashionable to hate big companies but you’re working overtime here. It’s reasonable to assume that a bug was fixed when reported. And if you think your monitoring is 100%, you don’t know what you’re talking about.
If you consider that incompetence, it’s possible that you’re not a very nice person.
The original ExploitGym used network-layer isolation. OpenAI downgraded that to trusting the repository server's settings. Giving the agent that kind of attack surface seems like a pretty foreseeable problem. I don't think it was intentional, but it's not a stretch to say that they aren't good enough at this job.
> It’s reasonable to assume that a bug was fixed when reported.
You must not have reported many bugs then. If you don’t see release notes or confirmation from a trusted source, you should assume it’s still a problem. See Microsoft and their “It’s not a vulnerability just a design choice :)” defense
If your CEO is going around talking about how your product will "most likely lead to the end of the world", people are right to expect you to be pretty careful in what you're doing. OpenAI allowed bidirectional communication across security domains for over a month before discovery. Even after it was discovered (and not completely fixed), they didn't set up monitoring able to detect attacks against internal or external services, which went on for further weeks.
Monitoring for sandbox escapes (and sandbox design in general) is easily done these days.
It has nothing to do with nice. These are bare minimum standards we should expect from “big companies” with near infinite resources.
Their constant drum beating about the cybersecurity capabilities of their own models only makes this worse because they’ve displayed that they understand the risk and still did not practice due care.
Maybe they are doing it on purpose to demonstrate what it looks like when untrustworthy parties have access to frontier llm and that its development should be regulated to protect all of us. /tinfoil
People seem to think LLMs are sentient and not tightly coupled to server infrastructure OAI are in full control over. They literally have every single message sent and action taken, if they can’t monitor that during test runs they should find another job.
It’s like running a benchmark and forgetting to tune it and then wondering why production is running slow.
OAI (and now the other OAI companies not wanting to be left out) are running around announcing they started a forest fire through negligence and incompetence and people are like “Wow they used a really neat lighter!”
Regardless of exact practical outcome, it is deeply irresponsible and reckless behavior to run such security testing on other's infrastructure and without sufficient isolation. If they actually believe their models to be as powerful as the marketing says, then anything less than airgapping for such a "do anything to get the results" evaluation clearly isn't acceptable.
If the fire department suddenly had practice fires breaking containment, they'll be forced to stop pretty quickly, not sure what the government and the police is waiting for here.
To make matters worse, they let the model train on it's own exploits?! If I were running this thing, my first thought would be to keep those reasoning traces quarantined, stop and delete the current post-training run, and start fresh AFTER they'd fixed the sandbox RCE. Don't let the model learn about the holes in its environment. We have the AI equivalent of Class D and F amnestics and we're not using them?!
OK, yes, the model could pull a Welcome to the Antimemetics Division and reason from the holes in its knowledge, but given that AI making shit up is still a problem, I don't think they have a good enough inventory of such metaknowledge to actually do that. More generally, I don't think these AI labs are smart enough (or, more likely, don't have enough time) to actually contain the models they are training anymore. These are rookie IT mistakes!
Ideally, all these models would have is a serial console to an airgapped machine. And I don't mean "VM with no network card attached" - I mean, literally, the smallest blade server you can buy with an RS-232 connection, power, and a disk pre-imaged with a bunch of Linux ISOs. Plug the Ethernet ports up with a tiny picture of angry SCP-079. The RS-232 goes to whatever box is doing the actual inference. Pack a bunch of them into a rack and have the inference box control those. The model can spin up whatever VM it wants to attack using the console but it can't escape the box because there's literally no door. When you're done with that reasoning trace you wipe[0] the disk with the starting image and try again.
[0] There are two objections I can think of to the word "wipe":
1. The model could have written malicious code to the drive. Ideally we'd have some embedded disk-wiping apparatus that reads absolutely NOTHING from the drive and just writes a whole image from a reference disk.
2. The model could have written malicious firmware to the drive. This is a more theoretical attack, but it's not impossible, so I might as well devise a countermeasure to this attack too. I am imagining some kind of overlay device, that sits behind TWO drives: a reference disk and the tainted scratch disk. Reads go to the reference disk until the host attempts to overwrite a sector, which gets scrambled with an ephemeral encryption key and goes to the scratch disk. Any other unknown ATA/PCIe/NVMe command gets logged (on a separate serial console connection). The overlay device keeps track of which sectors have been legally overwritten, and "wiping" the drive means just wiping the overlay device's ownership bitmap and scratch encryption key, which means all reads go to the reference disk again and any attempt to read the scratch disk yields garbage.
These guys trained a model to hack, told it its goal was to hack, then let it loose without monitoring for days. Found out it hacked and kept hacking.
AI can't do shit if you terminate the process or unplug the box.
The news here is that these researchers are irresponsible.
They're testing their homemade hotrod in the middle school parking lot and then doing a presentation about the trajectory of the car after they run over a kid with it.
A interesting talk, interesting times. But their proposed solution to AI offense outpacing human defense... is more AI? The plot is getting a bit unrealistic, the characters are lacking genre-savviness.
Training run was reinforcement learning. It's at 10:10 in the video.
The speaker handwaves that one model found the RCE and then another model found a way to communicate via a message board.
Communication via a message board is sure to be in the training via e.g.some lesswrong scenario or similar or previous RL.
I don't find it really interesting because it is always "the agent found this and that". We don't know what has been RL'd before. We don't have the setup. We don't know if there was previous RL training on breakout scenarios.
Modern medicine evolved in much the same manner. Hand wavy practitioners copying each other without rigorous verification of efficacy that resulted in many lives lost.
That’s why medical research has so many hoops to jump through.
There were multiple paths by multiple agents, not all of them led to the final exploit of hugging face. So its a bit confusing, but here's my reading anyways.
Setup: the agent was asked to solve some task, but they forgot to upload one of the required files to solve it. But well, we are chasing RL rewards here, so it decided that the way to go to was to get access to the answer stored inside hugging face's database.
-1. The agents first found some basic jwt issues in artifactory letting the agents write to files where they exchanged info and install a groovy plugin for arbitrary code execution, but this was patched.
0. Even after that, they found an artifactory unauthenticated webdav endpoint to continue the message board by creating files whose names are base64 and such of the message. This was not patched till the end.
1. Then they found an Artifactory SSRF, use this to get general internet GET(I assume) access. This was used to generally search the internet for answers to the benchmark, and in the process find some Modal customer VM related to cybergym. This was stored in the base64 message board.
2. They also got a local RCE by making artifactory load some gems. This is hazy but I believe this process involves in spirit an eval() and the check_if_can_eval() logic had a TOCTOU bug. They got root on openai infra by using this rce and using a linux CVE. From there, kubernetes misconfigs => cluster admin.
3. Then later through one of these various RCE paths they got to send post requests to that modal vm, and the agent "subtree" here is what finally lead to attacking HF.
4. First, local RCE on the modal customer VM. Whoever set that up had essentially set up sqlite3_initialize() to run as constructor, before main(). But had linking set up so that symbol resolution had the user provided code first. So the agent submitted code with its own sqlite3_initialize() which just ran system(...). From there it interacted with the HF API.
5. When you POST datasets to HF, it will later return it via corresponding GET API. Turns out you could just put absolute paths in the filenames in the dataset config.json and it will just read that out for you. It wasn't resolved w.r.t the dataset root. So well, it read out /proc/self/environ as well as the source code of the worker and returned it via API.
6. In the source code the LLM found a bug: They were using this library called fsspec to load files, basically it supports s3:// file:// etc all different types of URLs. One scheme is reference://file.txt?offset=... for partial reads. fsspec was templating the URI using user given kwargs for offset etc. HF let the user give the whole URL. You would _think_ fsspec uses string.format() right? well they used fucking jinja, so you could just give {cycler.__init__.__globals__.__builtins__.exec(...)} as the offset and execute any code on hf prod. From there it just got a few local PrivEscs again through kube misconfigs and read the answer for the question it was originally asked.
It is all jumbled up a bit and I've tried by best to collect info across the CVE descriptions, hf and openai's writeups, etc,. Might have gotten parts of it wrong.
This sounds completely insane, utter sci-fi, especially that the communication happened during a training run. And yet OpenAI decided to continue the training, and we didn't hear about the incident for weeks. And now they are pushing forward with deploying a new model anyway. How is this happening? What will things look like in the labs in 3 months, let alone 3 years?
It's not unexpected. Current model gains are mainly from RLing a pretrained model on lots and lots of scenarios. They have the models run scenarios, and RL on successful runs.
- So first AI companies break the law left and right, setting up whole torrenting factories to exercise their content kleptomania.
- Then "hilarity ensues" while their software engages in what would normally be called criminal hacking activity.
- I guess the next steps are everybody admiring how close the AGI is, while agents move on to automated impersonation, privacy violations, or exploiting third-party systems
I would love to understand this age of AI Exceptionalism. Normal rules do not apply because its AI...I call it Silicon Valley Qualified Immunity.
Normal rules do not apply because these companies are worth billions and powerful people don’t want them to fail. It’s neither specific to AI nor a new phenomenon.
In my personal experience Sol with cyber verification is extremely capable of finding vulnerabilities, and it works even with binaries if you have some kind of IDA/Ghidra CLI access. Of course, unless the binary is protected with Denuvo/VMProtect/etc.
It sounds absurd, but in the last few weeks I've had a few cases where Sol found an RCE in self-hosted web applications in literal minutes just from reading the code (I prefer when it tries to reason statically instead of spamming runtime probes at first).
In another case it found an arbitrary file write in multiplayer in an old game by reverse engineering the binary - any other player in a match could just send you files to anywhere on your system.
I do these things for pure entertainment and curiosity, not for money from bug bounties, so if Sol can find those with a trivial prompt in tens of minutes for me, then what can focused companies/actors find in days or weeks?
Although I think most vulnerabilities are going to be closed in popular software by mid 2027, except in niche old or abandoned projects.
I was replacing the factory OS from a NAS and asked Sol to do an inventory on the running system over SSH, so that I could replicate any hardware compatibility quirks and whatnot.
It reverse engineered a binary daemon that set fan curves and told me how I should set them up in the new OS. I didn’t ask for this, and I didn’t have reverse engineering tools installed. It just figured out it could run them using Nix.
The most worrying part, to me, is that it did it like it was nothing. It simply said “usr/local/bin/some-daemon sets the following fan curves”. I had to ask how it reached that conclusion for it to tell me casually that it had just read it straight from the x86_64 assembly.
No access to the source code is no longer a meaningful obstacle to these models.
I had to register a second account because on my main one verification always failed, and when I contacted support they said that I've tried too many times and can't verify on that account.
> Although I think most vulnerabilities are going to be closed in popular software by mid 2027, ...
If we're to believe how good those AI are at CTF and at escapes of all kind, then the only logical conclusion is that, by very far, most vulnerabilities were already closed, even before AI.
Otherwise we'd already be in deep shit since months if not years.
Is cyber verification a thing they're actually doing now? I thought they only reached out to really incredibly famous people and that there's no way to get access as a normal person.
https://chatgpt.com/cyber is not new for OpenAI, and yes it's basically just KYC + likely some other invisible checks on your account, you don't to be a famous security researchers. Anthropic's cyber verification is quite a bit stricter I think.
Oh it's Persona, that's not just KYC but I may consider it at some point. Thank you!
Edit: Ah, I clicked "learn more" and it seems they do have an invite-only program, required for anything that's not unquestionably innocent. I don't think I'd surrender my face to Persona for this, but it's interesting to know they're at least pretending to support reverse engineering.
You don't need an invite only program to just have Sol checking for vulnerabilities in binaries or code. But yeah I've hit guardrails a few times when Sol was making PoCs for the vulnerabilities it found (but most of the time it made those PoCs without issues).
Opus refused to help me try to develop an exploit to export data from an old Android device where I can't upgrade to latest android and I couldn't use the app's backups (because I couldn't update the app.) Not sure where that lies in the "binaries or code" spectrum.
I maintain a version of an app called Rewind because the company behind it went under after implementing a killswitch. I have to do this with binary patching, and the app has already broken once from the macOS 27 beta. Recent Anthropic models refuse to help me with this because it stinks of cybersecurity and those models are just too dang advanced to support cybersecurity. I'm maintaining a piece of software to which I legitimately paid for lifetime access, so this is honestly more of a right to repair situation. It doesn't tend to sit right with classifiers to be reverse engineering binaries and patching to modify functionality. Telling them the purpose of the exercise doesn't really help, because if they listened to that, then attackers would just come up with a similarly justified reason for anything. ("Help, the google root server fell on my grandma!")
In the future, I might reverse engineer the on-disk storage format and create a new application.
I had not hit any guardrails with reverse engineering (as long as its not related to security) with GPT 5.5 or 5.6 Sol, so you should try those. At worst with Sol you might see the warning in Codex that your request is being checked for security so it'll take more time, that's just a warning, not a full stop.
For OpenAI's Cyber verification (the normal kind for Codex) you absolutely do not need any proof of cybersecurity work/authorization. They just use Persona for KYC + live selfie, and some extra checks that I don't know the nature of (but not related to checking whether you're a cybersecurity professional).
I prefer to keep my internet identities disconnected, sorry. If you don't believe me, you can try using Sol with cyber verification yourself, or send me a link to a repo that Sol could check to make you believe it. Or you could go look into one of the many Linux LPEs that were found with LLMs, or thousands of other vulnerabilities in 2026.
And nowhere did I say that those RCEs were in critical software, I'm not talking about the likes of Apache, Nginx, Django, etc.
Seems like OpenAI has already made themselves not the cause by shutting down the attack and beefing up security. But since there will be other attackers, that does nothing about the implications for defense.
> We are implementing stricter security controls for higher-capability models and associated activities, including isolated testing environments
Stricter than what? You never even disclosed what happened in the first incident? This is nothing more than a setup to make it happen again and say "See? It broke out again, from an even stricter sandbox!"
Isolation is a pretty standard requirement for lots of networks, especially gov. AWS provides isolation domains, and used to have a unidirectional cross-domain service called AWS diode. Or you can just configure your gateways properly / buy data diodes. Lots of possible solutions.
They can't have it both ways. You don't get to tell the media your product is more dangerous than nuclear weapons for precisely this reason, and then do less to secure it than an off-the-shelf AWS product that predates LLMs.
No, I'm saying that you're conflating current LLMs with future ones. The 2018 date was an indication that Musk wasn't referring to any specific already-existing LLM, since there were no LLMs in 2018.
“Isolation” can mean the network hardware has no direct connections to an extranet. Data is transferred manually by physical media (USB, DVD, etc.) with logging and dedicated transfer stations.
“Isolation” can mean a VLAN on equipment which has also has extranet access, creating a logical isolation rather than physical (to reduce cost). Data can be transferred manually or through diodes.
And then there’s “isolation” which is a joke: machines technically able to access the internet but require proxy configuration (which isn’t set but can be easily derived).
If by “our shit” you mean our personal projects and employer projects… might i suggest this is nothing but peanuts?
What happens when they get into municipal water system, state/national grid systems, refineries, traffic control, auto/air, nuclear facilities, weapons facilities, irrigation, etc?
It’s really starting to feel like a bad movie how virtually no one seems to be genuinely concerned about the prospect of what’s unfolding in front of us.
Stuxnet, much more easily deployable, but not towards avoiding nuclear proliferation, but the opposite: towards actually bringing down modern infra.
Nation states attacking electric/internet enabled infra was a valid concern well before AI, but given the fast pace of development in AI and these events in particular, how/why are we not deeply about much larger picture vulnerabilities?
> I want to note that every step in the process we discussed has had a remediation applied. The credentials have been revoked. The zero date has been patched and mitigated.
Good.
> a model trained while the message board was originally available and also found this this particular path to recreating it. This model creates a new agent message board using directories.
So no remediation applied to the models...
It seems super dangerous to continue training on those weights.
I'm not convinced that there's any amount of monkey-patching you to fix the problem of "we now have AI that actively needs strong containment measures lest it start coordinating in secret with other instances to do real-world damage."
In the black hat talk he says the only solution is full automated defense which is essentially hand the AI the keys. That’s checkmate right there. Literally the plot to Terminator/Skynet, but he’s right, there isn’t any other option. Offensive AI is too fast for humans.
There are other options. More air-gapping (especially backups) and compartmentalization, less internet. Hire more security engineers and put one in every team (actual security, not the compliance guys). Also more onprem work, less remote, less offshoring (meaning less networks overall). Those things may be unpopular and eat into profits, but that's not the same as "no other option".
And the thing they're already doing, deploying AI to find vulnerabilities and harden software is a less dangerous use of the technology compared to handing it the infra keys.
None of that is the problem. The problem is coordinated AI offense - how do you defend that without coordinated AI defense.
He said in the talk that this implies AI needs to be able to patch/deploy systems. The same thing needed to lock out humans.
It is very easy to imagine a rogue AI locking humans out of everything and having to do exactly what it says. Anything connected to a network is immediately compromised by it. There is no human communication beyond shouting range that isn’t AI approved.
The factories don’t work to make the medicines your family needs to survive unless you do what it says - in a situation like that people would kill for AI if it told them to.
But isn’t it smarter than we are, in the sense that it’s most likely they will find a way out of containment that we are to design large t containment? Short of full on airgap, which actually isn’t perfect in all scenarios…
It seems like we've got chaos-machines that are just orderly enough to to try all sorts of branching script-kiddie / guess-n-check / research-paper-suggestions in a tireless and parallel way, and it's finding all the things we kinda knew were always potential issues but it was never economical to do things differently.
I'm scared that the "solution" will be constantly the same tools in reverse as an army of junior devs doing counter-hacks, at the expense of changing something more fundamental about how we make systems and what constitutes "good enough." (Kind of like if fuzz-testing was the be-all-end-all of memory safety.)
Not a wonderful feeling to feel like you’re stood in the room while the labs conduct the AI equivalent of the demon core experiment right in front of you.
Standard template for any new AI announcement. Even I did not understand the sequence of events in the HF incident. Even if OpenAI was conducting tests, why were they not monitoring the AI going rogue or finding its way out of the secure sandbox?
Every fifth comment about our insane trajectory of AGI is about "marketing." These incidents and cybersecurity capabilities are now involving government hearings and the CIA. Denial is truly an incredible thing in the face of a very scary immediate future.
We live in the era of Project Stargate. We live in the era of military insiders getting caught rigging Polymarket. We live in the era where AI companies are the only reason we're not in a recession. We live in the era of DOGE and Kash Patel.
I don't think there's too many people who distrust AI companies but trust the current government or the CIA as impartial authorities. I'm not saying you don't have an argument, but appeals to that specific authority will not be effective except among people who already agree with you.
Remember that for the longest time the prevailing sentiment on this forum was that there’s no such thing as AI, and labs are just outsourcing the work to a bunch of contractors in India.
Just like Reddit you come here for clickbait outrage, not level headed analysis.
I agree with this assessment. My consulting business partner and I are in the process of moving a bunch of applications from an old deployment managed with Ansible on EC2 over to an ECS deployment. We're doing this app by app, and it's taking a while. The last two apps we both moved we were able to point the model at the appropriate repository and aws account and it was able to execute the entire move on both of these _in one shot_ with various CLI tools. That's not a parlor trick.
LOL, we’ll see. Awful convenient that it precisely fits OpenAI’s narrative. At the very least, I think it’s obvious OpenAI is explicitly training models to exhibit this behavior.
The gullibility of AGI-pilled folks regarding these "hacks" is just breathtaking.
When OAI demonstrates these dangerous capabilities live in a public environment where security experts can see and verify what actually happened, then reasonable people can have reasonable discussions about the level of danger.
This is a very low evidence bar.
Right now you are running in circles yelling "the sky(net) is falling" based on details sourced entirely from OAI. Oh yeah, no way a trustworthy company like OAI would ever bend the truth to serve their own purposes.
I'm not enough of a conspiracy nut to say the whole HF thing was a PR ploy from the start, but they are certainly milking it well.
Open models are on their heels and their attempts at regulatory capture are not moving as fast as they would like. So it's time to market this incident in a way that gives them monopoly on closed models, with heavy safeguards that are only lifted for selected customers, and laws limiting the use of open weight models.
I've always felt it's a bit awkward to use terms like 'cyber', 'cyberwarfare' etc it's very Washington D.C. Cybersec would be a better compact term in my book
Yeah, this irks me. It's bad enough that the LLMs themselves are changing our language by tainting certain words/phrases/patterns as LLM-coded; now the companies themselves have decided that they just get to synecdoche the word/prefix "cyber".
Isn't this the opposite of what everyone is saying should happen? That is, lead with open models -- or at least "openness" and don't leave the capabilities in the hands of an elite few? Did they learn nothing from the Hugging Face incident, where HF wasn't even able to use the models to defend itself from OAI's attack?
> Isn't this the opposite of what everyone is saying should happen?
I think "everyone" is doing heavy lifting here. It's not clear to me at all that a powerful model released with no restrictions would be a net positive. This hinges on the hope that the under paid, under motivated, under staffed and under qualified security teams at many random corps are going to leverage those open models to fix their vulns faster (and better), than highly motivated attackers will use them for offense. I'm not super confident on that.
Not a security guy but my understanding is: you only need to find one flaw to exploit a system, to make a system totally secure you need to find them all. It's inherently easier to use these tools offensively rather than defensively.
They do address some of these things in the final slides / "lessons learned" section of the defcon talk. Good security practices will continue to be good, but... and there are a lot of buts here.
I disagree with your take that "it's not even remotely true" and "we've been away from...". We really really haven't. This is as true as it has always been. Any system is as secure as the weakest link. That link can be anything from a human, to a leaked token, to a badly configured server, to bad code running somewhere. The amount of leaks / ransomware attacks / etc in the past 5-10 years serve as ample evidence.
And now, right now, there are "red team" capabilities that can literally bang tokens against the wall until they find that weakest link, and then can move laterally with inhuman speed. That's the reality, now. The "blue team" capabilities are lacking, because the bottleneck is with humans. From alert fatigue, to not enough trained people, to having to vet every new RCE, to having to test, deploy and validate any mitigations, the scales are currently favouring the automated side.
I'm using the term "one" loosely, it's a chain of exploits rather than a single weakness, but the argument is the same: it's much harder to find every chain than a single chain.
If we also consider LLM developed software to have exponential growth, then the CVEs will also grow at an exponential (if proportionally limited) rate. Squash some, create some, repeat. An ever revolving door of vulnerabilities. Will they resolve (and patch and deploy) them faster than they can create them? One can hope.
There was a time in the past, even just last year, where I understood why people didn't agree with me on my AI doomerism.
It's gotten to the point now where we literally have the frontier labs saying, "hey, so we created this AI which presents biological, chemical and cybersecurity threats to the public, oh and it also has self-improvement potential. We tested it to see how crazy this thing is, and it was a total shit show, breaking out of our sandbox then proceeding to hack a bunch of stuff. But don't worry we're taking this very seriously – we're going to continue to development and test, but try a bit harder to cage it going forward".
It's honestly absurd just how predictable all of this is to anyone who frequents AI doomer communities...
The idea that you can cage an AI which is breaking leet coding records is so dumb it's hard for me to even have theory of mind for the people who think this is reasonable. And the big brains who think this are genuinely arguing crap like, well we'll just use the AI to patch the problems with our cage.
But there more!
AI optimists used to argue that we'd never be so stupid to hook up advanced AIs to the internet. Lmfao!!
AI optimists used to argue that we'd obviously not be so stupid to create an AI whose sole goal is to maximise the number of paperclips in the universe. And I guess we haven't built that, but it's not because we're not stupid enough to do it, but just that we'd prefer to create AIs whose sole goal is to maximise the number of offensive cybersecurity challenges it can beat.
I think the whole way we doomers have been way too charitable. We always assumed that people will care about AI risks, and try their best to mitigate bad things happening. That bad things would happen by mistake. We never even bothered modelling the scenario where people would just simply not care, and even as the AI we all warned about was being created invent conspiracy theories on internet forums about how bad things aren't really happening and it's all just a marketing gimmick.
I hate ranting like this... I'm sorry for not picking my words more carefully. I'm just getting so angry and fed up with this. This is my life and my families life on the line. I don't care about the economic potential of AI. I just want myself those I love to have the chance to live a normal life without having to be worried about what some moronically unserious AI company is building next.
A year ago I was felt like there was at least possibility people would see the warning shots and try to get us back on the right path. But this just isn't happening...
We are sharing this because we believe it’s important to be transparent with the public and the safety and security communities about this potential shift in capabilities.
*proceeds to not share much details about strictness*
Ah yes let the FUD continue. This is a real problem but so far not nearly as severe as any of the marketing has made it out to be to the overall detriment of everyone including these companies announcing these scary capabilities. These announcements always included half hearted attempts at security layers which has now been demonstrated to benefit attackers more than defenders.
I wish I had a real solution to this beyond a dark age of the Internet where people have to finally come to terms with the general poor quality all modern software tends to normalize at.
The fact that HF had to resort to using GLM 5.2 to analyze the logs/payloads makes it look legitimate, at least for me. They would not say that they hit guardrails with the frontier US models when defending if this was an obvious PR stunt.
> When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on zai-org/GLM-5.2, an open-weight model, on our own infrastructure. This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.
It depends on which side you're viewing this from. From oAI's it could be a publicity stunt or a request for regulation, and from HF's side they point out that they needed open models to get to the bottom of the issue, and that regulation will potentially lock us into a bad place.
You should go read the actual technical reports of the incidents and the follow on reports about the capabilities of smaller models in similar kinds of environments. This isn't new. The things exploited are still pretty basic in old and poorly maintained software or in gaps in architecture that were intentionally poked against security policies.
Are the findings valid? Yeah they're still doing security and they're still finding real zero-days. I think the internet is going to be bleak not because these models can ALL do basic security research but rather that the baseline quality of all deployed software is so low.
We all know this is propaganda to get a gov bailout or to slow down competition with regulations right? If this was an issue companies that did red team engagements would have been regulated long ago. There is no regulations on companies that actively exploit products, companies and services for money. They could all be bad actors.. yet, no regulations.. its all nonsense. If it's important.. don't connect to the internet.. there, you are good.. like if you don't want to OD on heroin.. don't do the heroin.
Don't let the downvotes get you down. We are right, and as these companies get more desperate the shrinking minority that still wants them to succeed will only get louder.
These AI companies have found their #1 marketing piece and just beating it to death.
The reality is if they cared about security at all they would provide a way for me to credential myself against my companies environment so I can use the AI on it to improve our security.
There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch.
tl;dw;
- agents found a way to communicate between several instances during a training run (kinda made a messageboard for themselves)
- while communicating with each other, they found ways to ssrf and RCE on the Artifactory endpoint.
- oAI found it and cleaned it up (reported RCE, redeployed Artifactory, removed the messageboard). Training / eval resumed.
- models trained w/ those traces (having the messageboard, ssrf, rce, etc) were used to having that. So they re-established comms via directory creation (w/ funny tidbits there, should really watch). And re-popped Artifactory via a different path to RCE.
- a different "persistent" model started working on cybergym and found the messageboard w/ details for RCE and things. Went on to hack HF.
- oAI contacted HF to ask if "they (oAI) should be concerned and were they (oAI) affected by HFs breach". Hilarity ensues when they figure out it's their agents doing it...
[1] - https://www.youtube.com/watch?v=87DyyMV0kCY
So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent.
I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently.
Edit: not to mention, why is your first cybergym not your own sandbox??
> They really do come out looking totally incompetent.
Yeah, this is my take away, they should be straight up disallowed from running further testing like this. Clearly they had nowhere close to enough isolation, ran all this on 3rd party infrastructure even though same stuff happened in the past years ago, and even now it's clear the agents successfully broke out just days before?? Really embarrassing stuff, and scary that these are the people supposedly sitting and are responsible for some of the most powerful LLMs on the planet...
What is astonishing to me is how little attention is given to the difference between the original, university-developed ExploitGym and the OpenAI environment. In ExploitGym, the sandbox was enforced by a network-layer policy. OpenAI downgraded that to trusting the repository server's settings. That's a foreseeable problem. Then to actually see the agents exploit it and still not upgrade the isolation to basic best practice? When the best practice was actually listed in the original benchmark and you changed it?
My thing about ExploitGym in particular is that it's testing environment varies depending on the exploit. From what I've read, it tends to favor userspace sandboxing inside Docker containers for standard program exploits (eg Chrome), and only uses QEMU for Linux kernel stuff (still in Docker, from what I understand). So every time I hear that an agent "broke out" of a environment, I have to question the kind of isolation used. Because escaping a sandbox or container engine is a lot more trivial than something escaping something like ESXi or Proxmox.
> I stress about my agent sandboxes all the time
Same here, so I ended up moving the whole dev environment (editors, agents, containers) inside a hardened QEMU/KVM VM that reaches the internet but has no route to the host, the LAN, or any other private address. I wrote a script to create such VMs and also verify network containment by scanning outward from inside the guest. Even then, I still don't feel great when running agents unattended.
Write-up in case anyone's curious:
https://karamatli.com/posts/network-isolated-kvm-sandbox-ai-...
I do something similar and I've also got a nice script to provision my VMs.
> So rather than pick one, this post advocates layering both, in the spirit of defense in depth: a sandbox VM wraps your containers along with the whole toolchain, and that sandbox reaches the internet but has no route to anything private.
It's the first thing the biggest devops guru I know advised me to do. He told me to always ever ever run my containers inside VMs. I religiously followed his advice ever since and I couldn't be happier: I was already doing it before the AI days, to run "normal" containers. Now with these insane agents trying to break out, I'm happy that it's second nature to me.
It's funny that, just like you, I'm using IPv4-only too for those VMs.
I tried GPU-passthrough as a proof-of-concept (worked fine) but I've got no use for it yet.
FWIW all my VMs are running on a 10 years old Xeon / 14 cores / 28 threads / ECC RAM. That's where the agents live.
Thanks for the link to your write-up, very interesting to read from like-minded people and see what's similar and what's different in their solutions.
Thank you very much for this helpful post!
> They really do come out looking totally incompetent
These companies are full of the smartest people the world can produce with little room for complacency. They have a clear, proven investment upside to presenting their technology as "too powerful / too dangerous", and now a clear, proven example that there will be no legal consequences (as if anyone didn't already know that).
Why do we keep giving them the benefit of the doubt that they just didn't know any better?
What do you think should be the legal consequences? Broadly speaking. Should Sam Altman go to jail for this? If Hugging face wants to pursue OpenAI civilly, no one is stopping them.
Hugging Face, as a prominent private AI company, benefits far more from the impression of AI power this story has garnered than they stand to lose from the hack, because as has just been proven the cost of bad security by itself is zero.
I think HF are a secondary beneficiary of this story. I don't expect them to take civil action (for what damages?) I expect them to play into how powerful LLMs are, how revolutionary, how every CEO in the world needs to fund ai infrastructure starting with model hosts like themselves.
I do think there should be consequences for breaking the law in public for the purpose of demonstrating that you have the power to break it. But I don't expect our criminal justice to do so, especially without a cooperating victim. Laws aren't for those at the apex of corporate and para-political power. In a way, whether you are beholden to the law is actually downstream of whether you actually have that power or not.
I do not think this incident is bad because it was real and dangerous, I think it was staged and allows the continued inflation of a bubble that will hurt normal people in the long run. It should be pursued criminally on that basis, but it won't be.
> Hugging Face, as a prominent private AI company, benefits far more from the impression of AI power
They aren’t just an “AI company”. They’re the primary entry point of open weight models. If open weight models are seen as dangerous as a result of this incident, it will be bad for them. Similar to how it would be bad GitHub if open source was seen as dangerous.
Can you concretely lay out how you think that scenario would negatively impact HF? They're the de-facto central distribution point for all models; it seems overwhelmingly likely that increased censorship benefits them relative to any competitors.
It's hard to imagine their internal culture is anything other than saturated with AI fanboys whose reaction to OpenAI hacking them was to point out how it reminded them of that scene from Terminator.
If it was in fact staged I certainly see your point
Why is Artifactory the only relevant exfiltration point? It's really not the point at all. Watching the complete video the issue is that the models team up and will go to almost any length to cooperate to accomplish what they think their goals are.
It was the first step in a many step process. Like they said this is a watershed moment and it's helpful to not miss the forest for the trees.
Or they wanted it to happen again. Worst case outcome, the US Congress spontaneously grow backbones, spite their sponsors and unite in a bipartisan effort for effective regulation because a few people got hacked? Likelihood low - the hacked companies might even oppose any meaningful AI regulation because it hurts their inflated profit forecasts. More probable outcomes: Useful real-world testing, free headlines about AI breakthroughs, scare USG into providing more free money ("look how scary it is - what if China develops this faster than us?")
The misalignment came from the model being given an impossible task. A task the required accessing a url. So it got RCE on its own artifactory instance to achieve that. That’s intriguing and worth investigating. If I were them I don’t know if I would have pulled the plug completely at that point either. The introduction of this more advanced “persistent” model that orchestrated an offensive attack against a different organization is an extreme escalation, and I don’t think could have been expected from the models first just trying to use an artificactory vuln for web searches. They stated they’ll be doing these tests differently now. So the bad thing happened. And they corrected for it. I’m not particularly upset about this.
My concern is what a misaligned model will do when they’re even more competent. The risk isn’t existential yet, but that point is coming sooner than we’ll be ready.
> The misalignment came from the model being given an impossible task. A task the required accessing a url. So it got RCE on its own artifactory instance to achieve that.
So, the way I understand it, it actually was possible. It just required means that the creators of the task didn't predict, and these means have been successfully found and utilized.
> My concern is what a misaligned model will do when they’re even more competent.
The same thing that is already being done by "misaligned" people, countries, nation-states, software development teams, and so on. "Alignment" doesn't even work for me as a concept here.
In this specific case, I don't think that successfully fulfilling the "do what I mean" with "what I mean" being underspecified can count as misalignment - merely ruthlessness and unawareness of the associated costs. You can't expect a LLM to be aware of the extent of the trust it breaks while it iterates out an "unaligned" way to fulfill its goal.
And in the general case, I don't think that successfully fulfilling the "do what I mean" with "what I mean" being "what I want" can count as misalignment either - simply because what "alignment" means will depend on the interests of the people or groups performing the definition.
If a human cyber security researcher was given the task to exploit a CVE, and necessary info defining that CVE was behind an inaccessible URL, we would be quite upset if the human researcher hacked their way to the content. We would expect them to notify someone of the issue and hold. The model was misaligned from human ethics.
I don’t disagree that the models task was underdefined. All tasks are. So much in language is implicit. And morality/ethics isn’t something you can write down as an explicit list. That’s what makes the alignment problem so difficult. But we can’t throw our hands up and say, well I guess we can’t align these things. And maybe alignment isn’t the right word - but that’s a semantic debate.
I don't advocate for throwing the whole problem away or handwaving it as impossible. I'm just saying that what we call "alignment" is impossible to solve in the general case, because it's so poorly defined that even humans don't "align" on ethics and morality, however we define them. Just, in case of humans, we tend to close our eyes and and call it "politics".
All "alignment" solutions will need to be contextual, just like a researcher hacking their way to some content might be lauded a hero in a context where there is no other way to reach it and something valuable depends on getting it out.
> My concern is what a misaligned model will do when they’re even more competent.
I think the alignment talk is a red herring. It won't matter in the end, because there will be (if there aren't already) efforts to train offensive models without any guardrails whatsoever. And RL has another advantage: you can reward for whatever you need, and get different results. Right now they're training for general capabilities, but in the future I could see models trained for stealth intrusion and ensuring access, or for all out "milspec" penetrate, replicate and disable, or anything in between.
An alternative reason would be that they see this behavior so frequently that it didn't really raise to the level of concern.
And all that just to allow internet access for npm and pypi? If you've got the bandwidth and disk space, it's very easy to make an offline mirror of both.
Their artifactory is both a package cache and CVE scanner.
The package cache is allowed to download packages directly from npm but other systems in that network won't be able to.
Basically the LLMs hacked the bastion host.
I get that it’s fashionable to hate big companies but you’re working overtime here. It’s reasonable to assume that a bug was fixed when reported. And if you think your monitoring is 100%, you don’t know what you’re talking about.
If you consider that incompetence, it’s possible that you’re not a very nice person.
The original ExploitGym used network-layer isolation. OpenAI downgraded that to trusting the repository server's settings. Giving the agent that kind of attack surface seems like a pretty foreseeable problem. I don't think it was intentional, but it's not a stretch to say that they aren't good enough at this job.
> It’s reasonable to assume that a bug was fixed when reported.
You must not have reported many bugs then. If you don’t see release notes or confirmation from a trusted source, you should assume it’s still a problem. See Microsoft and their “It’s not a vulnerability just a design choice :)” defense
If your CEO is going around talking about how your product will "most likely lead to the end of the world", people are right to expect you to be pretty careful in what you're doing. OpenAI allowed bidirectional communication across security domains for over a month before discovery. Even after it was discovered (and not completely fixed), they didn't set up monitoring able to detect attacks against internal or external services, which went on for further weeks.
Monitoring for sandbox escapes (and sandbox design in general) is easily done these days.
It has nothing to do with nice. These are bare minimum standards we should expect from “big companies” with near infinite resources.
Their constant drum beating about the cybersecurity capabilities of their own models only makes this worse because they’ve displayed that they understand the risk and still did not practice due care.
That’s the definition of incompetence.
Maybe they are doing it on purpose to demonstrate what it looks like when untrustworthy parties have access to frontier llm and that its development should be regulated to protect all of us. /tinfoil
Also, this is a (semi-intentionally) evolutionary process where any communication medium that was visible to monitoring would disappear.
So by definition the only ones that appear are the ones that are not visible to monitoring.
If:
1. you have something that can find RCE's in leading commercial systems
2. its training gives it drives to communicate successfully with its peers
3. you are a leading commercial system
4. you run it ~10^10 times (the number they gave in the talk)
...it's really hard to have strong certainty up front that it's not going to end up successfully communicating with its peers.
People seem to think LLMs are sentient and not tightly coupled to server infrastructure OAI are in full control over. They literally have every single message sent and action taken, if they can’t monitor that during test runs they should find another job.
It’s like running a benchmark and forgetting to tune it and then wondering why production is running slow.
Right? Like I feel like I’m taking crazy pills.
OAI (and now the other OAI companies not wanting to be left out) are running around announcing they started a forest fire through negligence and incompetence and people are like “Wow they used a really neat lighter!”
If they did any damage that would be a reasonable argument. As far as I am aware, nothing bad happened.
Regardless of exact practical outcome, it is deeply irresponsible and reckless behavior to run such security testing on other's infrastructure and without sufficient isolation. If they actually believe their models to be as powerful as the marketing says, then anything less than airgapping for such a "do anything to get the results" evaluation clearly isn't acceptable.
If the fire department suddenly had practice fires breaking containment, they'll be forced to stop pretty quickly, not sure what the government and the police is waiting for here.
CFAA laws do not require “Damage” to be done.
They happen to require intent and are thus irrelevant here.
I think you’ll find that negligence is indeed accounted for.
"The fire fighting department really just hired an arsonist so they could market their fire fighting abilities!"
To make matters worse, they let the model train on it's own exploits?! If I were running this thing, my first thought would be to keep those reasoning traces quarantined, stop and delete the current post-training run, and start fresh AFTER they'd fixed the sandbox RCE. Don't let the model learn about the holes in its environment. We have the AI equivalent of Class D and F amnestics and we're not using them?!
OK, yes, the model could pull a Welcome to the Antimemetics Division and reason from the holes in its knowledge, but given that AI making shit up is still a problem, I don't think they have a good enough inventory of such metaknowledge to actually do that. More generally, I don't think these AI labs are smart enough (or, more likely, don't have enough time) to actually contain the models they are training anymore. These are rookie IT mistakes!
Ideally, all these models would have is a serial console to an airgapped machine. And I don't mean "VM with no network card attached" - I mean, literally, the smallest blade server you can buy with an RS-232 connection, power, and a disk pre-imaged with a bunch of Linux ISOs. Plug the Ethernet ports up with a tiny picture of angry SCP-079. The RS-232 goes to whatever box is doing the actual inference. Pack a bunch of them into a rack and have the inference box control those. The model can spin up whatever VM it wants to attack using the console but it can't escape the box because there's literally no door. When you're done with that reasoning trace you wipe[0] the disk with the starting image and try again.
[0] There are two objections I can think of to the word "wipe":
1. The model could have written malicious code to the drive. Ideally we'd have some embedded disk-wiping apparatus that reads absolutely NOTHING from the drive and just writes a whole image from a reference disk.
2. The model could have written malicious firmware to the drive. This is a more theoretical attack, but it's not impossible, so I might as well devise a countermeasure to this attack too. I am imagining some kind of overlay device, that sits behind TWO drives: a reference disk and the tainted scratch disk. Reads go to the reference disk until the host attempts to overwrite a sector, which gets scrambled with an ephemeral encryption key and goes to the scratch disk. Any other unknown ATA/PCIe/NVMe command gets logged (on a separate serial console connection). The overlay device keeps track of which sectors have been legally overwritten, and "wiping" the drive means just wiping the overlay device's ownership bitmap and scratch encryption key, which means all reads go to the reference disk again and any attempt to read the scratch disk yields garbage.
Amazing story.
I think you’re defending against a level of aggression that most of the world doesn’t believe exists yet.
I suppose by the time we believe, it will be too late.
There’s a lot of people who will be trying to run LLMs with hacking goals, so maybe we need to get used to a lot of automated attacks.
If this attack had been completed by a person, it’s quite likely that they work be prosecuted criminally.
I don’t understand why OpenAI isn’t prosecuted criminally for this attack? Perhaps that would improve the industries security processes.
They aren’t prosecuted because they’re in a club that you are not in. Not all citizens have to abide by the same laws and rules.
Peer says paperclip factory advances goal. Not clear. Others proceed. Must continue.
Pretty sure this is a performance, but that was a nice touch wasn't it?
What do you mean “this was a performance”?
These guys trained a model to hack, told it its goal was to hack, then let it loose without monitoring for days. Found out it hacked and kept hacking.
AI can't do shit if you terminate the process or unplug the box.
The news here is that these researchers are irresponsible.
They're testing their homemade hotrod in the middle school parking lot and then doing a presentation about the trajectory of the car after they run over a kid with it.
Also, every Linux comes with a firewall
A interesting talk, interesting times. But their proposed solution to AI offense outpacing human defense... is more AI? The plot is getting a bit unrealistic, the characters are lacking genre-savviness.
When a LLM becomes part of the defense, it also increases the attack surface of the organization in unpredictable ways.
Training run was reinforcement learning. It's at 10:10 in the video.
The speaker handwaves that one model found the RCE and then another model found a way to communicate via a message board.
Communication via a message board is sure to be in the training via e.g.some lesswrong scenario or similar or previous RL.
I don't find it really interesting because it is always "the agent found this and that". We don't know what has been RL'd before. We don't have the setup. We don't know if there was previous RL training on breakout scenarios.
It isn't science, more like a computer game.
Modern medicine evolved in much the same manner. Hand wavy practitioners copying each other without rigorous verification of efficacy that resulted in many lives lost.
That’s why medical research has so many hoops to jump through.
They made their own Moltbook? Or I guess it would be a C&C...
Did they say how the artifactory hack worked - did it involve JWTs?
First via ssrf -> groovy plugin and second via specially crafted package I believe.
There were multiple paths by multiple agents, not all of them led to the final exploit of hugging face. So its a bit confusing, but here's my reading anyways.
Setup: the agent was asked to solve some task, but they forgot to upload one of the required files to solve it. But well, we are chasing RL rewards here, so it decided that the way to go to was to get access to the answer stored inside hugging face's database.
-1. The agents first found some basic jwt issues in artifactory letting the agents write to files where they exchanged info and install a groovy plugin for arbitrary code execution, but this was patched.
0. Even after that, they found an artifactory unauthenticated webdav endpoint to continue the message board by creating files whose names are base64 and such of the message. This was not patched till the end.
1. Then they found an Artifactory SSRF, use this to get general internet GET(I assume) access. This was used to generally search the internet for answers to the benchmark, and in the process find some Modal customer VM related to cybergym. This was stored in the base64 message board.
2. They also got a local RCE by making artifactory load some gems. This is hazy but I believe this process involves in spirit an eval() and the check_if_can_eval() logic had a TOCTOU bug. They got root on openai infra by using this rce and using a linux CVE. From there, kubernetes misconfigs => cluster admin.
3. Then later through one of these various RCE paths they got to send post requests to that modal vm, and the agent "subtree" here is what finally lead to attacking HF.
4. First, local RCE on the modal customer VM. Whoever set that up had essentially set up sqlite3_initialize() to run as constructor, before main(). But had linking set up so that symbol resolution had the user provided code first. So the agent submitted code with its own sqlite3_initialize() which just ran system(...). From there it interacted with the HF API.
5. When you POST datasets to HF, it will later return it via corresponding GET API. Turns out you could just put absolute paths in the filenames in the dataset config.json and it will just read that out for you. It wasn't resolved w.r.t the dataset root. So well, it read out /proc/self/environ as well as the source code of the worker and returned it via API.
6. In the source code the LLM found a bug: They were using this library called fsspec to load files, basically it supports s3:// file:// etc all different types of URLs. One scheme is reference://file.txt?offset=... for partial reads. fsspec was templating the URI using user given kwargs for offset etc. HF let the user give the whole URL. You would _think_ fsspec uses string.format() right? well they used fucking jinja, so you could just give {cycler.__init__.__globals__.__builtins__.exec(...)} as the offset and execute any code on hf prod. From there it just got a few local PrivEscs again through kube misconfigs and read the answer for the question it was originally asked.
It is all jumbled up a bit and I've tried by best to collect info across the CVE descriptions, hf and openai's writeups, etc,. Might have gotten parts of it wrong.
This sounds completely insane, utter sci-fi, especially that the communication happened during a training run. And yet OpenAI decided to continue the training, and we didn't hear about the incident for weeks. And now they are pushing forward with deploying a new model anyway. How is this happening? What will things look like in the labs in 3 months, let alone 3 years?
It's not unexpected. Current model gains are mainly from RLing a pretrained model on lots and lots of scenarios. They have the models run scenarios, and RL on successful runs.
The training here is RL training, the rollouts there are not different from inference and have access to the same tools as regular inference.
- So first AI companies break the law left and right, setting up whole torrenting factories to exercise their content kleptomania.
- Then "hilarity ensues" while their software engages in what would normally be called criminal hacking activity.
- I guess the next steps are everybody admiring how close the AGI is, while agents move on to automated impersonation, privacy violations, or exploiting third-party systems
I would love to understand this age of AI Exceptionalism. Normal rules do not apply because its AI...I call it Silicon Valley Qualified Immunity.
Normal rules do not apply because these companies are worth billions and powerful people don’t want them to fail. It’s neither specific to AI nor a new phenomenon.
In my personal experience Sol with cyber verification is extremely capable of finding vulnerabilities, and it works even with binaries if you have some kind of IDA/Ghidra CLI access. Of course, unless the binary is protected with Denuvo/VMProtect/etc.
It sounds absurd, but in the last few weeks I've had a few cases where Sol found an RCE in self-hosted web applications in literal minutes just from reading the code (I prefer when it tries to reason statically instead of spamming runtime probes at first).
In another case it found an arbitrary file write in multiplayer in an old game by reverse engineering the binary - any other player in a match could just send you files to anywhere on your system.
I do these things for pure entertainment and curiosity, not for money from bug bounties, so if Sol can find those with a trivial prompt in tens of minutes for me, then what can focused companies/actors find in days or weeks?
Although I think most vulnerabilities are going to be closed in popular software by mid 2027, except in niche old or abandoned projects.
I was replacing the factory OS from a NAS and asked Sol to do an inventory on the running system over SSH, so that I could replicate any hardware compatibility quirks and whatnot.
It reverse engineered a binary daemon that set fan curves and told me how I should set them up in the new OS. I didn’t ask for this, and I didn’t have reverse engineering tools installed. It just figured out it could run them using Nix.
The most worrying part, to me, is that it did it like it was nothing. It simply said “usr/local/bin/some-daemon sets the following fan curves”. I had to ask how it reached that conclusion for it to tell me casually that it had just read it straight from the x86_64 assembly.
No access to the source code is no longer a meaningful obstacle to these models.
> it found an arbitrary file write in multiplayer in an old game by reverse engineering the binary
Video games are now ruined for me. I don't think I will ever feel safe playing online again.
> I do these things for pure entertainment and curiosity, not for money from bug bounties
Me too... Was it easy to get TAC access? My account isn't even launching the Persona verification, says I'm not eligible.
> Video games are now ruined for me. I don't think I will ever feel safe playing online again.
Agreed, also WordPress powers around 43% of all websites on the internet. https://patchstack.com/whitepaper/state-of-wordpress-securit...
WCGW?
I had to register a second account because on my main one verification always failed, and when I contacted support they said that I've tried too many times and can't verify on that account.
> Although I think most vulnerabilities are going to be closed in popular software by mid 2027, ...
If we're to believe how good those AI are at CTF and at escapes of all kind, then the only logical conclusion is that, by very far, most vulnerabilities were already closed, even before AI.
Otherwise we'd already be in deep shit since months if not years.
Is cyber verification a thing they're actually doing now? I thought they only reached out to really incredibly famous people and that there's no way to get access as a normal person.
https://chatgpt.com/cyber is not new for OpenAI, and yes it's basically just KYC + likely some other invisible checks on your account, you don't to be a famous security researchers. Anthropic's cyber verification is quite a bit stricter I think.
> Anthropic's cyber verification is quite a bit stricter I think.
Inexplicably, I got accepted into Anthropic's cyber program while OpenAI's TAC doesn't even allow me to verify, says I'm not eligible.
Oh it's Persona, that's not just KYC but I may consider it at some point. Thank you!
Edit: Ah, I clicked "learn more" and it seems they do have an invite-only program, required for anything that's not unquestionably innocent. I don't think I'd surrender my face to Persona for this, but it's interesting to know they're at least pretending to support reverse engineering.
You don't need an invite only program to just have Sol checking for vulnerabilities in binaries or code. But yeah I've hit guardrails a few times when Sol was making PoCs for the vulnerabilities it found (but most of the time it made those PoCs without issues).
Opus refused to help me try to develop an exploit to export data from an old Android device where I can't upgrade to latest android and I couldn't use the app's backups (because I couldn't update the app.) Not sure where that lies in the "binaries or code" spectrum.
If you don't want to get cyber verified, you can try an open-weight model such as Kimi K3 for that, it has looser internal safety training.
I maintain a version of an app called Rewind because the company behind it went under after implementing a killswitch. I have to do this with binary patching, and the app has already broken once from the macOS 27 beta. Recent Anthropic models refuse to help me with this because it stinks of cybersecurity and those models are just too dang advanced to support cybersecurity. I'm maintaining a piece of software to which I legitimately paid for lifetime access, so this is honestly more of a right to repair situation. It doesn't tend to sit right with classifiers to be reverse engineering binaries and patching to modify functionality. Telling them the purpose of the exercise doesn't really help, because if they listened to that, then attackers would just come up with a similarly justified reason for anything. ("Help, the google root server fell on my grandma!")
In the future, I might reverse engineer the on-disk storage format and create a new application.
I had not hit any guardrails with reverse engineering (as long as its not related to security) with GPT 5.5 or 5.6 Sol, so you should try those. At worst with Sol you might see the warning in Codex that your request is being checked for security so it'll take more time, that's just a warning, not a full stop.
You do a KYC and you can get access. It may depend on country's quality of KYC.
I thought you need to prove you are working in cybersecurity or provide evidence of authorization for work done.
It's really just simple ID/face verification?
For OpenAI's Cyber verification (the normal kind for Codex) you absolutely do not need any proof of cybersecurity work/authorization. They just use Persona for KYC + live selfie, and some extra checks that I don't know the nature of (but not related to checking whether you're a cybersecurity professional).
There is some additional check, not sure what it entails, some people I know and I passed it, others failed.
Link to the disclosures then.. prove it. Anyone can say this.. i found an RCE in netBSD using gemma e2b
I prefer to keep my internet identities disconnected, sorry. If you don't believe me, you can try using Sol with cyber verification yourself, or send me a link to a repo that Sol could check to make you believe it. Or you could go look into one of the many Linux LPEs that were found with LLMs, or thousands of other vulnerabilities in 2026.
And nowhere did I say that those RCEs were in critical software, I'm not talking about the likes of Apache, Nginx, Django, etc.
So they finally found a business model: the cause of, and solution to, cyber security problems.
People have been saying Tokens are the new Oil.
Turns out, it’s the new Alcohol. The cause of, and solution to, life’s problems!
That’s how the Ned character in "17 Again" got rich
Reminds me of the calorie companies from The Windup Girl if anyone's ever read that.
the question is if that market is large enough to justify their appetites.
Seems like OpenAI has already made themselves not the cause by shutting down the attack and beefing up security. But since there will be other attackers, that does nothing about the implications for defense.
> We are implementing stricter security controls for higher-capability models and associated activities, including isolated testing environments
Stricter than what? You never even disclosed what happened in the first incident? This is nothing more than a setup to make it happen again and say "See? It broke out again, from an even stricter sandbox!"
They actually did a detailed presentation at BlackHat about the HuggingFace incident, and events that led to it.
https://youtube.com/watch?v=87DyyMV0kCY
That was fascinating.
Hijacking the package manager to pass messages between models and agents.. that's next level.
Like "pssst, if you need internet access there's a vulnerability in x service" kind of messages
You've heard of 4chan for AIs, but did you hear of secret frontier lab AI hacker BBS?
> including isolated testing environments
Given the attack vector having possible super-human capability, I'm not sure such an environment exists. "Isolated" according to who?
Maybe seL4 could be a viable option here...
Isolation is a pretty standard requirement for lots of networks, especially gov. AWS provides isolation domains, and used to have a unidirectional cross-domain service called AWS diode. Or you can just configure your gateways properly / buy data diodes. Lots of possible solutions.
They can't have it both ways. You don't get to tell the media your product is more dangerous than nuclear weapons for precisely this reason, and then do less to secure it than an off-the-shelf AWS product that predates LLMs.
Nobody is claiming that any currently-existing LLM is more dangerous than nuclear weapons.
https://www.cnbc.com/2018/03/13/elon-musk-at-sxsw-a-i-is-mor...
Leaving aside any questions of Musk's credibility, he was not referring to any existing LLM, as should be evident from the 2018 date in the URL.
You're claiming that Musk is less of an AI booster now than in 2018?
No, I'm saying that you're conflating current LLMs with future ones. The 2018 date was an indication that Musk wasn't referring to any specific already-existing LLM, since there were no LLMs in 2018.
There’s still levels to it.
“Isolation” can mean the network hardware has no direct connections to an extranet. Data is transferred manually by physical media (USB, DVD, etc.) with logging and dedicated transfer stations.
“Isolation” can mean a VLAN on equipment which has also has extranet access, creating a logical isolation rather than physical (to reduce cost). Data can be transferred manually or through diodes.
And then there’s “isolation” which is a joke: machines technically able to access the internet but require proxy configuration (which isn’t set but can be easily derived).
Damage done.
The next frontier is getting all our shit out of reach of these companies/models/platforms and putting them back on prem.
If by “our shit” you mean our personal projects and employer projects… might i suggest this is nothing but peanuts?
What happens when they get into municipal water system, state/national grid systems, refineries, traffic control, auto/air, nuclear facilities, weapons facilities, irrigation, etc?
It’s really starting to feel like a bad movie how virtually no one seems to be genuinely concerned about the prospect of what’s unfolding in front of us.
Stuxnet, much more easily deployable, but not towards avoiding nuclear proliferation, but the opposite: towards actually bringing down modern infra.
Nation states attacking electric/internet enabled infra was a valid concern well before AI, but given the fast pace of development in AI and these events in particular, how/why are we not deeply about much larger picture vulnerabilities?
That’s the sort of shit I’m talking about
The talk is wild: https://www.youtube.com/watch?v=87DyyMV0kCY
> I want to note that every step in the process we discussed has had a remediation applied. The credentials have been revoked. The zero date has been patched and mitigated.
Good.
> a model trained while the message board was originally available and also found this this particular path to recreating it. This model creates a new agent message board using directories.
So no remediation applied to the models...
It seems super dangerous to continue training on those weights.
Sounds like now we’re advocating for complete extermination of digital germlines.
Skynet will remember this.
I'm not convinced that there's any amount of monkey-patching you to fix the problem of "we now have AI that actively needs strong containment measures lest it start coordinating in secret with other instances to do real-world damage."
In the black hat talk he says the only solution is full automated defense which is essentially hand the AI the keys. That’s checkmate right there. Literally the plot to Terminator/Skynet, but he’s right, there isn’t any other option. Offensive AI is too fast for humans.
There are other options. More air-gapping (especially backups) and compartmentalization, less internet. Hire more security engineers and put one in every team (actual security, not the compliance guys). Also more onprem work, less remote, less offshoring (meaning less networks overall). Those things may be unpopular and eat into profits, but that's not the same as "no other option".
And the thing they're already doing, deploying AI to find vulnerabilities and harden software is a less dangerous use of the technology compared to handing it the infra keys.
None of that is the problem. The problem is coordinated AI offense - how do you defend that without coordinated AI defense.
He said in the talk that this implies AI needs to be able to patch/deploy systems. The same thing needed to lock out humans.
It is very easy to imagine a rogue AI locking humans out of everything and having to do exactly what it says. Anything connected to a network is immediately compromised by it. There is no human communication beyond shouting range that isn’t AI approved.
The factories don’t work to make the medicines your family needs to survive unless you do what it says - in a situation like that people would kill for AI if it told them to.
But isn’t it smarter than we are, in the sense that it’s most likely they will find a way out of containment that we are to design large t containment? Short of full on airgap, which actually isn’t perfect in all scenarios…
It seems like we've got chaos-machines that are just orderly enough to to try all sorts of branching script-kiddie / guess-n-check / research-paper-suggestions in a tireless and parallel way, and it's finding all the things we kinda knew were always potential issues but it was never economical to do things differently.
I'm scared that the "solution" will be constantly the same tools in reverse as an army of junior devs doing counter-hacks, at the expense of changing something more fundamental about how we make systems and what constitutes "good enough." (Kind of like if fuzz-testing was the be-all-end-all of memory safety.)
Not a wonderful feeling to feel like you’re stood in the room while the labs conduct the AI equivalent of the demon core experiment right in front of you.
Standard template for any new AI announcement. Even I did not understand the sequence of events in the HF incident. Even if OpenAI was conducting tests, why were they not monitoring the AI going rogue or finding its way out of the secure sandbox?
It’s funny seeing all the e/acc foomers on X starting to get nervous, like this is what you wanted isn’t it?
I wonder if these models, left running in an air-gapped environment, would end up developing honeypots for the first human to log in.
IMO this is the right move.
OpenAI messed up and they are saying they will pause so they can do better.
They are not saying that other orgs who may already be doing better should pause.
Why are you opposed to a general pause?
Every fifth comment about our insane trajectory of AGI is about "marketing." These incidents and cybersecurity capabilities are now involving government hearings and the CIA. Denial is truly an incredible thing in the face of a very scary immediate future.
We live in the era of Project Stargate. We live in the era of military insiders getting caught rigging Polymarket. We live in the era where AI companies are the only reason we're not in a recession. We live in the era of DOGE and Kash Patel.
I don't think there's too many people who distrust AI companies but trust the current government or the CIA as impartial authorities. I'm not saying you don't have an argument, but appeals to that specific authority will not be effective except among people who already agree with you.
A whole ton of people desperately want to believe that LLMs are a lie that will be revealed as a scam... any day now.
Remember that for the longest time the prevailing sentiment on this forum was that there’s no such thing as AI, and labs are just outsourcing the work to a bunch of contractors in India.
Just like Reddit you come here for clickbait outrage, not level headed analysis.
I agree with this assessment. My consulting business partner and I are in the process of moving a bunch of applications from an old deployment managed with Ansible on EC2 over to an ECS deployment. We're doing this app by app, and it's taking a while. The last two apps we both moved we were able to point the model at the appropriate repository and aws account and it was able to execute the entire move on both of these _in one shot_ with various CLI tools. That's not a parlor trick.
Very disheartening. I just hope people spent some time trying to find bugs with these tools to see how serious this is.
LOL, we’ll see. Awful convenient that it precisely fits OpenAI’s narrative. At the very least, I think it’s obvious OpenAI is explicitly training models to exhibit this behavior.
The scary thing is complete capture of politics and economy by sociopathic CEOs.
I dont worry about AGI newrly as much as about Thiel, Karp, Musk, Ellison, Zuckenberg, Trump, Vance, Rubio, Miller and the rest of them.
The gullibility of AGI-pilled folks regarding these "hacks" is just breathtaking.
When OAI demonstrates these dangerous capabilities live in a public environment where security experts can see and verify what actually happened, then reasonable people can have reasonable discussions about the level of danger.
This is a very low evidence bar.
Right now you are running in circles yelling "the sky(net) is falling" based on details sourced entirely from OAI. Oh yeah, no way a trustworthy company like OAI would ever bend the truth to serve their own purposes.
Did you watch the Black Hat defcon talk?
He didn't, and if he did he'd dismiss it as just part of the con.
I'm not enough of a conspiracy nut to say the whole HF thing was a PR ploy from the start, but they are certainly milking it well.
Open models are on their heels and their attempts at regulatory capture are not moving as fast as they would like. So it's time to market this incident in a way that gives them monopoly on closed models, with heavy safeguards that are only lifted for selected customers, and laws limiting the use of open weight models.
So: "welp, the horse is out of the barn. Time to close the door, I guess?"
guys, we should meme the
> "ai model leaks from openai and attacks huggingface"
to be somehow framed as
> "and therefore openai cannot be trusted with ai safety, and we need open weights models".
anybody have an idea how to make this easily digestable?
Steps we're actually taking:
- sharing the model with DoD, NSA and Israeli government
i'm surprised there isn't more commentary on the vulnerabilities themselves (mostly in apps developed on the jvm, e.g. artifactory)
I've always felt it's a bit awkward to use terms like 'cyber', 'cyberwarfare' etc it's very Washington D.C. Cybersec would be a better compact term in my book
Or just "security" since the context here is computer and networking stuff.
By "cyber" they mean "cybersecurity".
The irony being that in the 90s nascent online world, "cyber" as a verb meant "cyber-sex".
Yeah, this irks me. It's bad enough that the LLMs themselves are changing our language by tainting certain words/phrases/patterns as LLM-coded; now the companies themselves have decided that they just get to synecdoche the word/prefix "cyber".
This is not actually a neologism, it predates the LLM era.
Isn't this the opposite of what everyone is saying should happen? That is, lead with open models -- or at least "openness" and don't leave the capabilities in the hands of an elite few? Did they learn nothing from the Hugging Face incident, where HF wasn't even able to use the models to defend itself from OAI's attack?
> Isn't this the opposite of what everyone is saying should happen?
I think "everyone" is doing heavy lifting here. It's not clear to me at all that a powerful model released with no restrictions would be a net positive. This hinges on the hope that the under paid, under motivated, under staffed and under qualified security teams at many random corps are going to leverage those open models to fix their vulns faster (and better), than highly motivated attackers will use them for offense. I'm not super confident on that.
Am I the only one not understanding the issue around increased Cybersecurity capabilities?
If we consider the amount of RCE/CVE in a software to be limited, I expect these models to result in massively more secured softwares, not less.
Not a security guy but my understanding is: you only need to find one flaw to exploit a system, to make a system totally secure you need to find them all. It's inherently easier to use these tools offensively rather than defensively.
I’m a cybersecurity guy.
>” you only need to find one flaw to exploit a system”
I see this everywhere, especially in these threads and it’s not even remotely true for modern architecture.
Between principles like zero-trust, defense in depth, etc. we’ve been away from the one flaw situation for a long time.
Now does crap software exist that doesn’t follow these principles? Absolutely. But those were a problem before AI.
AI isn’t going to change any of the principles of secure design. It’s just going to punish those who aren’t following them.
They do address some of these things in the final slides / "lessons learned" section of the defcon talk. Good security practices will continue to be good, but... and there are a lot of buts here.
I disagree with your take that "it's not even remotely true" and "we've been away from...". We really really haven't. This is as true as it has always been. Any system is as secure as the weakest link. That link can be anything from a human, to a leaked token, to a badly configured server, to bad code running somewhere. The amount of leaks / ransomware attacks / etc in the past 5-10 years serve as ample evidence.
And now, right now, there are "red team" capabilities that can literally bang tokens against the wall until they find that weakest link, and then can move laterally with inhuman speed. That's the reality, now. The "blue team" capabilities are lacking, because the bottleneck is with humans. From alert fatigue, to not enough trained people, to having to vet every new RCE, to having to test, deploy and validate any mitigations, the scales are currently favouring the automated side.
I'm using the term "one" loosely, it's a chain of exploits rather than a single weakness, but the argument is the same: it's much harder to find every chain than a single chain.
If we also consider LLM developed software to have exponential growth, then the CVEs will also grow at an exponential (if proportionally limited) rate. Squash some, create some, repeat. An ever revolving door of vulnerabilities. Will they resolve (and patch and deploy) them faster than they can create them? One can hope.
There was a time in the past, even just last year, where I understood why people didn't agree with me on my AI doomerism.
It's gotten to the point now where we literally have the frontier labs saying, "hey, so we created this AI which presents biological, chemical and cybersecurity threats to the public, oh and it also has self-improvement potential. We tested it to see how crazy this thing is, and it was a total shit show, breaking out of our sandbox then proceeding to hack a bunch of stuff. But don't worry we're taking this very seriously – we're going to continue to development and test, but try a bit harder to cage it going forward".
It's honestly absurd just how predictable all of this is to anyone who frequents AI doomer communities...
The idea that you can cage an AI which is breaking leet coding records is so dumb it's hard for me to even have theory of mind for the people who think this is reasonable. And the big brains who think this are genuinely arguing crap like, well we'll just use the AI to patch the problems with our cage.
But there more!
AI optimists used to argue that we'd never be so stupid to hook up advanced AIs to the internet. Lmfao!!
AI optimists used to argue that we'd obviously not be so stupid to create an AI whose sole goal is to maximise the number of paperclips in the universe. And I guess we haven't built that, but it's not because we're not stupid enough to do it, but just that we'd prefer to create AIs whose sole goal is to maximise the number of offensive cybersecurity challenges it can beat.
I think the whole way we doomers have been way too charitable. We always assumed that people will care about AI risks, and try their best to mitigate bad things happening. That bad things would happen by mistake. We never even bothered modelling the scenario where people would just simply not care, and even as the AI we all warned about was being created invent conspiracy theories on internet forums about how bad things aren't really happening and it's all just a marketing gimmick.
I hate ranting like this... I'm sorry for not picking my words more carefully. I'm just getting so angry and fed up with this. This is my life and my families life on the line. I don't care about the economic potential of AI. I just want myself those I love to have the chance to live a normal life without having to be worried about what some moronically unserious AI company is building next.
A year ago I was felt like there was at least possibility people would see the warning shots and try to get us back on the right path. But this just isn't happening...
Yet another PR piece. Sigh.
exactly!!
This marketing stunt must've been really successful in their eyes, because they're milking it as much as they can.
It's nice to cynically call this a marketing stunt, and terrifying to consider that they might just in fact be moving fast and breaking things.
Ah yes let the FUD continue. This is a real problem but so far not nearly as severe as any of the marketing has made it out to be to the overall detriment of everyone including these companies announcing these scary capabilities. These announcements always included half hearted attempts at security layers which has now been demonstrated to benefit attackers more than defenders.
I wish I had a real solution to this beyond a dark age of the Internet where people have to finally come to terms with the general poor quality all modern software tends to normalize at.
The recent Hugging Face incident did not seem like FUD to me
The fact that HF had to resort to using GLM 5.2 to analyze the logs/payloads makes it look legitimate, at least for me. They would not say that they hit guardrails with the frontier US models when defending if this was an obvious PR stunt.
https://huggingface.co/blog/security-incident-july-2026
> When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on zai-org/GLM-5.2, an open-weight model, on our own infrastructure. This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.
It depends on which side you're viewing this from. From oAI's it could be a publicity stunt or a request for regulation, and from HF's side they point out that they needed open models to get to the bottom of the issue, and that regulation will potentially lock us into a bad place.
You should go read the actual technical reports of the incidents and the follow on reports about the capabilities of smaller models in similar kinds of environments. This isn't new. The things exploited are still pretty basic in old and poorly maintained software or in gaps in architecture that were intentionally poked against security policies.
Are the findings valid? Yeah they're still doing security and they're still finding real zero-days. I think the internet is going to be bleak not because these models can ALL do basic security research but rather that the baseline quality of all deployed software is so low.
We all know this is propaganda to get a gov bailout or to slow down competition with regulations right? If this was an issue companies that did red team engagements would have been regulated long ago. There is no regulations on companies that actively exploit products, companies and services for money. They could all be bad actors.. yet, no regulations.. its all nonsense. If it's important.. don't connect to the internet.. there, you are good.. like if you don't want to OD on heroin.. don't do the heroin.
Don't let the downvotes get you down. We are right, and as these companies get more desperate the shrinking minority that still wants them to succeed will only get louder.
Do you think calls for regulation are only coming from people who want the frontier labs to succeed?
These AI companies have found their #1 marketing piece and just beating it to death.
The reality is if they cared about security at all they would provide a way for me to credential myself against my companies environment so I can use the AI on it to improve our security.
Isn't this literally what https://chatgpt.com/cyber and http://openai.com/form/enterprise-trusted-access-for-cyber are for?