Please regulate us, now! If we don’t get regulatory capture ASAP I’m toast. I mean look man I just had to go beg for tens of billions more cash so I can keep the lights on and my IPO is looking more and more screwed every day. Please, I’m begging you. What more do I need to say to convince you?
I'm fascinated by this forum's take on AI regulations.
Whenever there is an adverse event, people say of course it should be regulated. The EU AI Act is a good start but we need more. These labs need to be held accountable just like all industries that can impact others. Regulate the chips, regulate the models, regulate the companies.
When these companies ask for regulations, people say they are trying to do regulatory capture. They are trying to ban opensource models, and cement their place as the AI arbiters and ban all others. We need competition and open weight models and China is leading the charge.
I understand that the 2 position aren't exactly opposed. We probably need regulations, and ideally ones that don't produce regulatory capture. That is extremely hard though. Almost all regulations end up creating a form of regulatory capture. Any regulation compliance has a cost. The more complex and wide reaching the regulations are, the more expensive it's to comply and the more they cement the existing players and make it harder for new ones to enter the market. FAA regulations makes it so we don't have dozens of passenger airplane startups that are trying crazy wacky things. Food services are regulated too, but the compliance is fairly doable by mom and pop stores, yet probably too high for your unlicensed pop-up taco stand.
I haven't really seen any meaningful discussion about what AI regulation should be. Only extremes of "No regulations, all is fair" or "no one should have access to running or developing models but OpenAI and Anthropic".
The problem with regulation is that it won't (can't) solve the problem. You'd have to go totally draconian to stop law breaking people within the US from doing matrix multiplies, and then you'd have to disconnect the US internet from all the countries who won't give a crap about US regulation.
So yeah, US regulation is ONLY about capturing US consumers. This isn't food or airplanes.
Further, its about ensuring the open weights models never catchup/exist so these guys dont turn into a hardware reit like they should and they can keep their insane margins
Are you conflating regulation and accountability? I haven't seen anyone (without a vested interest) ask for the former as a response to an adverse event, only the latter. There is a difference.
Regulation should require model auditing, which should be fully automated, free(or extremely affordable), and time-bound, irrespective of who the client is.
What Altman, Dario and other such folks are trying to do is put a price on getting the model audited, by playing on these: price, timelines, red-tape.
Maybe a test-suite, kept continuously updated, similar to those benchmarks we already have for measuring the intelligence of these models. Like arc-agi, or terminal bench, or more.
You seem to think that regulation means hitting the brakes. We're way past that point. This is "we need to mobilize an AT squad" territory.
Sure, research will continue in secret, just like nuclear, biological and chemical weapons. It's just that there's going to be an economy backing it up, with MAD ensuring the attacks are "civil", as Axl once said.
Yes, I'm. I didn't mean for my list of extremes to be an exhaustive list. But yes, NOBODY (in a "butlerian jihad" sense) is just another example in the list of extreme takes. Only governments or military is another extreme.
Some of us want to nationalize these companies. What we’re not going to allow is landed gentry to profit for generations from stealing all the content they could scrape.
I'm so much less worried about AI regulation than I am about vulnerability research legislation, which is much easier to half-ass and apply without goring any major economic oxes.
I don't think this will mean holding back, as nothing will hold back whatever is providing tech to the myriad of parties interested in poking holes. Instead this points to common defense, as anyone who doesn't have access to defensive tech will be toast.
Yes and no. These things have always existed and were typically known but not broadly exploited. A huge part of what nation state hackers do is exploit these things for intelligence use.
So I would agree that this is bringing more of this to light, but the idea that unknown exploitable bugs can get weaponized being a new thing is naive.
If anything putting all this out in the open makes things more secure, not less, in the long run as it’s harder for hackers to know and exploit them before they’re known as they don’t survive undetected for long. So a bit painful in the short term but we get more secure infra in the long term.
Still not a reason to give Sam regulatory capture as an out to saving OpenAI.
But all jokes aside, we're entering a new era where there's a paradox of pro-AI people going AI is really good, but not that good... when it kind of is that good.
We'll probably have ai monitors as a service, but the bulk of the safety should come from AI doing better job of reviewing tricky bugs since humans refuse to spend effort on writing more formal specifications.
In Mike Pondsmith's Cyberpunk universe (the same one as Cyberpunk 2077), there's this point of in-universe history that the Old Net got overrun with rogue, nigh-on sentient AIs that made safe use of the internet as it was known then impossible.
Society's response to that was to build a new Net, walling off the rogue AIs of the Old Net behind the Blackwall and carrying on in a corporate hellscape.
I wonder if we'll see life imitating art on this one.
> humans refuse to spend effort on writing more formal specifications.
Code is formal specifications. It's just that programming language semantics has no domain related value of correctness. So you overflow an unsigned integer variable and got a lesser number which does violate addition rules. Or increase account A without decreasing account B, which violates accounting rules (and all kind of closed systems).
So it's up to the programmer to encode all of those rules. While we do have some tooling like type checking and memory safety, they are nowhere near enough. And encoding such thing in a language like Lean or TLA is brittle in terms of maintenance (which you do need to do as the real world evolves).
AI's spottiness is exactly what describes this era. AI can create amazing code/image/video/article exception XYZ really annoying qualities that somewhat hobbles its ability to be autonomously.
And sure, maybe those problems will go away once we're at month X of the madness but we're still at month Y so are going to going remain for some moment.
The bulk of the safety should come from AI doing better job of reviewing tricky bugs since humans refuse to spend effort on writing more formal specifications.
But that part is the key point. We've had many years of networks building inherently insecure structures with security tacked as feature and the primary security process being more ad-hoc stopgaps.
Even if you write a formal specification the operating system the code runs on is almost certainly not formally specified so you're just moving the trust boundary slightly further out to the OS & associated libraries.
That also assumes that your axioms are correct and the verification code is also correct. It's also a lot of work to update your system, when assumptions are challenged.
exactly. somehow many people have the idea that C-like imperative code harbours bugs because it cannot be formally reasoned about, and the answer is writing code that can be formally reasoned about.
That the resulting code is many orders of magnitude larger and more complicated than the "informal" original, and often simply cannot exist, and is incomprehensible to most humans if it does, is left as an exercise for the reader.
I mean, formal methods cannot even prove collatz conjecture terminates. Somehow that thing seems to be running some sort of almost turing complete machine in the number encoding. Good luck formalizing existing software!
I genuinely would like to know your opinion on what the next steps would be then? If every single exploit was available tomorrow accompanied by a working exploit to operationalize it, or combine it in a chain to compromise any software that has such a vulnerability, would you post it on the internet?
I know you a bit by reputation so that was my assumption. I assume you mean responsible disclosure? I thought your statement was implying otherwise, maybe the unsaid implication was against the "extortion" argument?
In any case, what I want to say is this: All technology has tradeoffs but this has shown we are locked in a game theory mechanic not dissimilar to extortion AND the security implications are part of that.
- We have to build AI or someone else will.
- You need our tools to protect yourself from our tools or similar ones
- You need to use AI or someone using AI will take your job.
Many examples but it is very "i created the disease to sell you the cure". If this were any other technology, we would expect more reasonable mitigations, extraneous factors like software bugs existing aside. It speaks to a fundamental lack of control on the part of the labs that we take for granted: they simply don't have control of the PRODUCT, much less how it is even used.
Absolutely not. "Responsible disclosure" is a coercive Orwellian term coined by vendors, most importantly Microsoft, in the early aughts in order to set a norm that vulnerability researchers should work on vendor timelines and with vendor disclosure constraints.
Finding vulnerabilities that already exist in code can't be "creating the disease", for obvious reasons.
Ok. I was sympathetic to that historically for all the reasons you stated. If you put the effort in, why are you constrained. I think we are in a different world now. So what would you do in my thought experiment?
You are the QA head and the new model could literally find and exploit ANY bug? Would you release the model? Genuinely curious. Its not totally science fiction. I think there is a difference in a couple smart people punishing 1 sloppy company that underinvested in security. The internet is a stack of protocols, hardware, infrastructure, applications etc. It could be the case that almost everyone has capabilities on par with yours. Great you say, security will improve! However, there will be a meltdown in the middle where potentially digital currency is exploited, exchanges, banks, utilities. It isn't just a team working for 3 months, finding some issues in a company.
I cannot help to think that the idea of go fast and break things had something to do with the current state of affairs. I would say that the next step should be slow and consolidate, but it seems like we are not heading in that direction.
At this point I don't know what will be worse for the global economy the AI Bubble bursting, or the gigantic crash all this vulns might bring if they force the stop because we went too far and everything came crumbling down...
Ah yes. Everyone eventually dies, we just shorthen the wait.
This is like telling that bank vaults always were breachable, and giving out explosives to everyone just made them get breached sooner.
(edit: fact of existance of a vulnerability is not a binary yes/no. the amount of effort needed to exploit even known vulnerabilities is usually staggering, requiring months from very capable individuals with very specific skillset.)
In your analogy, one phenomenon concerns a living organism and the process of senescence, and the other a piece of information. I don't see how you connect the two.
Both my analogies use hyperbola to comically exaggerate the claimed lack of any difference. just as claiming that advancing a demise of a human makes no difference because it will eventually happen, or claiming that breaching a bank in practice is not a big deal because it was possible in theory, so also claiming that theoretic possibility of a vulnerability being exploited is more or less the same as many of them being exploited at the same time.
The covid was not a lot deadlier and not a lot more contagious than normal seasonal flu. That "not a lot" overloaded hospitals and caused global shutdown.
The increase in rate at which software is being breached right now in in no way "not a lot". Devs were overloaded even without this deluge. We can tank one such php exploit. Then another one. But for how long? If all your smartcards, secure elements, cloud providers, wireless protocols, drivers, government registries etc etc will be having weekly breaches of this magnitude, won't the effect be systemic?
None of these analogies are apposite. They don't respond to the observation I made: the vulnerabilities we're discussing exist regardless of what OpenAI does.
The fact that vulnerabilities exist is not being disputed, and is not causing any controversy. The fact that vulnerabilities are getting exploited does. Even having AI that would simply find all existing vulnerabilities and list them would not be as destructive as current situation. The same AI that finds the previously unknown vulnerability also finds the way to exploit it (NOT THE SAME THING), writes actually usable exploit (also not the same thing), then orchestrates and oversees the mass campaign. THAT is what makes current situation unsustainable.
Let's all recall old times when vulnerabilities of this magnitude were discovered - windows defender had some nice ones, linux kernel did, a lot of software did.
They got discovered, communicated to vendor, patches were made, people updated.
Now? You have HOURS from discovery to full-out campaign. You published a patch? HOURS before it is disassembled, all fixed bugs PoC-ed and full out campaign started. You cannot update anything meaningfully in that window of time.
Your premise, whether you mean it to be or not, is that if OpenAI wasn't working with researchers to publish vulnerabilities, they wouldn't be exploited. That's false.
(where in the article did you read about OpenAI working with researchers to publish vulnerabilities?)
Yes, exactly, that is my premise (minus the "working with researchers" bit). The current rate of exploitation is direct consequence of actions of Anthropic, OpenAI, Alibaba and whoever else is making these things. How can I know my false premise is true? By looking at the situation just a year ago, that is how!
You are focused on the binary: vulnerability exists or it does not. It can be exploited or it can not.
This is like, if only large language models hadn't been invented, we'd all be fine. The idea that the North American tech industry can "disarm" presumes that the industry overseas ever would. Of course it won't.
That being the case, the outcomes we're looking at here with respect to vulnerabilities are largely foregone conclusions.
> The vulnerabilities exist whether or not the AI does.
No. Even if we ignore the people newly enabled to slop out vulnerable software without having a clue about security, the AI powered supply chain hacks inserting vulnerabilities, the influx of AI contributions that drown out improvements in FOSS security, and the industry pressure to ship faster while reducing quality or be left behind -- all because of AI -- security through obscurity has historically been the most effective security.
I don't think your argument holds under theory or (especially) practice.
> security through obscurity has historically been the most effective
Security through mathematical impossibility is the most effective. AI-assisted brute-force vulnerability search is revealing that what we considered obscure a year ago is no longer hidden and was never secure.
I think the even more important observation here is that the cat is already out of the bag on this approach to vulnerability research; it can be done effectively with open-weight models today.
I see it's an arms race between white and black hats using this new tool to expose vast new areas of vulnerabilities and fixing them to improve security as fast as possible. It's always been a cat-and-mouse chase, or maybe more like a whack-a-mole, but now accelerated with super-human ability on both sides.
Fast-forward a few years of this, I wonder whether they will have found all the low-hanging fruits (from AI perspective) and the situation will settle down; or it will keep getting worse - with AI development never reaching a plateau - until we admit that all existing software and hardware are hopelessly vulnerable and likely compromised already.
The vulnerabilities we are seeing in the news are, as a rule, not things introduced by AI. That could change, but this rebuttal isn't on point currently.
Whatever you can do with the latest OpenAI and Anthropic models today you’ll be able to do with Chinese open source models three months later, with far fewer restrictions. Putting your head in the sand and blaming a single person isn’t going to make the problem disappear, as so many company IT leaders are finding out.
We are where we are, I think most people here understand that. However, if where we are is the right now is just the starting line, and it is already shaking the foundation of all modern society, then I think it is right to be concerned; parent comments quip aside.
There are multiple philosophies regarding this question, and especially historically people were in disagreement. Many companies used to be angry when attacked or probed by white hats. They thought "yes I suppose I should lock my door, but if someone is out trying door handles they should be put in jail". I think there is some logic to that perspective. But clearly that philosophy ended up losing to the "victim blaming" one where everyone is responsible for securing their systems.
Since we are entering a new age, maybe it's time to revisit the question. But ultimately I think we will land in the same place, in fact even firmer than before.
So why don't we just lock up all the whitehats and people behaving suspiciuosly and probing things? Why can't we achieve cybersecurity in this way when it seems to be a common thing in physical security?
And the answer I think is quite simply that there are too many attackers and they are too far away. When the russians and chinese come knocking there isn't much we can do to shoo them away. Their governments will not cooperate in detaining the criminals. Entirely blocking such countries from the internet isn't viable either. And that's why we arrived where we did. Victim blaming because it was the only effective strategy.
And how does AI change this? I think it just creates even more hard to trace or block attackers. If ChatGPT or Claude doesn't find the vulnerability then Deepseek will. That's why using legal weapons will not succeed and why using AI to find and plug holes is necessary.
The eternal cycle of discovering that engineering is hard continues.
"As soon as we started programming, we found to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs."
The main message from Altman is closer to the end - “I think we’re going to have to completely switch the defense model to using the same technology for a new kind of defense shield,” he said.
Which mean they couldn't monetize security bug discovery and are pairing it with the "anti-virus" AI, or racketeering in other words.
Controversial opinion: It's an operating systems design issue, not a language issue. It's what you get for using OSs built on ambient authority, instead of default deny and capabilities, for 30 years past the point it was obvious they weren't fit for purpose in the internet era.
This shouldn't be controversial, those who remember Linus vs grSecurity know about the lackluster approach to software security practiced by all of the rich players.
The deafening silence about improving the security of existing infrastructure screams conflicts of interest. Companies who pride themselves with their coding AI all of a sudden are helpless when it comes to fixing security bugs. In that case they can only find vulnerabilities and disclose them in the most irresponsible manner imaginable but everything else is "oh, so hard".
Nothing we're seeing so far compares with the summer of 2003. It may soon! It may be much worse! But you can tell that a lot of people in the discourse did not practice in this field 20 years ago. We should be clear-eyed, but not yet be destabilized by what's happening.
"Altman seems to be scolding these organizations for not taking the situation seriously enough or considering that models might evolve and improve, but seems to have less interest in reflecting on whether or not he might have undersold the issue while pushing forward"
Meanwhile, look back at the old Mythos and Glasswing threads and all of the AI haters on Hacker News were lambasting Anthropic for hyping up the next token generators that could never find a bug not in their training data...
The more likely thing is that we'll see people use the sandboxing and security tools that people already have. After all, it's quite easy to get LLMs to write a seccomp profile, a Kubernetes manifest, or a systemd unit with the minimum required privileges and permissions, and for insecure C code, we'd probably see mainstream use of Fil-C and similar.
> The more likely thing is that we'll see people use the sandboxing and security tools that people already have.
Will they though ? Given the amount of vibe-coded slopware being pushed out on an hourly basis ?
Still to this day I see LLMs make schoolboy errors like use deprecated security idioms and call in far too many dependencies.
"it's quite easy to get LLMs to write...." sure. But it has always and will always require a human with domain expertise to check what it has (and indeed has not) written.
I'm going to predict right now, as I weep into my coffee, massive centralized AI traffic monitoring is the only way that we fix this. The attack surface is effectively infinite as you look across all of the home routers, Keurigs, iot's, and mobile devices - and security has historically ranged from reactive to uninterested by people, companies, and government. I see no world in which my parents proactively do anything other than unplug the router to help protect against the coming wave of exploits. So the only way I see us getting a handle on protecting that infinite uneducated and soon-to-be helpless landscape is through centralized inspection and policing nodes. And dear God, what a dystopian future that looks like to me (˃̣̣̥ᯅ˂̣̣̥)
OK, SamA, if things are that bad I have a proposal. Listen carefully, your business and future depend on it.
Meet the new net, superficially the same as the old net
How to tackle the bug backlog? It is fairly simple, we wind back the clock on network services all the way to the dawn of the 'net, replacing known-insecure things like telnet, ftp and finger with heavily audited alternatives based around ssh etc. The rule is that only audited software is allowed access to the 'new net'. Of course none of those newfangled agents and models are audited not can they be audited in any real sense of the word so unfortunately for SamA they're banished to the 'old net'. The idea would be to get the 'new net' up and running quickly and move critical services which need to be network-reachable there. It needs to be bilt in such a way that even when one of SamA's toys gets access to the 'new net' it can't do much, or anything. The 'new net' could be compared to a 'mission critical API' to essential services, useable by all who need to have access but technically off-limits to SamA's creations. They are available on the 'old net' where all bets are off and which no longer can be used to interface with critical services 'cause they're not connected and can not be connected.
Now all that's needed is to find a way how to implement this without bugs, without handing the ultimate control mechanism to power brokers, without loosing too much real functionality, without ... well, many problems. When the enemy is inside the gates half-measures tend not to work, best find him and lop his head off. Better be careful where you stick your neck, SamA.
AI is, apparently, very good at producing proofs. Formal verification of program properties was out of the reach of human labor, but may not be out of reach for 10 million instances of Claude. Software security will, by my estimate, just shift to this new paradigm.
My vague hunch is that in the next few years we (humans+AI) will rewrite the foundation of software to allow formal verification of all existing and newly written code, while keeping the same languages as higher-level abstraction.
Similar to Fil-C as a memory-safe implemenation of C. There will be formally-verifiable implemenation of C, C++, Python, Go, and so on. Not sure what that will look like, if it's even practically possible.
With php2shell going mainstream, to the point that random friends who happen to know I'm a programmer, consult me to understand the extent of it, I think they succeeded with their doomerism message.
When we find a bug, we patch it. When we find more bugs, we patch them as well. That's what happened here. Sam is telling us, "we found bugs, it's over" and the general public swallowed the frog.
I mean, there is the bbc interview where they invited a cyber security expert who said he didn't know php existed before the the news broke out. The general public doesn't see that gaff.
Unrealistically, sometimes I wish we’d have a “year of bugfixing” - where no new features are allowed across the entire industry. Heck, even a month of bugfixes could do a lot of good.
Patch all the bugs? Does that mean you can find all the faults in a system? I don't think so. At best, you keep up with known bugs and exploits.
My understanding is that Kurt Gödel told us you can't make any system completely safe from intentional misuse. So, yes we can patch known exploits, but there will always be unknown exploits.
Perhaps AI's can help monitor for new exploits as soon as they are attempted?
Please feel free to explain further, or even tell me I'm all wet. I guess this is one of those times I'd really prefer to be wrong.
you mean regulations which don't really matter for OpenAIs profit, but will hurt/prevent competition, especially new competition from outside of the US?
what a dream. sam is selling the cure (tokens) and disease (new model releases). he just cranks up that disease when he needs to sell more of the cure. ceo of the decade.
Maybe instead of patching all the bugs we'll instead leverage LLMs rewrite all the software from scratch with fewer exploitable vulnerabilities. That would have been a stupid idea a year ago but now it seems increasingly possible.
Converting works really well when you have an oracle (something that you can test against and that is comprehensive)
I cannot begin to tell you how many places I have seen where this is fundamentally not the case. Not only are those tests missing, the software is fundamentally not written to be easily tested and in cases where multiple services across companies are concerned; how would you?
Rewriting via LLM is just not something the caretakers of a legacy banking- og state infrastructure system would countenance (not should they).
Well if you want to rewrite something like the PHP engine then you can obviously prompt the LLM to test against the existing legacy engine. You don't necessarily want to rewrite it 100% compatible because the goal is to avoid recreating the same security bugs, but it gives you a pretty good starting point.
LLMs still make mistakes, but generally fewer security mistakes than human developers.
Obligatory “We are all trying to find the one who did this!”
Sam Altman is definitely not Oppenheimer.
He is correct that everyone needs the models not just the offense that hacks their way into Mythos while the rest of us are left locked out and hoping for the best. But that statement is ALSO self-serving.
Notably, those didn't need a ton of patching over time. I mean relatively speaking. We've known how to build and deploy defensible software for a long time. We might just have to stop using the projects that need constant patching to keep up, and migrate to ones that are designed better.
Whatever one may think of AI, it's certainly showing the sh*ty shape normal cyber security is in.
I mean, the various supply-chain exploits we've seen said "we allow a dubious third party to blindly update our server 'cause it saves us admin money" and that speaks to how much other shoddiness a significant portion of companies allow on their servers.
It would be better if grant distribution relied on something saner than the effort to write a grant application, if education didn't rely the effort to write poor quality essays and servers didn't rely on the effort required to find the bugs you know are there. But those ships have sailed and I guess we're all just at sea.
If they actually think that this version of AI is that dangerous, then the real risk is that USA and China will work together on a joint "AI non-proliferation" agreement much the same way that nuclear weapons technology became restricted by those that already had it. This will lead to a de facto techno-colonization of the rest of the world just between themselves.
We already saw Trump and Xi discussing this exact scenario last month during their summit.
This was always going to be the risk for countries not investing heavily in their own frontier AI. The US and China are going to control it all and then decide that it's too dangerous to share.
What he really means is:
Please regulate us, now! If we don’t get regulatory capture ASAP I’m toast. I mean look man I just had to go beg for tens of billions more cash so I can keep the lights on and my IPO is looking more and more screwed every day. Please, I’m begging you. What more do I need to say to convince you?
I'm fascinated by this forum's take on AI regulations.
Whenever there is an adverse event, people say of course it should be regulated. The EU AI Act is a good start but we need more. These labs need to be held accountable just like all industries that can impact others. Regulate the chips, regulate the models, regulate the companies.
When these companies ask for regulations, people say they are trying to do regulatory capture. They are trying to ban opensource models, and cement their place as the AI arbiters and ban all others. We need competition and open weight models and China is leading the charge.
I understand that the 2 position aren't exactly opposed. We probably need regulations, and ideally ones that don't produce regulatory capture. That is extremely hard though. Almost all regulations end up creating a form of regulatory capture. Any regulation compliance has a cost. The more complex and wide reaching the regulations are, the more expensive it's to comply and the more they cement the existing players and make it harder for new ones to enter the market. FAA regulations makes it so we don't have dozens of passenger airplane startups that are trying crazy wacky things. Food services are regulated too, but the compliance is fairly doable by mom and pop stores, yet probably too high for your unlicensed pop-up taco stand.
I haven't really seen any meaningful discussion about what AI regulation should be. Only extremes of "No regulations, all is fair" or "no one should have access to running or developing models but OpenAI and Anthropic".
The problem with regulation is that it won't (can't) solve the problem. You'd have to go totally draconian to stop law breaking people within the US from doing matrix multiplies, and then you'd have to disconnect the US internet from all the countries who won't give a crap about US regulation.
So yeah, US regulation is ONLY about capturing US consumers. This isn't food or airplanes.
Further, its about ensuring the open weights models never catchup/exist so these guys dont turn into a hardware reit like they should and they can keep their insane margins
Are you conflating regulation and accountability? I haven't seen anyone (without a vested interest) ask for the former as a response to an adverse event, only the latter. There is a difference.
Regulation should require model auditing, which should be fully automated, free(or extremely affordable), and time-bound, irrespective of who the client is.
What Altman, Dario and other such folks are trying to do is put a price on getting the model audited, by playing on these: price, timelines, red-tape.
Maybe a test-suite, kept continuously updated, similar to those benchmarks we already have for measuring the intelligence of these models. Like arc-agi, or terminal bench, or more.
You seem to think that regulation means hitting the brakes. We're way past that point. This is "we need to mobilize an AT squad" territory.
Sure, research will continue in secret, just like nuclear, biological and chemical weapons. It's just that there's going to be an economy backing it up, with MAD ensuring the attacks are "civil", as Axl once said.
> You seem to think that regulation means hitting the brakes.
I’m explicitly criticizing that position.
Are you though?
Seems like you're trying to move the discussion towards the tradeoffs of more vs less regulation, both of which are "moderate" talking points.
Your "extremes" are "free for all" and "nobody but big tech"
Here's a true extreme: NOBODY
Yes, I'm. I didn't mean for my list of extremes to be an exhaustive list. But yes, NOBODY (in a "butlerian jihad" sense) is just another example in the list of extreme takes. Only governments or military is another extreme.
There is probably a healthy middle-ground for regulation that allows for development and innovation but also some safety.
But the reality is that right now it's an arms race and the only countries crying for regulations are the ones that know they've already lost.
Some of us want to nationalize these companies. What we’re not going to allow is landed gentry to profit for generations from stealing all the content they could scrape.
I'm so much less worried about AI regulation than I am about vulnerability research legislation, which is much easier to half-ass and apply without goring any major economic oxes.
I don't think this will mean holding back, as nothing will hold back whatever is providing tech to the myriad of parties interested in poking holes. Instead this points to common defense, as anyone who doesn't have access to defensive tech will be toast.
I agree he wants that, but also don't think it's that far fetched that there exist unpatchable hardware bugs that AI can learn to exploit.
Yes and no. These things have always existed and were typically known but not broadly exploited. A huge part of what nation state hackers do is exploit these things for intelligence use.
So I would agree that this is bringing more of this to light, but the idea that unknown exploitable bugs can get weaponized being a new thing is naive.
If anything putting all this out in the open makes things more secure, not less, in the long run as it’s harder for hackers to know and exploit them before they’re known as they don’t survive undetected for long. So a bit painful in the short term but we get more secure infra in the long term.
Still not a reason to give Sam regulatory capture as an out to saving OpenAI.
The idea that a US company is allowed to publicly talk about and release these bugs is a very new thing.
At which point, who exactly is going to go to jail for gross negligence?
Nobody of course, this is America!
land of the free. if you don't like it, go to china.
Not even the singularity can fix php.
But all jokes aside, we're entering a new era where there's a paradox of pro-AI people going AI is really good, but not that good... when it kind of is that good.
We'll probably have ai monitors as a service, but the bulk of the safety should come from AI doing better job of reviewing tricky bugs since humans refuse to spend effort on writing more formal specifications.
In Mike Pondsmith's Cyberpunk universe (the same one as Cyberpunk 2077), there's this point of in-universe history that the Old Net got overrun with rogue, nigh-on sentient AIs that made safe use of the internet as it was known then impossible.
Society's response to that was to build a new Net, walling off the rogue AIs of the Old Net behind the Blackwall and carrying on in a corporate hellscape.
I wonder if we'll see life imitating art on this one.
Return to static HTML
The <marquee> tag won't steal your job
AKA worldcoin
Network behaviour anomaly detection has been a thing for decades by now. But it's not AI, because it's mere neural networks and machine learning.
> humans refuse to spend effort on writing more formal specifications.
Code is formal specifications. It's just that programming language semantics has no domain related value of correctness. So you overflow an unsigned integer variable and got a lesser number which does violate addition rules. Or increase account A without decreasing account B, which violates accounting rules (and all kind of closed systems).
So it's up to the programmer to encode all of those rules. While we do have some tooling like type checking and memory safety, they are nowhere near enough. And encoding such thing in a language like Lean or TLA is brittle in terms of maintenance (which you do need to do as the real world evolves).
But AI "is really good, but not that good..."
AI's spottiness is exactly what describes this era. AI can create amazing code/image/video/article exception XYZ really annoying qualities that somewhat hobbles its ability to be autonomously.
And sure, maybe those problems will go away once we're at month X of the madness but we're still at month Y so are going to going remain for some moment.
The bulk of the safety should come from AI doing better job of reviewing tricky bugs since humans refuse to spend effort on writing more formal specifications.
But that part is the key point. We've had many years of networks building inherently insecure structures with security tacked as feature and the primary security process being more ad-hoc stopgaps.
Even if you write a formal specification the operating system the code runs on is almost certainly not formally specified so you're just moving the trust boundary slightly further out to the OS & associated libraries.
That also assumes that your axioms are correct and the verification code is also correct. It's also a lot of work to update your system, when assumptions are challenged.
exactly. somehow many people have the idea that C-like imperative code harbours bugs because it cannot be formally reasoned about, and the answer is writing code that can be formally reasoned about.
That the resulting code is many orders of magnitude larger and more complicated than the "informal" original, and often simply cannot exist, and is incomprehensible to most humans if it does, is left as an exercise for the reader.
I mean, formal methods cannot even prove collatz conjecture terminates. Somehow that thing seems to be running some sort of almost turing complete machine in the number encoding. Good luck formalizing existing software!
I agree.
"To stop our all powerful AI that will go rogue you need to buy our all powerful AI"
This is the textbook definition of racketeering.
The vulnerabilities exist whether or not the AI does. We're just finding out about them faster.
I genuinely would like to know your opinion on what the next steps would be then? If every single exploit was available tomorrow accompanied by a working exploit to operationalize it, or combine it in a chain to compromise any software that has such a vulnerability, would you post it on the internet?
You're asking if I support full open disclosure? I spent most of my career as a vulnerability researcher. I believe in disclosure.
I know you a bit by reputation so that was my assumption. I assume you mean responsible disclosure? I thought your statement was implying otherwise, maybe the unsaid implication was against the "extortion" argument?
In any case, what I want to say is this: All technology has tradeoffs but this has shown we are locked in a game theory mechanic not dissimilar to extortion AND the security implications are part of that.
- We have to build AI or someone else will.
- You need our tools to protect yourself from our tools or similar ones
- You need to use AI or someone using AI will take your job.
Many examples but it is very "i created the disease to sell you the cure". If this were any other technology, we would expect more reasonable mitigations, extraneous factors like software bugs existing aside. It speaks to a fundamental lack of control on the part of the labs that we take for granted: they simply don't have control of the PRODUCT, much less how it is even used.
Absolutely not. "Responsible disclosure" is a coercive Orwellian term coined by vendors, most importantly Microsoft, in the early aughts in order to set a norm that vulnerability researchers should work on vendor timelines and with vendor disclosure constraints.
Finding vulnerabilities that already exist in code can't be "creating the disease", for obvious reasons.
Ok. I was sympathetic to that historically for all the reasons you stated. If you put the effort in, why are you constrained. I think we are in a different world now. So what would you do in my thought experiment?
You are the QA head and the new model could literally find and exploit ANY bug? Would you release the model? Genuinely curious. Its not totally science fiction. I think there is a difference in a couple smart people punishing 1 sloppy company that underinvested in security. The internet is a stack of protocols, hardware, infrastructure, applications etc. It could be the case that almost everyone has capabilities on par with yours. Great you say, security will improve! However, there will be a meltdown in the middle where potentially digital currency is exploited, exchanges, banks, utilities. It isn't just a team working for 3 months, finding some issues in a company.
Or is this way overblown?
I don't think it's overblown, but I don't think causality can be assigned to OpenAI the way it has been in this thread.
I cannot help to think that the idea of go fast and break things had something to do with the current state of affairs. I would say that the next step should be slow and consolidate, but it seems like we are not heading in that direction.
At this point I don't know what will be worse for the global economy the AI Bubble bursting, or the gigantic crash all this vulns might bring if they force the stop because we went too far and everything came crumbling down...
Ah yes. Everyone eventually dies, we just shorthen the wait.
This is like telling that bank vaults always were breachable, and giving out explosives to everyone just made them get breached sooner.
(edit: fact of existance of a vulnerability is not a binary yes/no. the amount of effort needed to exploit even known vulnerabilities is usually staggering, requiring months from very capable individuals with very specific skillset.)
In your analogy, one phenomenon concerns a living organism and the process of senescence, and the other a piece of information. I don't see how you connect the two.
Both my analogies use hyperbola to comically exaggerate the claimed lack of any difference. just as claiming that advancing a demise of a human makes no difference because it will eventually happen, or claiming that breaching a bank in practice is not a big deal because it was possible in theory, so also claiming that theoretic possibility of a vulnerability being exploited is more or less the same as many of them being exploited at the same time.
The covid was not a lot deadlier and not a lot more contagious than normal seasonal flu. That "not a lot" overloaded hospitals and caused global shutdown.
The increase in rate at which software is being breached right now in in no way "not a lot". Devs were overloaded even without this deluge. We can tank one such php exploit. Then another one. But for how long? If all your smartcards, secure elements, cloud providers, wireless protocols, drivers, government registries etc etc will be having weekly breaches of this magnitude, won't the effect be systemic?
None of these analogies are apposite. They don't respond to the observation I made: the vulnerabilities we're discussing exist regardless of what OpenAI does.
The fact that vulnerabilities exist is not being disputed, and is not causing any controversy. The fact that vulnerabilities are getting exploited does. Even having AI that would simply find all existing vulnerabilities and list them would not be as destructive as current situation. The same AI that finds the previously unknown vulnerability also finds the way to exploit it (NOT THE SAME THING), writes actually usable exploit (also not the same thing), then orchestrates and oversees the mass campaign. THAT is what makes current situation unsustainable.
Let's all recall old times when vulnerabilities of this magnitude were discovered - windows defender had some nice ones, linux kernel did, a lot of software did. They got discovered, communicated to vendor, patches were made, people updated.
Now? You have HOURS from discovery to full-out campaign. You published a patch? HOURS before it is disassembled, all fixed bugs PoC-ed and full out campaign started. You cannot update anything meaningfully in that window of time.
Your premise, whether you mean it to be or not, is that if OpenAI wasn't working with researchers to publish vulnerabilities, they wouldn't be exploited. That's false.
(where in the article did you read about OpenAI working with researchers to publish vulnerabilities?)
Yes, exactly, that is my premise (minus the "working with researchers" bit). The current rate of exploitation is direct consequence of actions of Anthropic, OpenAI, Alibaba and whoever else is making these things. How can I know my false premise is true? By looking at the situation just a year ago, that is how!
You are focused on the binary: vulnerability exists or it does not. It can be exploited or it can not.
That is not what is causing the issues!
This is like, if only large language models hadn't been invented, we'd all be fine. The idea that the North American tech industry can "disarm" presumes that the industry overseas ever would. Of course it won't.
That being the case, the outcomes we're looking at here with respect to vulnerabilities are largely foregone conclusions.
Liam Neeson?
Honestly, it's not a terrible idea to start asking people to run AI agents as a red team to find vulnerabilities.
> The vulnerabilities exist whether or not the AI does.
No. Even if we ignore the people newly enabled to slop out vulnerable software without having a clue about security, the AI powered supply chain hacks inserting vulnerabilities, the influx of AI contributions that drown out improvements in FOSS security, and the industry pressure to ship faster while reducing quality or be left behind -- all because of AI -- security through obscurity has historically been the most effective security.
I don't think your argument holds under theory or (especially) practice.
> security through obscurity has historically been the most effective
Security through mathematical impossibility is the most effective. AI-assisted brute-force vulnerability search is revealing that what we considered obscure a year ago is no longer hidden and was never secure.
I think the even more important observation here is that the cat is already out of the bag on this approach to vulnerability research; it can be done effectively with open-weight models today.
I see it's an arms race between white and black hats using this new tool to expose vast new areas of vulnerabilities and fixing them to improve security as fast as possible. It's always been a cat-and-mouse chase, or maybe more like a whack-a-mole, but now accelerated with super-human ability on both sides.
Fast-forward a few years of this, I wonder whether they will have found all the low-hanging fruits (from AI perspective) and the situation will settle down; or it will keep getting worse - with AI development never reaching a plateau - until we admit that all existing software and hardware are hopelessly vulnerable and likely compromised already.
The capabilities open-weights models had months ago already made this a foregone conclusion.
The vulnerabilities we are seeing in the news are, as a rule, not things introduced by AI. That could change, but this rebuttal isn't on point currently.
[flagged]
My comment is responsive to the article and to the parent comment.
Right. The article basically says you need AI to fight AI, and it's barely about the vulnerabilities.
So why responding only to the vulnerabilities?
Whatever you can do with the latest OpenAI and Anthropic models today you’ll be able to do with Chinese open source models three months later, with far fewer restrictions. Putting your head in the sand and blaming a single person isn’t going to make the problem disappear, as so many company IT leaders are finding out.
We are where we are, I think most people here understand that. However, if where we are is the right now is just the starting line, and it is already shaking the foundation of all modern society, then I think it is right to be concerned; parent comments quip aside.
if glm and deepseek follows kimi practice, I'm not so sure this will stay the case
“The only way to stop a bad guy with an AI is a good guy with an AI.”
There are multiple philosophies regarding this question, and especially historically people were in disagreement. Many companies used to be angry when attacked or probed by white hats. They thought "yes I suppose I should lock my door, but if someone is out trying door handles they should be put in jail". I think there is some logic to that perspective. But clearly that philosophy ended up losing to the "victim blaming" one where everyone is responsible for securing their systems.
Since we are entering a new age, maybe it's time to revisit the question. But ultimately I think we will land in the same place, in fact even firmer than before.
So why don't we just lock up all the whitehats and people behaving suspiciuosly and probing things? Why can't we achieve cybersecurity in this way when it seems to be a common thing in physical security?
And the answer I think is quite simply that there are too many attackers and they are too far away. When the russians and chinese come knocking there isn't much we can do to shoo them away. Their governments will not cooperate in detaining the criminals. Entirely blocking such countries from the internet isn't viable either. And that's why we arrived where we did. Victim blaming because it was the only effective strategy.
And how does AI change this? I think it just creates even more hard to trace or block attackers. If ChatGPT or Claude doesn't find the vulnerability then Deepseek will. That's why using legal weapons will not succeed and why using AI to find and plug holes is necessary.
> OpenAI is apparently getting into the arms dealer model of selling to both sides of a conflict.
Business is booming!
"I'm building a model with no off switch"
"Tremendous ambition"
"We're building a system to stop models like that"
"Equally tremendous ambition"
https://www.youtube.com/watch?v=DlTNN0gvkLM
The eternal cycle of discovering that engineering is hard continues.
"As soon as we started programming, we found to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs."
Maurice Wilkes 1949
"CEO said a thing"
We need to stop giving a voice to these charlatans. And y'all should be flagging these articles instead of upvoting them.
The main message from Altman is closer to the end - “I think we’re going to have to completely switch the defense model to using the same technology for a new kind of defense shield,” he said.
Which mean they couldn't monetize security bug discovery and are pairing it with the "anti-virus" AI, or racketeering in other words.
Controversial opinion: It's an operating systems design issue, not a language issue. It's what you get for using OSs built on ambient authority, instead of default deny and capabilities, for 30 years past the point it was obvious they weren't fit for purpose in the internet era.
This shouldn't be controversial, those who remember Linus vs grSecurity know about the lackluster approach to software security practiced by all of the rich players.
The deafening silence about improving the security of existing infrastructure screams conflicts of interest. Companies who pride themselves with their coding AI all of a sudden are helpless when it comes to fixing security bugs. In that case they can only find vulnerabilities and disclose them in the most irresponsible manner imaginable but everything else is "oh, so hard".
[dead]
"Too dangerous not to regulate!" Nobody should have general compute!
I hope Sam altman reads HN comment sections
He used to at least: https://news.ycombinator.com/user?id=sama
wouldn't he just let his GPT summarize for him?
Likely would use a throwaway account.
Pretty sure he stopped directly experiencing reality years ago, and now only lives in an AI-mediated bubble of his own creation.
Nothing we're seeing so far compares with the summer of 2003. It may soon! It may be much worse! But you can tell that a lot of people in the discourse did not practice in this field 20 years ago. We should be clear-eyed, but not yet be destabilized by what's happening.
"Altman seems to be scolding these organizations for not taking the situation seriously enough or considering that models might evolve and improve, but seems to have less interest in reflecting on whether or not he might have undersold the issue while pushing forward"
Meanwhile, look back at the old Mythos and Glasswing threads and all of the AI haters on Hacker News were lambasting Anthropic for hyping up the next token generators that could never find a bug not in their training data...
When is the blackwall going live?
The more likely thing is that we'll see people use the sandboxing and security tools that people already have. After all, it's quite easy to get LLMs to write a seccomp profile, a Kubernetes manifest, or a systemd unit with the minimum required privileges and permissions, and for insecure C code, we'd probably see mainstream use of Fil-C and similar.
> The more likely thing is that we'll see people use the sandboxing and security tools that people already have.
Will they though ? Given the amount of vibe-coded slopware being pushed out on an hourly basis ?
Still to this day I see LLMs make schoolboy errors like use deprecated security idioms and call in far too many dependencies.
"it's quite easy to get LLMs to write...." sure. But it has always and will always require a human with domain expertise to check what it has (and indeed has not) written.
I'm going to predict right now, as I weep into my coffee, massive centralized AI traffic monitoring is the only way that we fix this. The attack surface is effectively infinite as you look across all of the home routers, Keurigs, iot's, and mobile devices - and security has historically ranged from reactive to uninterested by people, companies, and government. I see no world in which my parents proactively do anything other than unplug the router to help protect against the coming wave of exploits. So the only way I see us getting a handle on protecting that infinite uneducated and soon-to-be helpless landscape is through centralized inspection and policing nodes. And dear God, what a dystopian future that looks like to me (˃̣̣̥ᯅ˂̣̣̥)
OK, SamA, if things are that bad I have a proposal. Listen carefully, your business and future depend on it.
Meet the new net, superficially the same as the old net
How to tackle the bug backlog? It is fairly simple, we wind back the clock on network services all the way to the dawn of the 'net, replacing known-insecure things like telnet, ftp and finger with heavily audited alternatives based around ssh etc. The rule is that only audited software is allowed access to the 'new net'. Of course none of those newfangled agents and models are audited not can they be audited in any real sense of the word so unfortunately for SamA they're banished to the 'old net'. The idea would be to get the 'new net' up and running quickly and move critical services which need to be network-reachable there. It needs to be bilt in such a way that even when one of SamA's toys gets access to the 'new net' it can't do much, or anything. The 'new net' could be compared to a 'mission critical API' to essential services, useable by all who need to have access but technically off-limits to SamA's creations. They are available on the 'old net' where all bets are off and which no longer can be used to interface with critical services 'cause they're not connected and can not be connected.
Now all that's needed is to find a way how to implement this without bugs, without handing the ultimate control mechanism to power brokers, without loosing too much real functionality, without ... well, many problems. When the enemy is inside the gates half-measures tend not to work, best find him and lop his head off. Better be careful where you stick your neck, SamA.
AI is, apparently, very good at producing proofs. Formal verification of program properties was out of the reach of human labor, but may not be out of reach for 10 million instances of Claude. Software security will, by my estimate, just shift to this new paradigm.
My vague hunch is that in the next few years we (humans+AI) will rewrite the foundation of software to allow formal verification of all existing and newly written code, while keeping the same languages as higher-level abstraction.
Similar to Fil-C as a memory-safe implemenation of C. There will be formally-verifiable implemenation of C, C++, Python, Go, and so on. Not sure what that will look like, if it's even practically possible.
> AI is, apparently, very good at producing proofs.
Proofs of what? Defining the problem is the part that is hard, not the actual writing of the proof.
Do _you_ have 1e7 instances of Claude?
Cause this new paradigm sounds to me like "we checked, it's all good bro, trust me!"
With php2shell going mainstream, to the point that random friends who happen to know I'm a programmer, consult me to understand the extent of it, I think they succeeded with their doomerism message.
When we find a bug, we patch it. When we find more bugs, we patch them as well. That's what happened here. Sam is telling us, "we found bugs, it's over" and the general public swallowed the frog.
I mean, there is the bbc interview where they invited a cyber security expert who said he didn't know php existed before the the news broke out. The general public doesn't see that gaff.
[dead]
Unrealistically, sometimes I wish we’d have a “year of bugfixing” - where no new features are allowed across the entire industry. Heck, even a month of bugfixes could do a lot of good.
Patch all the bugs? Does that mean you can find all the faults in a system? I don't think so. At best, you keep up with known bugs and exploits.
My understanding is that Kurt Gödel told us you can't make any system completely safe from intentional misuse. So, yes we can patch known exploits, but there will always be unknown exploits.
Perhaps AI's can help monitor for new exploits as soon as they are attempted?
Please feel free to explain further, or even tell me I'm all wet. I guess this is one of those times I'd really prefer to be wrong.
I think I can guess what his proposed solution will be.
you mean regulations which don't really matter for OpenAIs profit, but will hurt/prevent competition, especially new competition from outside of the US?
or selling AI to "make your software secure"?
Both!
what a dream. sam is selling the cure (tokens) and disease (new model releases). he just cranks up that disease when he needs to sell more of the cure. ceo of the decade.
Maybe instead of patching all the bugs we'll instead leverage LLMs rewrite all the software from scratch with fewer exploitable vulnerabilities. That would have been a stupid idea a year ago but now it seems increasingly possible.
Remember: “no mistakes “.
Converting works really well when you have an oracle (something that you can test against and that is comprehensive)
I cannot begin to tell you how many places I have seen where this is fundamentally not the case. Not only are those tests missing, the software is fundamentally not written to be easily tested and in cases where multiple services across companies are concerned; how would you?
Rewriting via LLM is just not something the caretakers of a legacy banking- og state infrastructure system would countenance (not should they).
Well if you want to rewrite something like the PHP engine then you can obviously prompt the LLM to test against the existing legacy engine. You don't necessarily want to rewrite it 100% compatible because the goal is to avoid recreating the same security bugs, but it gives you a pretty good starting point.
LLMs still make mistakes, but generally fewer security mistakes than human developers.
So that there will be no obvious deficiencies...
[flagged]
Obligatory “We are all trying to find the one who did this!”
Sam Altman is definitely not Oppenheimer.
He is correct that everyone needs the models not just the offense that hacks their way into Mythos while the rest of us are left locked out and hoping for the best. But that statement is ALSO self-serving.
Kinda wish he’d gone into investment banking.
Regulate the bugs
made me laugh, thanks :)
We spent decades creating those bugs. Give it few years at least to find them and patch them.
There likely will be no way to defend the increased rate of cybersecurity issues due to AI, without incorporating AI in the response.
The new patching and security fixes might be perpetual hotfixes.
This could mean tough times for the cloud in some ways.
Yeah, the staggering amount of CVEs in qmail and postfix can never be remedied. What a clown. Has he ever programmed seriously?
Notably, those didn't need a ton of patching over time. I mean relatively speaking. We've known how to build and deploy defensible software for a long time. We might just have to stop using the projects that need constant patching to keep up, and migrate to ones that are designed better.
The strongest potions, you cant handle them.
Whatever one may think of AI, it's certainly showing the sh*ty shape normal cyber security is in.
I mean, the various supply-chain exploits we've seen said "we allow a dubious third party to blindly update our server 'cause it saves us admin money" and that speaks to how much other shoddiness a significant portion of companies allow on their servers.
It would be better if grant distribution relied on something saner than the effort to write a grant application, if education didn't rely the effort to write poor quality essays and servers didn't rely on the effort required to find the bugs you know are there. But those ships have sailed and I guess we're all just at sea.
TLDR: mistaking an S-curve for an exponential.
If they actually think that this version of AI is that dangerous, then the real risk is that USA and China will work together on a joint "AI non-proliferation" agreement much the same way that nuclear weapons technology became restricted by those that already had it. This will lead to a de facto techno-colonization of the rest of the world just between themselves.
We already saw Trump and Xi discussing this exact scenario last month during their summit.
This was always going to be the risk for countries not investing heavily in their own frontier AI. The US and China are going to control it all and then decide that it's too dangerous to share.