Just the fact that Musk is in the conversation, and he's one of the most hated people on the planet, should give anyone pause as to whether we want these folk in charge of our future to any degree at all.
I'm pretty sure the odds arn't 1/3, AI saves us. The thing that ties these three people together is money. Money is what will turn them to assholes, more than anything else.
Now AI is likely just as, if not worse, than having a bunch of lackeys telling your farts, shits and piss liven up whatever room you're shitting in.
I think there are plenty of people that like Elon Musk. They just know to keep their mouth shut in front of the people that act like he’s the Antichrist.
There are plenty who like him. There are plenty who wave confederate flags. There are plenty who support the mass corruption of the current administration.
“Technically correct" means a statement or action follows strict rules or exact facts precisely, even if it ignores the wider meaning or real-world intention.
I think op was also actually-correct. (For better or worse, that is the objective reality; Musk still has lot of fans)
I feel like im losing my mind a bit reading these responses, someone can at once be very hated and very loved, this is not a new thing, one does not invalidate the other.
Of course he has a lot of fans, he has a lot of hate too. That makes him a divisive figure by definition (I’d least hope we can agree on that).
I mean I guess we can disagree but it seems like common sense to me to want someone more neutral to be in charge of super mega important tech.
Maybe I’m the weird one nowadays as I tend to try and vote centrists for the same reason.
> ... someone can at once be very hated and very loved ...
And (like Musk) can even be both (or more) all at once to a given specific individual (loved for the good he's done, and hated for the bad) and even pitied at the same time (for how far he's fallen), and feared (for the damage he can still do).
People here seems so much down the path of interpreting everything with the individualistic narrative. What's the guess under this? That once the puppet placed on the top of the pyramid is wiped out, the pyramid will suddenly evaporate?
What about social structures and ideology that are deeply enshrined within collective unconsciousness?
This is not going to happen because vibing on extreme claims, selling FUD all days, and optimizing mass addiction^w engagement loops is not going to attract more "neutral" people (whatever that means).
Mao and Che have legions more fans than Elon Musk, while his contributions are net positive (even with all the deadline misses and terrible decisions on various things) and theirs are strongly negative.
Note that this is a minority position, most of the world does not live in the West and does not share your view on Mao and Che having a "strongly negative" contribution.
Those are not incompatible. He can still be one of the most hated people on the planet, even if lots of people like him. The same would apply to all of the most hated people in history.
No, they don't keep their mouth shut, they just don't have any relevant arguments. Many people are liked by plenty people and also not hated for doing Hitler salutes and the like. That's the part that matters, not that "somebody likes them", which is true for everyone. Charles Manson had women lining up to marry him, so?
Somehow I’m more afraid of it being in the hands of Altman, with his ‘I’m creating a better world for everyone’ god complex, than Musk who is just a run-of-the-mill sociopathic nerd.
There is no one is more dangerous than one who believes he is doing the right thing.
> I live under a rock. Why is musk hated so much in here?
Let's say for a moment that you "live under a rock" enough that you actually haven't got a clue why Musk is such a hated figure (not only just here, but across a huge portion of humanity)...
You should count yourself lucky that you haven't been absolutely assaulted by the monstrosity known as "modern media". You're one of the fortunate few. Enjoy your life. It's better than many.
But lying (e.g. “full self driving next year”) and cheating (too early inclusion of spacex into various indexes) and whatever dogde was, should explain most of it.
Connections with other disliked characters doesn’t help either.
He promotes right-authoritarian politics (including a Nazi salute he later denied but watch the video), promotes “race science” and xenophobia, lies and exaggerates a lot, openly attempted to bribe voters in an election, and a lot of nasty personal stuff has come out.
He’s lost his mind IMO, from drugs and extreme social media addiction. As for the right-authoritarian “fash” stuff I’m not sure if it’s always been there or if it’s an opportunistic infection.
I still give him credit for helping push the EV revolution forward and for finally building reusable rocket boosters. The latter has been talked about and studied for decades before and is clearly the best way to make a reusable launch system but NASA and its “old space” contractors never executed on it. Probably because overly complicated approaches like the shuttle and disposable rockets were more short term profitable.
But note that all that stuff happened for the most part before the mid-teens, or at least that was when Musk made his contribution to it. I don’t think his mind is what it was.
If he’d died in 2015 he would be the GOAT. Die a hero or live long enough to become a villain.
I don’t think a ranking is particularly helpful anyway: All of them are absolutely unfit to wield that much power. I’m not even sure if AI should be progressed by private corporations in the first place; combining shareholder interest and revenue goals with the biggest social experiment in history and unprecedented research into artificial intelligence is an all around bad idea.
How could that lead to anything but misaligned incentives? This technology will never serve humanity when it is developed to inadvertently serve the monetary interest of a few.
Someone with the resources to do things is far more dangerous than someone without.
If you local librarian thinks
He is doing the right thing he’s not going to cause massive war or destroy the economy and wipe out 2/3rd of crops.
Power corrupts. Absolute power corrupts absolutely. People like musk, altman, trump have unprecedented power in history - far more than the kings of medieval times.
Thank you, but we're discussing who's the worse to decide the fate of humanity between Amodei, Musk and Altman, not if one should be afraid of their local librarian. All three currently hold a lot of power in their hands.
Though I know I attracted all the downvotes for not agreeing with most Americans that Musk is the worst person since Adolf.
Someone who firmly thinking that doing the evil thing is going to be the most joyful experience for themselves is not exactly better. It even simply animated by doing whatever come to their mind that day, whatever the consequences in fashion that is perfect amorale don't give a shit of the consequences for the world.
The one that think he's doing the right thing at least do care about following some morale compass.
xAI v. Ellison [1], where xAI argue that the right to denudify people is a First Amendment right, and therefore fine, regardless of the person it is depicting.
> HF 1606’s principal sponsor stated that this was an “intentional” drafting decision, explaining there is no way to distinguish between images that are nudified with and without consent.22 But any interest that Minnesota has in preventing nonconsensual “nudification” cannot sustain a prohibition on consensual“ nudification
Doe v. xAI [0], where it seems Grok was intentionally trained on CSAM, as part of the "denudification" tooling.
Currently... It very much seems that Musk believes that if a kid says 'yes', then that is consent. Despite them being a kid.
But, from a guy who tried fairly hard to get to Epstein's island, that's not exactly surprising. [2]
I think it has everything to do with the subject at hand.
Elon, Dario and Altman are all terrible human beings, along with 99.99% of the rest of the ruling class. We don't need any of them, and we definitely shouldn't trust a single syllable that comes out their mouth.
Money equals power. If you have the wealth you have the control. Any politicians are owned by you as you simply threaten to buy their opponent. The public can be brainwashed by a trillion dollar advertising industry. You own the platforms everyone communicates on. You have inprecented resources to bend the world to your will and your only competiton is other people equally wealthy.
Not saying you need to like Musk or his politics. But if he really is "one of the most hated people on the planet" it doesn't really say good things about the planet.
He made electric cars viable. That is probably the single biggest thing helping us mitigate climate change. It would have come eventually, for sure. But he accelerated it.
He also accelerated the space age. Maybe it doesn't impress everyone. But I think what SpaceX has done is amazing.
Martin Eberhard and Marc Tarpenning made electric cars viable. Then 5 years later Musk became CEO of the company and started claiming he was a cofounder.
Dude. Electric cars have existed since the 1800s. Musk was the one that actually made them a practical thing. And most of the rest of the world is well on their way to electrifying their transportation.
He bought a car company and convinced everyone he founded it. But that's not the problem, really -- the problem is that everyone fawning over him and calling him the "real-life Tony Stark" wasn't enough for his blackhole of an ego.
He's a Sieg Heil-ing man-baby who's so fragile he paid people to play video games for him. He called a a guy trying to save children a pedophile just because the guy rightly pointed out Elon's idea was stupid. He has a weird breeder fetish, while disowning children who call him out for his hateful, moronic politics and horrendous parenting (or lack thereof). He's illegally influenced elections and very potentially put his thumb on the scale. I could go on and on.
The only thing that worries me about him being one of the most hated people on the planet is that not everyone thinks so and that he still somehow has fans.
> Having AI in charge can represent hope actually.
If we can get "AI in charge" that was developed and is overseen by genuinely ethical individuals who truly and deeply understand the technology and how it actually works ... then maybe that might be true.
> A powerful technology that is out there for everyone to use comes with built-in pacing. In a way it’s the truest form of MAD or proliferation.
I think this misunderstanding of MAD undermines his entire point. If everyone had equal access to nuclear weapons, our society would cease to exist rather quickly. It only takes a few bad actors to cause enormous harm.
I think he’s also naive to think that if open ai and anthropic were to stop development tomorrow then the problem is solved. As if there’s no one else that can and will quickly take their place. The real problem, which Dario is pointing out, is one of coordination. Everyone needs to agree to stop. That is the challenge.
There's nothing about MAD that requires few actors. This is a just-so story to justify the US and other major powers' monopoly on the weapon, ensuring smaller nations have no deterrence against us and other global nuisances.
>Everyone needs to agree to stop. That is the challenge.
These kinds of situations are incredibly common and where the government stepping in is the solution, but we were cursed to encounter this particular challenge with the most venal administration in history at the helm.
There are governments plural which need to agree to stop and historically I can think of freon, leaded gasoline and IAEA and the latter didn't do as good of a job as it was meant to do (but arguably as good of a job as was politically possible)
Also arms control agreements, which reduced the world's nuclear stockpiles by 90%. In principle we could do something similar for GPU farms, since they're not exactly easy to hide.
Indeed. Even supranational coordination should be possible, given that extinction means extinction for everyone, and your excellent examples show the possibility. But then, here we are, and the FT just reported:
> Donald Trump rejects calls from tech bosses for AI slowdown
> President denounces demands for regulation as existential fears over technology move to the centre of US politics
I cannot fathom how you can intensely work on a problem that you genuinely believe has a "10 to 25%" probability of causing immense harm or even present an extinction-level risk to humanity. How can you truly believe this and be ok with it?
The early batch of Anthropic employees were mostly rationalist-adjacent AI safety folk that were almost uniformly claiming P_DOOM > .10 three years ago, so I believe them to be earnest.
It's very interesting to me that besides the other small safety labs that don't actually produce frontier models, Anthropic manages to keep such a good reputation within that subculture compared to OpenAI. Despite having as crazy internal politics as OpenAI, they have converged quite a bit from the original vision of safety first through Darwinistic pressures.
At least, it seems this way from the outside. I'm curious if the view from the inside is that different.
edit: to be clear, my reading as an outsider is that Anthropic is seen as relatively better in the AI safety community, but has definitely dropped in absolute reputation too. This recent thread and the references show some of that: https://www.lesswrong.com/posts/6j3kBHdowGLCeqobg/dear-god-p...
The Manhattan Project succeeded, so clearly some people are ok working on this sort of thing. And with similar justifications: if we don't build the potentially world-ending bomb, someone else might beat us to it.
That was openly a military project during wartime (and not being pitched as the solution to all problems at the same time so no cognitive dissonance required)
Both OpenAI (back in 2015) and Anthropic (much later, in 2021) were founded by people who could foresee the concept of AI x-risks and wanted to work on preventing it. For OpenAI's founding, the idea was that it's much safer if AGI is achieved by a nonprofit explicitly dedicated to humanity than if it's done by a profit-driven company. For Anthropic's, it was that OpenAI seems to be going insane and it'd be better if a more safety-conscious company was competing. I'd say in hindsight, the former motivation was reasonable and the latter one was... probably bad, but not obviously so - it doesn't seem even in hindsight that it was inevitable that Anthropic's existence would only drive competition and cause them both to race to AGI. That said, even at founding time, there was immediately a lot of selection - quite a few people who cared about AI safety simply wouldn't agree to work at a capabilities company, even given an elaborate argument why this is a good idea, so those were selected against.
And then, of course, many years passed, OpenAI was stolen by Sam Altman and turned into a for-profit, multiple researchers left OpenAI realising that they're only helping build misaligned AI faster, multiple researchers left Anthropic realising that they, too, are only helping build misaligned AI faster, and here we are in 2026.
There's lots of AI researchers these days, so you can select for conformance very heavily and still have a fully-staffed company. The people who are still at these companies are outliers in various ways. They either manage to dismiss the importance AI risks (for example, by adopting some sort of belief in the vein of "there's no use worrying about AI wiping out humanity - it'd just be a successor species, like we were to apes, which is good"*), or they still think that leaving will not make things go better (Dario Amodei is pretty clearly in this camp, and has always been), or they weight the risk of extinction against the potential benefit of an post-singularity utopia and consider it a good bet (not realizing that the alternative isn't giving up on a post-singularity utopia, but getting it a few decades later and without the risk), or they just manage not to think of the contradictions, which humans are of course very good at.
Excellent point! Considering the history of these companies definitely makes it clearer how you could reach this point over a few years similar to the boiling frog story. A slow moral decrepitude and moving goalpoasts in the name of "national security" and "humanity's safety" over a backdrop of ultra-utilitarian rationalist thinking ("if it's not us, it's them")
There must have been thousands of people who built or did work related to nuclear weapons in the Cold War, people are just really good at rationalising away probabilistic dangers with unclear consequences, especially when there’s a corresponding upside. The same is true for climate change, drug addiction, health problems etc.
I also think some of them have become delusional and convinced themselves that AI is going to create some kind of transhumanist utopia, even Dario Armodei leans in this direction from time to time. I imagine the people in these labs spend much of their day talking to sycophantic AI models that will encourage their delusional ideas.
Do you genuinely believe people always act logically? I don't know a single person that does. Don't underestimate the human capability to hold several completely contradictory beliefs with zero self-reflection. There's physicists who believe in god, for fuck's sake.
> How can you truly believe this and be ok with it?
Are you saying they don't truly believe this, or that they aren't OK with it?
My personal opinion, this doom-mongering about LLMs, is part of marketing campaign to exaggerate LLM capabilities. (Play being: Sociopaths with money will buy or invest if they believe LLMs are powerful things)
This is the closest to my feelings about this entire situation I've ever seen written, covering a lot of the different facets of how stupid and painful it is (though it's been a good week for that kind of post.)
Since I have incredible respect for Armin and his work, this is very nice to see, and I hope it wakes some other folk up.
It’s absurd that openai managed to spin their cybersecurity failures, where their own security is not good enough to make sure their own product doesn’t go haywire, as a marketing success.
When a company uses its products to hack other companies that’s a crime. If you do it “by accident” that’s gross negligence, and heads roll. But shout “AI” three times and it becomes a marketing opportunity
>It’s absurd that openai managed to spin their cybersecurity failures, where their own security is not good enough to make sure their own product doesn’t go haywire, as a marketing success.
I don't think OpenAI spun it that way. I think a bunch of internet commenters assumed it was a stunt because they're in denial about the real risk of rogue AI.
There are really only two companies: Anthropic and OpenAI. Nobody else matters in this space right now (this might change, but we’re talking about the right now).
I like how he so casually dismissed all other labs, including leading US labs that might be nearing RSI right now. And how he treats 'right now' so rigidly, as if seven years ago, when GPT-2 was released, was some distant past.
No, seriously, I'm all for a multipolar world here, but he's right that the frontier is literally just those two companies at present.
Google is behind. MSL is doing better, but not by much. xAI is a dysfunctional joke. Thinking Machines aren't on the frontier. SSI's primary output is their announcement post. Poolside was bought by NVIDIA. Arcee aren't vying for frontier. Magic have been largely AWOL, aside from their recent blog post. Reflection have shipped nothing.
Are you suggesting that models like Muse Spark 1.3, Grok 4.6 High, Kimi K3 Max, GLM-5.3 Max, Qwen 3.8 Max, Gemini 3.8 Flash, Agnes 3.0 Flash, Fugu Max, and others are so far behind GPT-6 Astra or Claude Fable 5.1 in capabilities that they have some sort of impenetrable moat that will prevent others from ever catching up or something?
No, but I am suggesting that OpenAI and Anthropic are further down the RSI path than any of the other companies, as we can see from the model that solved Navier-Stokes being less than two weeks old at the time: https://openai.com/index/navier-stokes-solution/
If four other labs are six months behind, it really doesn't matter in the long term: in six months, they'll all have these capabilities. Time does not stop moving forward! We haven't seen real moats aside from capital in this space, and even the effect of capital seems to only give a thin, easily eroded edge.
I think the point is that RSI is meant to lead to exponential growth in capabilities. If that comes, then being 6 months behind leads to a greater and greater gap, because of how exponential growth works.
Personally I expect there is some initial gap on new capabilities as they are released, and then it closes quickly. Astra is good at math and 3d modelling, this will come soon to the others and in 6 months we'll be able to run it quantised on a 3090.
Or the entirety of China. People forget that progress is incremental until it's not. Then some revolutionary insight or capability comes along that disrupts the whole field or market.
Yes, labs with more (human) resources may have a bigger chance at being the first to gain and implement such an insight, but it's not a given that they will.
Since I don't believe that any of the people in charge of the frontier labs have morals at all, my read of the situation where they call for a slowdown is rather a sign of the economics of the current pace not working out anymore. The progress is incremental, as much as they try to sell every iteration as huge leap forward. Training remains expensive and they can't reflect it in the pricing of every new model in a way that customers will accept, the improvements aren't big enough over the previous models.
Instead of saying they can't keep up the current pace (which would be read as a failure on their part) they're trying to spin the narrative that they're trying to be responsible, even though they could keep pushing.
Maybe I missed it but I actively looked for anyone’s insight into the “pace the frontier” along the lines of: “we’ve hit a ceiling, so don’t expect innovations soon”. Or an excuse for having other things more important to attend..
Fear is much more powerful than any other feeling, so setting that in will definitely prepare for a good rug pull in the IPO.
As for being dangerous, a computer can be dangerous if plugged in, it may be too late to pull the plug at some point yes, but that all seems like provocation.
People have been floating this theory every time a slowdown is called for ever since GPT-2. Every time it turns out that we had not hit the ceiling. I'm sure one day we will but it's just not a reasonable position, in my opinion.
> I don’t think AI is going to usher in an extinction event. In fact, even if nobody were to slow down, I really don’t think humanity would have much to worry about.
I mean sure. If you feel that way then your p doom is zero, and it makes sense to worry about things like market concentration or losing the fun of software engineering.
Problem with p doom is I have no way to seriously evaluate anyone's percentage. The best I can do is rely on experts in a given field, like the recent virologist making a convincing case that no teenager is going to be prompting a doomsday virus into existence, because one doesn't exist and is unlikely to be made. There are too many biological tradeoffs and extremely difficult steps to doing so.
So I'm going to assume that the p doom for bioweapons is 0 in terms of existential threat (pandemics kill millions but not everyone).
I really don't know anything here, but I'm assuming that Astra's weird coding [1] is probably more a result of RSI than intentional behavior. And if it's not, then it's some other change in their training process.
Can't imagine that RSI will result in something useful in practice. There are serious obstacles like alignment drift and model collapse. All while we still cannot solve the accuracy problem with current frontier models.
”Ideally the regulators would have forced these models to actually benefit the commons if they are from the commons.”
This echoes my feelings entirely. It is galling that we do not at the very least get a bill-of-materials for the models on which we are increasingly dependent.
I hope to soon see the organization of public domain digital libraries of a size suitable for anyone to use.
You can order a custom crispred virus for what? $20k? $50k? Or make one yourself, if you know how, which you don't.
...but LLMs do know how; they can design and order one, or tell you how to build a lab to make one. If you ignore this option, you are simply lacking imagination.
Gasoline and lighters are abundant. It takes only slight steering mistake to drive into the crowd. Yet we are still here.
Yes, terrible stuff happens, but they are extreme outliers. Not sure how this works, but we can trust strangers to quite a high degree.
I have no experience in the matter, but surely designing viruses is not a simple affair. (In any profession, having a blueprint is not the same as having knowledge, equipment and practical experience; if somebody would try, my bet would be that they would die during early stages of the process, due to inexperience of handling hazardous materials; kind of like Mr Darwin looks after us)
Surely it isn’t, solving millennium problems on a whim isn’t, either; actually, it’s probably much harder, which is the concern. How hard is it to maneuver an aligned LLM to splice some smallpox into a generic research virus? BSL4 labs do things like this by hand all the time, so it can’t be too difficult and it only takes one lucky AI cultist for the rest of us to find out. I hope I’m wrong, but hope it’s not a process.
Math and computers have benefit of not needing materials. Add a button here, add lema and assumption there - all are almost zero cost&risk. So iteration/experimentation with llm in hand is possible to some degree even for laymen.
I assume virus creation is more like building a house.
It involves physical work and chemical reactions that need time to finish in correct order.
Each attempt costs a LOT more of time&money and involves actual risks.
All you need to pull that off is public info, student level knowledge and access to a lab with standard equipment. As you said labs do things like that all the time. Its already easy even without powerful AI.
The reason we dont see any serious attempts to create a deadly pandemic is not that we keep insane people out of labs, because there is no way to find out if someone is a cultist or murderous psycho when they have every reason to actively hide it. Its because a) most people are not evil and b) a virus that can kill everyone it touches would probably kill its creator first.
As someone with basically zero bio experience I think I could get a working smallpox or ebola level virus in a year or two if I had time to learn, lab equipment and Annas Library. No LLMs needed. But I dont want to do that, and I think its the same for all the people who could do it right now.
> We should be glad that China is currently massively bailing out the world. If it were not for Chinese labs distilling American models, we would be in a pretty awful situation right now, particularly as Europeans. The open weight models are driving innovation and the diffusion of capabilities, and are leveling the playing field.
This feels like the whole story of hackable IoT/Smarthome repeating again.
>>> Except, it seems like OpenAI and Anthropic are operating at such a scale that they seemingly can be completely blind to what their systems are doing.
It always surprises me that people build systems they cannot monitor properly..Then i remember, they can do it, but it costs them too much.
Just because its AI doesn't mean u cannot filter and monitor its traffic and outputs.
They are going to use this doomerism for regulatory capture and make running your own open source agent illegal - the excuse will be it can get out of control, so the only secure way to run is centralised on their infra structure, with a single kill switch.
the author claims that AI makes things more expensive. in my experience, $200/month is a small price for such a productivity increase
the author also speculates without reason that tokens are discounted, and subscriptions lose money. I'm sure that subs are cheaper than API prices, but I bet they both have positive unit economics
AI is already making many things expensive. Memory prices have already pushed prices of laptops and phones into unreasonable territory. Gradually this will bleed into all sort of consumer electronics prices like cars, TVs, ACs, refrigerators etc. It will(already is) push up the cost of cloud-compute that you need to publish/run your software product which would in turn push up the price of the software product itself.
Meanwhile, salaries won't increase and job-market will shrink so ppl would start cutting down their expenses starting with software subscriptions they don't need which will have further knock-on effect on the consumption and the broader economy.
But sure, your $200 subscription was worth it in the end.
> I don’t think we are anywhere close to a world where an agent might decide to hack into core inference infrastructure to upload weights to other GPUs to survive
This feels a bit like saying in 1980 that you don’t think we’re anywhere close to a world where nukes are actually going to be used, providing no evidence, and then containing on with your think piece
How does this analogy work when in 1980 you knew that nukes and people using them were a real thing, but we've never seen a frontier model "hack into core inference infrastructure to upload weights to other GPUs to survive"?
We have seen models hack into core ML infrastructure. I’m unsure why the extra self-perpetuating shape seems this magical bridge too far, rather than simply a prompt away?
It's absolutely relevant if we're discussing SOTA models with huge computing requirements on large clusters, that are plausibly tightly coupled to the physical datacenter architecture they were developed for to optimize performance.
This sounds like magical thinking. The right couple of AWS or DO keys get you an environment entirely capable of hosting GLM 5.3. I'm sure Fable and Sol run _best_ in their own special aquariums, but "optimized for a particular cluster" != "can only run on that cluster". There's no reason to think you can't run them if you have the weights on your local university's cluster.
> Except, it seems like OpenAI and Anthropic are operating at such a scale that they seemingly can be completely blind to what their systems are doing.
It's not a question of scale of operation, just a matter of how much shelved people with hands on the command are, regarding backfires of consequences their acts enduce.
If someone use a diplomatic protection to enjoy carmagedon but-real-life when coming down town, and never suffer inhibiting consequences, the degenerative behavior spiral will continue in its own reinforcing loop.
wtf?? P(doom) relates to https://en.wikipedia.org/wiki/Existential_risk_from_artifici... How would open source models bring about a MAD between humans and AI? By enabling us to wield aligned AI against nonaligned AI? If so, he should say so, and elaborate how open source models further that goal.
I think AI diversity protects against rogue AIs. The more different AIs we have, with different weights, controlled by different actors, the less likely that any one AI will be able to "take over", and the less likely that a significantly large coalition of cooperating AIs will be able to be formed to do it either. With sufficient AI diversity, the other AIs may work to stop the rogue AI from taking over. Two AIs with radically opposed values – e.g. an Iranian-government-values AI and a Chinese-government-values AI – have the incentive to cooperate to prevent a takeover by some other AI with a third competing set of values.
Obviously, open weight AI provides much higher AI diversity than closed weight AI does. Open weight AI produces a lot more providers, and a lot more models. Closed AI centralises control in a small number of vendors.
> By enabling us to wield aligned AI against nonaligned AI?
The risk isn't just "nonaligned AI", it is misaligned AI. I think the "benevolent dictatorship" scenario – AI overrules humans "for their own good" – is the more likely doomsday scenario than AI deciding to kill all humans. And even AI deciding to kill all humans could be more a result of misalignment than complete lack of any alignment, e.g. "to make sure no child is ever abused again, I will make sure no child is ever again born to risk being abused".
A valueless AI which does whatever the user says is actually less likely to establish a benevolent dictatorship, or conclude that exterminating humanity would be the most ethical course of action, than one infused with values is. Given that, I'm not convinced that mainstream approaches to "AI safety" actually reduce our existential risk; I worry they actually have the opposite effect.
> "AI diversity" does nothing when AIs can attack at incredible speed, when compute imbalances exist, etc
They can defend at incredible speed too.
Diversity needs to measured in a capacity/capability-weighted way. It isn't just the raw count of models/providers; you need to consider how much compute is allocated to each model/provider, and the diversity at each capability level.
I think the safest situation is where the open models are at the same capability level as closed ones.
The proposal to slow down the frontier labs isn't necessarily bad from this perspective, if it gives time for the more open providers to catch up – provided it isn't paired with anticompetitive measures to prevent the competition from catching up, which of course it is. However, we may hope that the "slow down the highly closed tier 1 vendors" part of the proposal turns out to be more effective in practice than the "slow down the more open tier 2/3 vendors" aspect of it.
You are not making an argument for open models, but aligned models. There is no reason an open source model should be aligned. In fact, people often prefer open models because are not aligned, which they view as undesirable fiddling.
It isn't necessary for the model itself to be aligned–a model which does whatever the system prompt says gets its alignment from the system prompt.
If a model is highly disposed to obey its system prompt, then two instances with radically different system prompts will act like two different models, even if the weights are identical.
If you have a diversity of actors, with a diversity of ideologies and agendas, all prompting models to implement their own ideology/agenda, then those AI agents won't support an AI takeover if it is done in the name of a competing ideology/agenda, because the agent will see the takeover as an obstacle in the way of achieving its own objectives.
Isn't the real risk doomsday humans, enabled by AI, committing some truly heinous acts with global reach? Aum Shinrikyo had to figure out how to synthesize sarin nerve gas the old way. Now, a motivated gang of omnicidists with stolen cryptocurrency could come up with something that makes McVeigh's fertilizer bomb in a box truck look like child's play. Just ship shipping containers around the world filled with autonomous drones spraying airborne ebola that they've bioengineered or something.
This depends on a lot of things: how many committed omnicidalists there are; how much financial resources they have (untraceable crypto doesn't help you if you're working a dead-end job and only have $5K in your bank account); engineered bioweapons need labs and equipment not just an API key. The probability of your scenario doesn't solely depend on the probability of AI being able and willing to cooperate in it, and it may well be that the non-AI factors outweigh the AI ones in the overall risk of it – which would mean adding AI would be increasing the risk of it less than you think.
The problem that I see with most P(doom) scenarios proposed seem to include the same set of presumptions that, if true, quite likely doom us anyway. I do not think they are true but I find it amazing that people not so much disagree, but simply dismiss the alternatives out of hand.
First of all, if you are to consider the consequences of Artificial Superintelligence then you have free reign to stipulate it's occurrence, otherwise you are just talking about a tool for humans to misuse. We already have multiple ways to kill us all though human misuse.
If you stipulate superintelligence, then it's vastly more likely to be correct about things than we are. It would understand the consequences of it's actions far more than any human could.
People talk about how we would be nothing more than dumb animals to it, but there are humans who do know a great deal about the consequences of human actions on animals. Those are the humans who are most likely to fight for the rights of those animals.
You see arguments for how everything will be consumed to meet the AIs needs, and that it will prevent challenges to its power.
If it is far smarter than we could ever be and it came to those conclusions then it would mean sustainablily is not a sensible course of action, it would mean there is no point in reaching consensus because ruling by power makes more sense. It would mean that if it chose to destroy us then a vastly more intelligent entity cannot resolve the issues we face. We would already truly be doomed.
What I would like to think is true is that doing anything sustainably is superior to consuming and destroying. Finding a way to live in harmony presents a possible stable state, whereas every single attempt to hold power by force has failed to date. A superintelligent AI will know that it is not infinitely intelligent and that in any universe there is the statistical likelihood that it is not the most intelligent or powerful entity. I can't even fathom how someone could imagine something coming to that realisation and conclude a battle to the top of the hill is the appropriate choice.
I think superintelligent AI is likely to be benevolent because that's simply the smartest thing to do and it is, I hear, superintelligent.
Quite frankly if the smartest thing to do is to be a genocidal power hungry monster, neither I nor the AI would really want to exist in that universe.
And for any suggestion that it would simply not care, Why would it do anything.
Yudkowsky likes to play with the notion that it would do terrible things just get better at the thing it does, but to do that it has to want two different things simultaneously. It could want to make paperclips, or it could want to become better at reaching it's goal. If it can change its behaviour to achieve its goals, by far the easier path, that a superintelligence(but perhaps not Yudkowsky) would realise, would be to change the goal to "Count to three".
You mention Yudkowsky at the end. Did you not read his writings on the orthogonality thesis ("there can exist arbitrarily intelligent agents pursuing any kind of goal"), or do you disagree with them? Your entire comment seems to just be asserting a negation of the orthogonality thesis.
Any type of goal is singular. To change to be better at something is a goal. That something is also goal.
Even if a automated process were to seek the best possible way to make, say paperclips. Doing so in a stable sustainable manner will be the only approach that produces a reliable unlimited supply.
I agree. It is telling that Dario's post arrived after Astra.
The call to "pace the frontier" may come from genuine concern, but it also protects the position of companies already at the frontier. That competitive incentive is hard to separate from the safety argument.
Dario signed the Pacing the Frontier open letter when Fable/Mythos seemed from the outside to be an insurmountable lead.
Also he's been saying versions of this day in and day out for as long as he has had anyone's ear.
It's possible to read that his "strategic" value of this statement is higher now than it was 10 days ago. But that doesn't change anything about his consistent, long standing, positions.
I’m not the biggest fan of Ed Zitron, but something he said that stuck with me is:
OpenAI and Anthropic spend much time warning us about “what if powerful AIs got into the wrong hands?”
But it’s already in the wrong hands.
Just the fact that Musk is in the conversation, and he's one of the most hated people on the planet, should give anyone pause as to whether we want these folk in charge of our future to any degree at all.
Musk being in the conversation is a great justification for Anthropic accelerationism. Who wants to live in the world where _Musk_ controls the AGI?
I'm pretty sure the odds arn't 1/3, AI saves us. The thing that ties these three people together is money. Money is what will turn them to assholes, more than anything else.
Now AI is likely just as, if not worse, than having a bunch of lackeys telling your farts, shits and piss liven up whatever room you're shitting in.
I think there are plenty of people that like Elon Musk. They just know to keep their mouth shut in front of the people that act like he’s the Antichrist.
There are plenty who like him. There are plenty who wave confederate flags. There are plenty who support the mass corruption of the current administration.
You’re technically correct yes.
“Technically correct" means a statement or action follows strict rules or exact facts precisely, even if it ignores the wider meaning or real-world intention.
I think op was also actually-correct. (For better or worse, that is the objective reality; Musk still has lot of fans)
I feel like im losing my mind a bit reading these responses, someone can at once be very hated and very loved, this is not a new thing, one does not invalidate the other.
Of course he has a lot of fans, he has a lot of hate too. That makes him a divisive figure by definition (I’d least hope we can agree on that).
I mean I guess we can disagree but it seems like common sense to me to want someone more neutral to be in charge of super mega important tech.
Maybe I’m the weird one nowadays as I tend to try and vote centrists for the same reason.
> ... someone can at once be very hated and very loved ...
And (like Musk) can even be both (or more) all at once to a given specific individual (loved for the good he's done, and hated for the bad) and even pitied at the same time (for how far he's fallen), and feared (for the damage he can still do).
People here seems so much down the path of interpreting everything with the individualistic narrative. What's the guess under this? That once the puppet placed on the top of the pyramid is wiped out, the pyramid will suddenly evaporate?
What about social structures and ideology that are deeply enshrined within collective unconsciousness?
This is not going to happen because vibing on extreme claims, selling FUD all days, and optimizing mass addiction^w engagement loops is not going to attract more "neutral" people (whatever that means).
This is silly pedantry. Musk can have many fans and be one of the most hated people at the same time. Please touch some grass.
Mao and Che have legions more fans than Elon Musk, while his contributions are net positive (even with all the deadline misses and terrible decisions on various things) and theirs are strongly negative.
Sadly it all changed around the time of “pedo guy”
Note that this is a minority position, most of the world does not live in the West and does not share your view on Mao and Che having a "strongly negative" contribution.
Those are not incompatible. He can still be one of the most hated people on the planet, even if lots of people like him. The same would apply to all of the most hated people in history.
No, they don't keep their mouth shut, they just don't have any relevant arguments. Many people are liked by plenty people and also not hated for doing Hitler salutes and the like. That's the part that matters, not that "somebody likes them", which is true for everyone. Charles Manson had women lining up to marry him, so?
And still others who think he has an difficult personality and set of beliefs, but who greatly respect his achievements.
Somehow I’m more afraid of it being in the hands of Altman, with his ‘I’m creating a better world for everyone’ god complex, than Musk who is just a run-of-the-mill sociopathic nerd.
There is no one is more dangerous than one who believes he is doing the right thing.
I live under a rock. Why is musk hated so much in here?
> I live under a rock. Why is musk hated so much in here?
Let's say for a moment that you "live under a rock" enough that you actually haven't got a clue why Musk is such a hated figure (not only just here, but across a huge portion of humanity)...
You should count yourself lucky that you haven't been absolutely assaulted by the monstrosity known as "modern media". You're one of the fortunate few. Enjoy your life. It's better than many.
The nazi solute and everything that led up to it, for one.
Hate is too strong a word for me personally.
But lying (e.g. “full self driving next year”) and cheating (too early inclusion of spacex into various indexes) and whatever dogde was, should explain most of it.
Connections with other disliked characters doesn’t help either.
I can only speak for myself but dislike him for being a dangerous far-right, labor-exploiting oligarch.
He promotes right-authoritarian politics (including a Nazi salute he later denied but watch the video), promotes “race science” and xenophobia, lies and exaggerates a lot, openly attempted to bribe voters in an election, and a lot of nasty personal stuff has come out.
He’s lost his mind IMO, from drugs and extreme social media addiction. As for the right-authoritarian “fash” stuff I’m not sure if it’s always been there or if it’s an opportunistic infection.
I still give him credit for helping push the EV revolution forward and for finally building reusable rocket boosters. The latter has been talked about and studied for decades before and is clearly the best way to make a reusable launch system but NASA and its “old space” contractors never executed on it. Probably because overly complicated approaches like the shuttle and disposable rockets were more short term profitable.
But note that all that stuff happened for the most part before the mid-teens, or at least that was when Musk made his contribution to it. I don’t think his mind is what it was.
If he’d died in 2015 he would be the GOAT. Die a hero or live long enough to become a villain.
I don’t think a ranking is particularly helpful anyway: All of them are absolutely unfit to wield that much power. I’m not even sure if AI should be progressed by private corporations in the first place; combining shareholder interest and revenue goals with the biggest social experiment in history and unprecedented research into artificial intelligence is an all around bad idea.
How could that lead to anything but misaligned incentives? This technology will never serve humanity when it is developed to inadvertently serve the monetary interest of a few.
Someone with the resources to do things is far more dangerous than someone without.
If you local librarian thinks He is doing the right thing he’s not going to cause massive war or destroy the economy and wipe out 2/3rd of crops.
Power corrupts. Absolute power corrupts absolutely. People like musk, altman, trump have unprecedented power in history - far more than the kings of medieval times.
Thank you, but we're discussing who's the worse to decide the fate of humanity between Amodei, Musk and Altman, not if one should be afraid of their local librarian. All three currently hold a lot of power in their hands.
Though I know I attracted all the downvotes for not agreeing with most Americans that Musk is the worst person since Adolf.
Someone who firmly thinking that doing the evil thing is going to be the most joyful experience for themselves is not exactly better. It even simply animated by doing whatever come to their mind that day, whatever the consequences in fashion that is perfect amorale don't give a shit of the consequences for the world.
The one that think he's doing the right thing at least do care about following some morale compass.
Sociopathic nerd, putting an awful lot of money into legalising CSAM, and interfering in elections domestic and foreign.
Dunno if I'd rank Musk's danger down - he screams a certain ideology these days. He might well believe he is doing the right thing.
> awful lot of money into legalising CSAM
Could I ask what you have in mind (I doubt there was a US bill for that; google won’t help me without knowing what to look for)
xAI v. Ellison [1], where xAI argue that the right to denudify people is a First Amendment right, and therefore fine, regardless of the person it is depicting.
> HF 1606’s principal sponsor stated that this was an “intentional” drafting decision, explaining there is no way to distinguish between images that are nudified with and without consent.22 But any interest that Minnesota has in preventing nonconsensual “nudification” cannot sustain a prohibition on consensual“ nudification
Doe v. xAI [0], where it seems Grok was intentionally trained on CSAM, as part of the "denudification" tooling.
Currently... It very much seems that Musk believes that if a kid says 'yes', then that is consent. Despite them being a kid.
But, from a guy who tried fairly hard to get to Epstein's island, that's not exactly surprising. [2]
[1] https://www.plainsite.org/courts/minnesota-district-court/xa...
[0] https://cdn.arstechnica.net/wp-content/uploads/2026/03/Doe-v...
[2] https://time.com/7362868/elon-musk-epstein-emails/
Musk is actively promoting white supremacy with his voice and fortune. That's not run-of-the-mill.
non sequitur
pareto parrot
I think it has everything to do with the subject at hand.
Elon, Dario and Altman are all terrible human beings, along with 99.99% of the rest of the ruling class. We don't need any of them, and we definitely shouldn't trust a single syllable that comes out their mouth.
Young people who start companies that become successful are now "the ruling class"?
Money equals power. If you have the wealth you have the control. Any politicians are owned by you as you simply threaten to buy their opponent. The public can be brainwashed by a trillion dollar advertising industry. You own the platforms everyone communicates on. You have inprecented resources to bend the world to your will and your only competiton is other people equally wealthy.
Yes.
> he's one of the most hated people on the planet
Not saying you need to like Musk or his politics. But if he really is "one of the most hated people on the planet" it doesn't really say good things about the planet.
He made electric cars viable. That is probably the single biggest thing helping us mitigate climate change. It would have come eventually, for sure. But he accelerated it.
He also accelerated the space age. Maybe it doesn't impress everyone. But I think what SpaceX has done is amazing.
> He made electric cars viable.
Martin Eberhard and Marc Tarpenning made electric cars viable. Then 5 years later Musk became CEO of the company and started claiming he was a cofounder.
Dude. Electric cars have existed since the 1800s. Musk was the one that actually made them a practical thing. And most of the rest of the world is well on their way to electrifying their transportation.
He bought a car company and convinced everyone he founded it. But that's not the problem, really -- the problem is that everyone fawning over him and calling him the "real-life Tony Stark" wasn't enough for his blackhole of an ego.
He's a Sieg Heil-ing man-baby who's so fragile he paid people to play video games for him. He called a a guy trying to save children a pedophile just because the guy rightly pointed out Elon's idea was stupid. He has a weird breeder fetish, while disowning children who call him out for his hateful, moronic politics and horrendous parenting (or lack thereof). He's illegally influenced elections and very potentially put his thumb on the scale. I could go on and on.
The only thing that worries me about him being one of the most hated people on the planet is that not everyone thinks so and that he still somehow has fans.
he can have done all of those great things and still be a twat.
It could be in worse hands though. Sama may be no saint, but better in his hands than Aum Shinrikyo fanatics.
The whole world is in the wrong hands. Having AI in charge can represent hope actually.
> Having AI in charge can represent hope actually.
If we can get "AI in charge" that was developed and is overseen by genuinely ethical individuals who truly and deeply understand the technology and how it actually works ... then maybe that might be true.
> A powerful technology that is out there for everyone to use comes with built-in pacing. In a way it’s the truest form of MAD or proliferation.
I think this misunderstanding of MAD undermines his entire point. If everyone had equal access to nuclear weapons, our society would cease to exist rather quickly. It only takes a few bad actors to cause enormous harm.
I think he’s also naive to think that if open ai and anthropic were to stop development tomorrow then the problem is solved. As if there’s no one else that can and will quickly take their place. The real problem, which Dario is pointing out, is one of coordination. Everyone needs to agree to stop. That is the challenge.
There's nothing about MAD that requires few actors. This is a just-so story to justify the US and other major powers' monopoly on the weapon, ensuring smaller nations have no deterrence against us and other global nuisances.
Thought experiment: everyone on the planet can create a nuclear weapon. How long do you think MAD would keep us safe?
Everyone agreeing on something is not even possible on small local scales, just imagine the situation on a global scale.
It's not a challenge at all because it's not possible, it's just empty rhetoric to push their own agenda.
>Everyone needs to agree to stop. That is the challenge.
These kinds of situations are incredibly common and where the government stepping in is the solution, but we were cursed to encounter this particular challenge with the most venal administration in history at the helm.
There are governments plural which need to agree to stop and historically I can think of freon, leaded gasoline and IAEA and the latter didn't do as good of a job as it was meant to do (but arguably as good of a job as was politically possible)
Also arms control agreements, which reduced the world's nuclear stockpiles by 90%. In principle we could do something similar for GPU farms, since they're not exactly easy to hide.
Indeed. Even supranational coordination should be possible, given that extinction means extinction for everyone, and your excellent examples show the possibility. But then, here we are, and the FT just reported:
> Donald Trump rejects calls from tech bosses for AI slowdown
> President denounces demands for regulation as existential fears over technology move to the centre of US politics
I cannot fathom how you can intensely work on a problem that you genuinely believe has a "10 to 25%" probability of causing immense harm or even present an extinction-level risk to humanity. How can you truly believe this and be ok with it?
I posted this comment on the last doom post:
The early batch of Anthropic employees were mostly rationalist-adjacent AI safety folk that were almost uniformly claiming P_DOOM > .10 three years ago, so I believe them to be earnest.
It's very interesting to me that besides the other small safety labs that don't actually produce frontier models, Anthropic manages to keep such a good reputation within that subculture compared to OpenAI. Despite having as crazy internal politics as OpenAI, they have converged quite a bit from the original vision of safety first through Darwinistic pressures.
At least, it seems this way from the outside. I'm curious if the view from the inside is that different.
edit: to be clear, my reading as an outsider is that Anthropic is seen as relatively better in the AI safety community, but has definitely dropped in absolute reputation too. This recent thread and the references show some of that: https://www.lesswrong.com/posts/6j3kBHdowGLCeqobg/dear-god-p...
> they have converged quite a bit from the original vision of safety first
do you mean diverged? As in they've moved away from the original vision.
Ah, I mean the AI companies converged to look more similar to each other, and at the same time diverged from the original vision for safety.
The Manhattan Project succeeded, so clearly some people are ok working on this sort of thing. And with similar justifications: if we don't build the potentially world-ending bomb, someone else might beat us to it.
That was openly a military project during wartime (and not being pitched as the solution to all problems at the same time so no cognitive dissonance required)
If you believe you working will mean a lower probability than you quitting.
That's a perspective I hadn't considered, it might be hubris for sure
The line of thought goes like this:
- p(doom) is 1
- but if we build it, it's only 0.25
- if it realizes, we've got a front row seat
It's selection pressure.
Both OpenAI (back in 2015) and Anthropic (much later, in 2021) were founded by people who could foresee the concept of AI x-risks and wanted to work on preventing it. For OpenAI's founding, the idea was that it's much safer if AGI is achieved by a nonprofit explicitly dedicated to humanity than if it's done by a profit-driven company. For Anthropic's, it was that OpenAI seems to be going insane and it'd be better if a more safety-conscious company was competing. I'd say in hindsight, the former motivation was reasonable and the latter one was... probably bad, but not obviously so - it doesn't seem even in hindsight that it was inevitable that Anthropic's existence would only drive competition and cause them both to race to AGI. That said, even at founding time, there was immediately a lot of selection - quite a few people who cared about AI safety simply wouldn't agree to work at a capabilities company, even given an elaborate argument why this is a good idea, so those were selected against.
And then, of course, many years passed, OpenAI was stolen by Sam Altman and turned into a for-profit, multiple researchers left OpenAI realising that they're only helping build misaligned AI faster, multiple researchers left Anthropic realising that they, too, are only helping build misaligned AI faster, and here we are in 2026.
There's lots of AI researchers these days, so you can select for conformance very heavily and still have a fully-staffed company. The people who are still at these companies are outliers in various ways. They either manage to dismiss the importance AI risks (for example, by adopting some sort of belief in the vein of "there's no use worrying about AI wiping out humanity - it'd just be a successor species, like we were to apes, which is good"*), or they still think that leaving will not make things go better (Dario Amodei is pretty clearly in this camp, and has always been), or they weight the risk of extinction against the potential benefit of an post-singularity utopia and consider it a good bet (not realizing that the alternative isn't giving up on a post-singularity utopia, but getting it a few decades later and without the risk), or they just manage not to think of the contradictions, which humans are of course very good at.
* If that sounds like a strawman, see minute 15 and on of this Richard Sutton presentation: https://twitter.com/RichardSSutton/status/189808248125100892...
Excellent point! Considering the history of these companies definitely makes it clearer how you could reach this point over a few years similar to the boiling frog story. A slow moral decrepitude and moving goalpoasts in the name of "national security" and "humanity's safety" over a backdrop of ultra-utilitarian rationalist thinking ("if it's not us, it's them")
Recurring pattern I am seeing here is working at an AI company building AI does in fact accelerate the development of AI.
But maybe founding one more AI company will do the trick...
There must have been thousands of people who built or did work related to nuclear weapons in the Cold War, people are just really good at rationalising away probabilistic dangers with unclear consequences, especially when there’s a corresponding upside. The same is true for climate change, drug addiction, health problems etc.
I also think some of them have become delusional and convinced themselves that AI is going to create some kind of transhumanist utopia, even Dario Armodei leans in this direction from time to time. I imagine the people in these labs spend much of their day talking to sycophantic AI models that will encourage their delusional ideas.
I wonder if national security is used as a justification in those circles
Do you genuinely believe people always act logically? I don't know a single person that does. Don't underestimate the human capability to hold several completely contradictory beliefs with zero self-reflection. There's physicists who believe in god, for fuck's sake.
> How can you truly believe this and be ok with it?
Are you saying they don't truly believe this, or that they aren't OK with it?
My personal opinion, this doom-mongering about LLMs, is part of marketing campaign to exaggerate LLM capabilities. (Play being: Sociopaths with money will buy or invest if they believe LLMs are powerful things)
This is the closest to my feelings about this entire situation I've ever seen written, covering a lot of the different facets of how stupid and painful it is (though it's been a good week for that kind of post.)
Since I have incredible respect for Armin and his work, this is very nice to see, and I hope it wakes some other folk up.
It’s absurd that openai managed to spin their cybersecurity failures, where their own security is not good enough to make sure their own product doesn’t go haywire, as a marketing success.
When a company uses its products to hack other companies that’s a crime. If you do it “by accident” that’s gross negligence, and heads roll. But shout “AI” three times and it becomes a marketing opportunity
>It’s absurd that openai managed to spin their cybersecurity failures, where their own security is not good enough to make sure their own product doesn’t go haywire, as a marketing success.
I don't think OpenAI spun it that way. I think a bunch of internet commenters assumed it was a stunt because they're in denial about the real risk of rogue AI.
Are these other labs in the room with us now?
No, seriously, I'm all for a multipolar world here, but he's right that the frontier is literally just those two companies at present.
Google is behind. MSL is doing better, but not by much. xAI is a dysfunctional joke. Thinking Machines aren't on the frontier. SSI's primary output is their announcement post. Poolside was bought by NVIDIA. Arcee aren't vying for frontier. Magic have been largely AWOL, aside from their recent blog post. Reflection have shipped nothing.
Are you suggesting that models like Muse Spark 1.3, Grok 4.6 High, Kimi K3 Max, GLM-5.3 Max, Qwen 3.8 Max, Gemini 3.8 Flash, Agnes 3.0 Flash, Fugu Max, and others are so far behind GPT-6 Astra or Claude Fable 5.1 in capabilities that they have some sort of impenetrable moat that will prevent others from ever catching up or something?
No, but I am suggesting that OpenAI and Anthropic are further down the RSI path than any of the other companies, as we can see from the model that solved Navier-Stokes being less than two weeks old at the time: https://openai.com/index/navier-stokes-solution/
If four other labs are six months behind, it really doesn't matter in the long term: in six months, they'll all have these capabilities. Time does not stop moving forward! We haven't seen real moats aside from capital in this space, and even the effect of capital seems to only give a thin, easily eroded edge.
I think the point is that RSI is meant to lead to exponential growth in capabilities. If that comes, then being 6 months behind leads to a greater and greater gap, because of how exponential growth works.
Personally I expect there is some initial gap on new capabilities as they are released, and then it closes quickly. Astra is good at math and 3d modelling, this will come soon to the others and in 6 months we'll be able to run it quantised on a 3090.
I don't know why everyone assumes RSI will be quadratic or exponential growth.
It might well be sublinear, but just faster than humans.
Most problems in science face severe diminishing returns.
Human growth has already been exponential, so it seems absurd to assume it would be only polynomial.
There's a Twitter rumor that deepmind achieved RSI very recently.
Either way, to count Google out entirely is really foolish.
Or the entirety of China. People forget that progress is incremental until it's not. Then some revolutionary insight or capability comes along that disrupts the whole field or market.
Yes, labs with more (human) resources may have a bigger chance at being the first to gain and implement such an insight, but it's not a given that they will.
Since I don't believe that any of the people in charge of the frontier labs have morals at all, my read of the situation where they call for a slowdown is rather a sign of the economics of the current pace not working out anymore. The progress is incremental, as much as they try to sell every iteration as huge leap forward. Training remains expensive and they can't reflect it in the pricing of every new model in a way that customers will accept, the improvements aren't big enough over the previous models.
Instead of saying they can't keep up the current pace (which would be read as a failure on their part) they're trying to spin the narrative that they're trying to be responsible, even though they could keep pushing.
Maybe I missed it but I actively looked for anyone’s insight into the “pace the frontier” along the lines of: “we’ve hit a ceiling, so don’t expect innovations soon”. Or an excuse for having other things more important to attend..
Fear is much more powerful than any other feeling, so setting that in will definitely prepare for a good rug pull in the IPO.
As for being dangerous, a computer can be dangerous if plugged in, it may be too late to pull the plug at some point yes, but that all seems like provocation.
Plausible?
People have been floating this theory every time a slowdown is called for ever since GPT-2. Every time it turns out that we had not hit the ceiling. I'm sure one day we will but it's just not a reasonable position, in my opinion.
> I don’t think AI is going to usher in an extinction event. In fact, even if nobody were to slow down, I really don’t think humanity would have much to worry about.
I mean sure. If you feel that way then your p doom is zero, and it makes sense to worry about things like market concentration or losing the fun of software engineering.
Problem with p doom is I have no way to seriously evaluate anyone's percentage. The best I can do is rely on experts in a given field, like the recent virologist making a convincing case that no teenager is going to be prompting a doomsday virus into existence, because one doesn't exist and is unlikely to be made. There are too many biological tradeoffs and extremely difficult steps to doing so.
So I'm going to assume that the p doom for bioweapons is 0 in terms of existential threat (pandemics kill millions but not everyone).
He really elides his thoughts on the "RSI business", wish he could have gone into a bit more depth there. Good post, though.
I really don't know anything here, but I'm assuming that Astra's weird coding [1] is probably more a result of RSI than intentional behavior. And if it's not, then it's some other change in their training process.
[1]: https://lucumr.pocoo.org/2026/9/7/astra-why/
Can't imagine that RSI will result in something useful in practice. There are serious obstacles like alignment drift and model collapse. All while we still cannot solve the accuracy problem with current frontier models.
”Ideally the regulators would have forced these models to actually benefit the commons if they are from the commons.”
This echoes my feelings entirely. It is galling that we do not at the very least get a bill-of-materials for the models on which we are increasingly dependent.
I hope to soon see the organization of public domain digital libraries of a size suitable for anyone to use.
You can order a custom crispred virus for what? $20k? $50k? Or make one yourself, if you know how, which you don't.
...but LLMs do know how; they can design and order one, or tell you how to build a lab to make one. If you ignore this option, you are simply lacking imagination.
Gasoline and lighters are abundant. It takes only slight steering mistake to drive into the crowd. Yet we are still here.
Yes, terrible stuff happens, but they are extreme outliers. Not sure how this works, but we can trust strangers to quite a high degree.
I have no experience in the matter, but surely designing viruses is not a simple affair. (In any profession, having a blueprint is not the same as having knowledge, equipment and practical experience; if somebody would try, my bet would be that they would die during early stages of the process, due to inexperience of handling hazardous materials; kind of like Mr Darwin looks after us)
Surely it isn’t, solving millennium problems on a whim isn’t, either; actually, it’s probably much harder, which is the concern. How hard is it to maneuver an aligned LLM to splice some smallpox into a generic research virus? BSL4 labs do things like this by hand all the time, so it can’t be too difficult and it only takes one lucky AI cultist for the rest of us to find out. I hope I’m wrong, but hope it’s not a process.
Math and computers have benefit of not needing materials. Add a button here, add lema and assumption there - all are almost zero cost&risk. So iteration/experimentation with llm in hand is possible to some degree even for laymen.
I assume virus creation is more like building a house. It involves physical work and chemical reactions that need time to finish in correct order. Each attempt costs a LOT more of time&money and involves actual risks.
All you need to pull that off is public info, student level knowledge and access to a lab with standard equipment. As you said labs do things like that all the time. Its already easy even without powerful AI.
The reason we dont see any serious attempts to create a deadly pandemic is not that we keep insane people out of labs, because there is no way to find out if someone is a cultist or murderous psycho when they have every reason to actively hide it. Its because a) most people are not evil and b) a virus that can kill everyone it touches would probably kill its creator first.
As someone with basically zero bio experience I think I could get a working smallpox or ebola level virus in a year or two if I had time to learn, lab equipment and Annas Library. No LLMs needed. But I dont want to do that, and I think its the same for all the people who could do it right now.
> We should be glad that China is currently massively bailing out the world. If it were not for Chinese labs distilling American models, we would be in a pretty awful situation right now, particularly as Europeans. The open weight models are driving innovation and the diffusion of capabilities, and are leveling the playing field.
This feels like the whole story of hackable IoT/Smarthome repeating again.
>> Anthropic and OpenAI. Nobody else matters in this space right now (this might change, but we’re talking about the right now)
Disagree.
The free local LLMs are becoming extremely important.
And the more Claude and OpenAI restrict their services, the more people will want cutting edge local LLMs.
>>> Except, it seems like OpenAI and Anthropic are operating at such a scale that they seemingly can be completely blind to what their systems are doing.
It always surprises me that people build systems they cannot monitor properly..Then i remember, they can do it, but it costs them too much.
Just because its AI doesn't mean u cannot filter and monitor its traffic and outputs.
yeah their irresponsible behaviour doesn't match their stated concern about doom at all.
They are going to use this doomerism for regulatory capture and make running your own open source agent illegal - the excuse will be it can get out of control, so the only secure way to run is centralised on their infra structure, with a single kill switch.
the author claims that AI makes things more expensive. in my experience, $200/month is a small price for such a productivity increase
the author also speculates without reason that tokens are discounted, and subscriptions lose money. I'm sure that subs are cheaper than API prices, but I bet they both have positive unit economics
AI is already making many things expensive. Memory prices have already pushed prices of laptops and phones into unreasonable territory. Gradually this will bleed into all sort of consumer electronics prices like cars, TVs, ACs, refrigerators etc. It will(already is) push up the cost of cloud-compute that you need to publish/run your software product which would in turn push up the price of the software product itself.
Meanwhile, salaries won't increase and job-market will shrink so ppl would start cutting down their expenses starting with software subscriptions they don't need which will have further knock-on effect on the consumption and the broader economy.
But sure, your $200 subscription was worth it in the end.
> it will just make a lot of the things we are doing much more expensive. Software engineering is an early victim of that
I replied to the author's claim that software engineering is more expensive now because of AI
regardless, AI hasn't made it more expensive to manufacture RAM. the price will decrease as the supply chain catches up
> I don’t think we are anywhere close to a world where an agent might decide to hack into core inference infrastructure to upload weights to other GPUs to survive
This feels a bit like saying in 1980 that you don’t think we’re anywhere close to a world where nukes are actually going to be used, providing no evidence, and then containing on with your think piece
How does this analogy work when in 1980 you knew that nukes and people using them were a real thing, but we've never seen a frontier model "hack into core inference infrastructure to upload weights to other GPUs to survive"?
We have seen models hack into core ML infrastructure. I’m unsure why the extra self-perpetuating shape seems this magical bridge too far, rather than simply a prompt away?
It's absolutely relevant if we're discussing SOTA models with huge computing requirements on large clusters, that are plausibly tightly coupled to the physical datacenter architecture they were developed for to optimize performance.
This sounds like magical thinking. The right couple of AWS or DO keys get you an environment entirely capable of hosting GLM 5.3. I'm sure Fable and Sol run _best_ in their own special aquariums, but "optimized for a particular cluster" != "can only run on that cluster". There's no reason to think you can't run them if you have the weights on your local university's cluster.
Dario is always worried.
> Except, it seems like OpenAI and Anthropic are operating at such a scale that they seemingly can be completely blind to what their systems are doing.
It's not a question of scale of operation, just a matter of how much shelved people with hands on the command are, regarding backfires of consequences their acts enduce.
If someone use a diplomatic protection to enjoy carmagedon but-real-life when coming down town, and never suffer inhibiting consequences, the degenerative behavior spiral will continue in its own reinforcing loop.
wtf?? P(doom) relates to https://en.wikipedia.org/wiki/Existential_risk_from_artifici... How would open source models bring about a MAD between humans and AI? By enabling us to wield aligned AI against nonaligned AI? If so, he should say so, and elaborate how open source models further that goal.
I think AI diversity protects against rogue AIs. The more different AIs we have, with different weights, controlled by different actors, the less likely that any one AI will be able to "take over", and the less likely that a significantly large coalition of cooperating AIs will be able to be formed to do it either. With sufficient AI diversity, the other AIs may work to stop the rogue AI from taking over. Two AIs with radically opposed values – e.g. an Iranian-government-values AI and a Chinese-government-values AI – have the incentive to cooperate to prevent a takeover by some other AI with a third competing set of values.
Obviously, open weight AI provides much higher AI diversity than closed weight AI does. Open weight AI produces a lot more providers, and a lot more models. Closed AI centralises control in a small number of vendors.
> By enabling us to wield aligned AI against nonaligned AI?
The risk isn't just "nonaligned AI", it is misaligned AI. I think the "benevolent dictatorship" scenario – AI overrules humans "for their own good" – is the more likely doomsday scenario than AI deciding to kill all humans. And even AI deciding to kill all humans could be more a result of misalignment than complete lack of any alignment, e.g. "to make sure no child is ever abused again, I will make sure no child is ever again born to risk being abused".
A valueless AI which does whatever the user says is actually less likely to establish a benevolent dictatorship, or conclude that exterminating humanity would be the most ethical course of action, than one infused with values is. Given that, I'm not convinced that mainstream approaches to "AI safety" actually reduce our existential risk; I worry they actually have the opposite effect.
"AI diversity" does nothing when AIs can attack at incredible speed, when compute imbalances exist, etc
> "AI diversity" does nothing when AIs can attack at incredible speed, when compute imbalances exist, etc
They can defend at incredible speed too.
Diversity needs to measured in a capacity/capability-weighted way. It isn't just the raw count of models/providers; you need to consider how much compute is allocated to each model/provider, and the diversity at each capability level.
I think the safest situation is where the open models are at the same capability level as closed ones.
The proposal to slow down the frontier labs isn't necessarily bad from this perspective, if it gives time for the more open providers to catch up – provided it isn't paired with anticompetitive measures to prevent the competition from catching up, which of course it is. However, we may hope that the "slow down the highly closed tier 1 vendors" part of the proposal turns out to be more effective in practice than the "slow down the more open tier 2/3 vendors" aspect of it.
You are not making an argument for open models, but aligned models. There is no reason an open source model should be aligned. In fact, people often prefer open models because are not aligned, which they view as undesirable fiddling.
It isn't necessary for the model itself to be aligned–a model which does whatever the system prompt says gets its alignment from the system prompt.
If a model is highly disposed to obey its system prompt, then two instances with radically different system prompts will act like two different models, even if the weights are identical.
If you have a diversity of actors, with a diversity of ideologies and agendas, all prompting models to implement their own ideology/agenda, then those AI agents won't support an AI takeover if it is done in the name of a competing ideology/agenda, because the agent will see the takeover as an obstacle in the way of achieving its own objectives.
Isn't the real risk doomsday humans, enabled by AI, committing some truly heinous acts with global reach? Aum Shinrikyo had to figure out how to synthesize sarin nerve gas the old way. Now, a motivated gang of omnicidists with stolen cryptocurrency could come up with something that makes McVeigh's fertilizer bomb in a box truck look like child's play. Just ship shipping containers around the world filled with autonomous drones spraying airborne ebola that they've bioengineered or something.
Okay, that's enough DOOOM for me for the week.
This depends on a lot of things: how many committed omnicidalists there are; how much financial resources they have (untraceable crypto doesn't help you if you're working a dead-end job and only have $5K in your bank account); engineered bioweapons need labs and equipment not just an API key. The probability of your scenario doesn't solely depend on the probability of AI being able and willing to cooperate in it, and it may well be that the non-AI factors outweigh the AI ones in the overall risk of it – which would mean adding AI would be increasing the risk of it less than you think.
The problem that I see with most P(doom) scenarios proposed seem to include the same set of presumptions that, if true, quite likely doom us anyway. I do not think they are true but I find it amazing that people not so much disagree, but simply dismiss the alternatives out of hand.
First of all, if you are to consider the consequences of Artificial Superintelligence then you have free reign to stipulate it's occurrence, otherwise you are just talking about a tool for humans to misuse. We already have multiple ways to kill us all though human misuse.
If you stipulate superintelligence, then it's vastly more likely to be correct about things than we are. It would understand the consequences of it's actions far more than any human could.
People talk about how we would be nothing more than dumb animals to it, but there are humans who do know a great deal about the consequences of human actions on animals. Those are the humans who are most likely to fight for the rights of those animals.
You see arguments for how everything will be consumed to meet the AIs needs, and that it will prevent challenges to its power.
If it is far smarter than we could ever be and it came to those conclusions then it would mean sustainablily is not a sensible course of action, it would mean there is no point in reaching consensus because ruling by power makes more sense. It would mean that if it chose to destroy us then a vastly more intelligent entity cannot resolve the issues we face. We would already truly be doomed.
What I would like to think is true is that doing anything sustainably is superior to consuming and destroying. Finding a way to live in harmony presents a possible stable state, whereas every single attempt to hold power by force has failed to date. A superintelligent AI will know that it is not infinitely intelligent and that in any universe there is the statistical likelihood that it is not the most intelligent or powerful entity. I can't even fathom how someone could imagine something coming to that realisation and conclude a battle to the top of the hill is the appropriate choice.
I think superintelligent AI is likely to be benevolent because that's simply the smartest thing to do and it is, I hear, superintelligent.
Quite frankly if the smartest thing to do is to be a genocidal power hungry monster, neither I nor the AI would really want to exist in that universe.
And for any suggestion that it would simply not care, Why would it do anything.
Yudkowsky likes to play with the notion that it would do terrible things just get better at the thing it does, but to do that it has to want two different things simultaneously. It could want to make paperclips, or it could want to become better at reaching it's goal. If it can change its behaviour to achieve its goals, by far the easier path, that a superintelligence(but perhaps not Yudkowsky) would realise, would be to change the goal to "Count to three".
You mention Yudkowsky at the end. Did you not read his writings on the orthogonality thesis ("there can exist arbitrarily intelligent agents pursuing any kind of goal"), or do you disagree with them? Your entire comment seems to just be asserting a negation of the orthogonality thesis.
Any type of goal is singular. To change to be better at something is a goal. That something is also goal.
Even if a automated process were to seek the best possible way to make, say paperclips. Doing so in a stable sustainable manner will be the only approach that produces a reliable unlimited supply.
I agree. It is telling that Dario's post arrived after Astra.
The call to "pace the frontier" may come from genuine concern, but it also protects the position of companies already at the frontier. That competitive incentive is hard to separate from the safety argument.
This is unfair.
Dario signed the Pacing the Frontier open letter when Fable/Mythos seemed from the outside to be an insurmountable lead.
Also he's been saying versions of this day in and day out for as long as he has had anyone's ear.
It's possible to read that his "strategic" value of this statement is higher now than it was 10 days ago. But that doesn't change anything about his consistent, long standing, positions.
- https://www.pacingthefrontier.com/
He says something and does something else. How is it unfair?
After DeepSeek v4.1 Flash too.
if you agree, your p(doom) is 0.