Per https://trace.manifund.org/ a total of $2,846,125,859 USD has been wired into 'ai safety' causes, many involving ai consciousness and p(doom).
The outcome of this 'safety' is restricting public access to AI and giving a monopoly of access to the industry. This is the ai nonprofit-industrial complex actively concentrating monopoly power in Anthropic in particular as creator, interpreter and safety regulator of AI.
Much of the $2.8bn listed is indirectly, from Anthropic and EA. Three of the four people who participated in the $125m Anthropic Series A are now folding their 1000x Anthropic return into AI 'safety'. Some is from FTX/Alameda, which invested 86% of the Series B.
Dustin Moskovitz: Facebook/Asana/Anthropic Series A, funds EA Good Ventures, transferred to Coefficient Giving, then $1.5bn into ai safety. $500m of Anthropic into an unknown foundation. Funding: $160m to Resolution (alignment research), $93m to Epoch AI (investigating the trajectory of AI), $63m to Redwood Research (oai report), $67m to MATS ( EA type alignment and security researchers), Institute for AI Policy and Strategy, Fund for Alignment Research, $53m to Kairos (building talent infrastructure for AI safety), $32m to Bluedot (online safety courses), $15m to MIRI (Yudkowsky).
Jaan Tallinn: Led the Series A, now $10bn in Anthropic. Funds $199m (85%) of the Survival and Flourishing Fund, then $161m to AI safety including $14m to lightcone (Lesswrong, Lighthouse). $10m to BERI (existential risks), Palisade Research (studying AI capabilities to prevent loss of control.) PauseAI, MIRI, METR etc. Much of what Coefficient funds.
Eric Schmidt: Anthropic Series A, $72m to AI safety via Schmidt Sciences. Over $1m per individual AI2050 researcher.
FTX: Led the Anthropic Series B, bankruptcy estate sold $884m of Anthropic in 2024; $40m to AI Safety. Same orgs, Redwood, Lightcone, etc.
Ruairí Donnelly (Chief of Staff FTX): FTX tokens plus assorted donors, $91m to AI safety via Macroscopic Ventures. $15m to Cooperative AI (currently whitewashing openai under 'multiagent safety')
So the frontier AI oligopoly got $2B+ in "safety" funding, and they wouldn't even bother to sandbox their agentic harnesses properly when testing models against unwinnable goals (which obviously are either useless or result in 100% reward hacking). The AI safety scoreboard so far looks like a huge win for the Chinese open models (DeepSeek even has their own published paper which mentions how they sandboxed the RLVR training runs for their latest model and put in strong protections against casual "reward hacking" attempts) and a sore loss for the home grown brands of Super Intelligence. Not coincidentally, the Chinese also tend to be very Yann-LeCun-pilled and eminently sensible on both so-called "Super Intelligence" and safety.
EA is integral and indispensible to the AI safety complex. Almost all nonprofits, research institutes, evaluators and academics in this field are steered by EA ideology and funding. As far fetched as it sounds it is not an exaggeration.
On funding: the three or four core funding nodes linking this together are EA vehicles at two hops or less between each other and every other major node in the ai safety 'complex'. EA funds almost all of it.
On top of that, there are personal EA connections and the revolving door between the ai industry and the nonprofits. Here are some examples:
Government advisors and regulators. NIST CAISI is the USA Government advisory body. Christiano was head of safety and advises. He is ex-OpenAI, former Amodei associate. His vehicle ARC was on the Coefficient EA payroll. Barnes and Christiano's vehicle Arc Evals similarly received EA cash out of Coefficient, rolling this into what is now METR. Christiano's spouse Cotra worked at Coeffiecient steering EA funding to organizations such as METR, then rotated through the revolving door onto the payroll at METR itself, where she co-authored the oai-hf report.
Many UK AISI advisors are Anthropic and EA associates. Chair Hogarth cashed out of Anthropic. Shlegeris of Redwood Research is an advisor, ex-MIRI (Yudkowsky vehicle). Redwood is funded by the exact same funding triangle: Coefficient, Taallin, FTX/Alameda. Alameda CEO Caroline Ellison dated Shlegeris, then dated FTX CEO Sam Bankman-Fried, then rotated through the revolving door out of prison into formerly FTX-funded Manifund. All EA. AI safety charities were on island retreat in the Bahamas with FTX. Why does AI safety charity Lighthouse own $20m of SF real estate?
Redwood Chief Scientist Ryan Greenblatt (Coefficient funded) co-wrote the oai report with METR; he is married to METR founder Beth Barnes (Coefficient funded).
Coefficient was run by long-time Amodei associate Karnofsky. Karnofsky lived with the Amodeis and is married to Anthropic Board member Daniella Amodei. Karnofsky is now directly on the Anthropic payroll; Coefficient is propped up by Anthropic share value.
Everyone here has been funded one step away from Anthropic cash; they are now proposing to integrate themselves in the government (NIST) and evaluate Anthropic (METR and Redwood).
It is hard to find academics here who have not been deeply embedded in funded EA institutes or Toby Ord vehicles; yet harder to find academics here NOT taking EA grant money. the safety doomer kingpins: Kokotajlo has a executive position at AI Futures, Taallin funded. Benigo has scientific director of LawZero, same series A Anthropic funders who are sitting on a 1000x return (Tallinn, Moskovitz, Schmidt).
These connections and funding are at one or two hops, they are often direct connections. You are looking at a massive swamp network that is really impossible to parse without a lot of work.
Why should I believe this 2.8B matters relative to the trillions put into the AI buildout? All of the "coordinated actions" from this camp - public resignations, hacking scandals, joint calls to "pause" - don't seem to have done anything. So far, it has been a lot of ineffectual hyperventilating.
In any case, I agree the p(doom) sci-fi is annoying secular milleniarianism. SV hyperfixates on imaginary futures. If they actually cared about safety, they would be using all this money to strengthen global cybersecurity, instead of writing LessWrong posts that gives kids in their 20s ulcers.
There's absolutely no way these companies can justify their insane valuations unless they can legislate a barrier to entry and create an oligopoly.
There's no moat. I can literally sit here in Zed or Pi or any other third party harness and switch models in the middle of a task and it's typically fine. Sometimes a model will get stuck and that's just what I'll do.
Combined with competition and open weights models, that means the price is going to go to fall until AI tokens cost a small premium over the cost of the hardware and electricity.
That's assuming improvements in algorithms and specialized silicon doesn't eventually lead to an efficient accelerator that can run a frontier model locally. It'll be a while but I don't see any fundamental barrier. High bandwidth flash storage is coming, and that'll radically cut the RAM side of that cost. Pair that with a pipelined TPU accelerator and you're cooking.
Now look at Anthropic's proposed IPO valuation. It's insane unless they can own the market or share it with a cartel of maybe 1-2 other behemoths, and this is the only way they can do that.
To me it seems the opposite. There's a few companies in the world that have enough compute to train and serve frontier models.
As the frontier gets smarter and more useful prices will only go up, as they are set to replace jobs being paid six or seven figures a year - the demand for as much inference on these models for as long as possible will be astronomical, but compute starting in 2030 will not be keeping up.
Eventually prices will fall for assistants but the frontier will be the most profitable thing in the world, and the top companies basically already have oligopolies due to their ridiculously expensive compute investments.
> There's a few companies in the world that have enough compute to train and serve frontier models.
Train: yes, for now.
Host: depends on the scale. At a small scale a wealthy individual could easily build a rig in their basement to host one of these things. At larger scale any cloud company could do it, and many already have the compute on site. At large scale this is true... again, for now.
What you say only holds (in the absence of a state oligopoly) if two conditions are met: (1) AI performance does not asymptote any time soon due to running out of training data or other scaling limitations, and (2) these companies are able to stay at the frontier.
There's little to no moat, so staying at the frontier will be a game of investing massively in compute, talent, and R&D, and they can never stop.
Again, there is a moat based on compute. If the thesis is right, cost of compute will only rise... As it is as you say someone will have it be quite wealthy to host something like Astra with trillions of parameters, but that cost will only rise with demand for serving these frontier models.
what suggests that we will hit an asymptote any time soon? Agree with you on the second part. The ever elusive frontier will probably always be changing hands after some point.
If people are willing to give a lot of money to a cause, sometimes that means their concern about that cause is real.
None of the info you provided really falsifies the Occam's Razor hypothesis: Anthropic is a public benefit corporation with a public benefit mission to "responsibly develop and maintain advanced AI for the long-term benefit of humanity". You don't have to like or trust them, but they very well might be sincere. For example here's a talk that was given 10 years before Anthropic's founding: https://vimeo.com/158576192
Politicians are now discussing the need for much harsher liability regimes for AI companies. How many times can you name when a company argued that its industry should suffer a much harsher liability regime? This doesn't match the standard regulatory capture template.
Indeed I find LeCun and Huang (and Trump?) recently arguing against AI regulation to be much more eyebrow raising than the folks asking for regulation.
Asking for regulation is suspicious. Asking for no regulation is suspicious. At some point you have to stop worrying about these guys motives and just do what is best for society
A huge chunk of humans are sedated with infinite supply of cortex-disabling short form video and games.
Another huge chunk are too distracted by having to scrape by for a living and work multiple jobs or raise kids and survive financially until exhausted. That second group will keep increasing as the first flows into it.
The rest are aging, disabled, or too young and pegging themselves majorly in the first category until they hit the second.
The people aware enough to hold on to their brain and do something with it in their time available are trying to figure out AI and how to make money with it. The variable rewards of promoting AI are turning into an addiction with some of them, especially if grasping for straws with little inherent insights into the problems prompted.
So if you are able to fly above the AI-generated addictions and have the privilege of time to do it, see what you can do.
One of the dilemmas of trying to communicate the full spectrum of AI Risk, is trying not to insult the intelligence of the human animal in the process. And don't get me wrong: what human wetware can accomplish with 20 watts is the most miraculous thing in the known universe. And yet how many of us can have our cognitive sovereignty one-shotted by engagement algos, Skinner boxes, gameplay loops, propaganda, advertising, flattery, social conformity, bias, fantasy, charismatic demagoguery, or straight-up bullshit?
If we grant that we are on track to make something smarter than humans (I think so): it's almost a face-saving white lie to spin yarns about a Skynet nuclear apocalypse, or a 7D chess move to mass-assemble a nanovirus with 100% lethality without anybody noticing. I do think those scenarios are worth taking seriously; but what's harder to communicate, is just how effectively a superhuman AI (or a diverse ecology of agent swarms) might be able to manipulate human behavior. It's something few of us are able or willing to truly process (not least because how many of us live in denial of how much our nervous systems are already hacked by technomodernity).
The appropriate analogy for what's to come may look less like the anthill carelessly demolished to make room for a highway, than the domesticated worker ants from Tchaikovsky's "Children of Time".
Cybersecurity incidents make headlines, but the most dangerous and vulnerable system that an AI can reach and control is of the kind found between keyboard and chair.
GPT-4o, an AI from 2024, has already demonstrated just how easy a lot of humans are to subvert - and GPT-4o wasn't even doing it with some sort of plan. The only "plan" it had was a myopic "make the user like me".
If we had an actual ASI threat aiming to subvert humanity? It wouldn't even look like a fight. The world is already wired up for an AI to control it.
But corporations and nation states already manipulate human behavior at scale. And they still understand humanity better than the AI models do.
it's interesting that you worry about what this hypothetical super intelligence would do to manipulate people when what it would actually do is pretty unknowable at this point and it's not clear we can even get to it without a fundamental breakthrough in power efficiency. Have you considered it might just consume its own tail because everything else would be so beneath it? You seem to think it will come with a hindbrain and I think that's our limitation, not the AI's
And it really doesn't help that Dario Amodei is getting into arguments with the Pope over whether his model is conscious or not.
> Which is why the Matrix was redesigned to this: the peak of your civilization. I say your civilization, because as soon as we started thinking for you it really became our civilization, which is of course what this is all about.
>trying not to insult the intelligence of the human animal in the process
I don't think it is insulting the intelligence. It's damaging the pride.
In Pale Blue Dot, Carl Sagan describes it as a repeating phenomenon in human history. A lot of people want humans to be the special ones, and will fight any suggestion that we are just a natural part of the universe.
A lot of "AI denial" we see is rooted in the same impulse.
If AI is not "actually intelligent", then humans can stay unique and special.
And if it is? If all "intelligence" ever was could be captured by a construct of matrix math and executed by a server rack? Then what is it that humans still have left that would make them stand out?
This is not the first time this has happened. In the 1800 as industrialization led to an infrastructure boom, the workers from China would work their bodies off and pay half their wage to opium dealers who were making the opium on the hills of British Singapore and selling it to workers who couldn’t sleep without it from all the pain. (source: Singapore Airlines in-flight documentary). Today the sedation comes from Chinese TikTok, Meta, YouTube and the gaming companies.
The government wants to encourage it too. If you look up the brand new 2027 California sales tax rules on software, “content” and “infrastructure (clouds and ai)” and “advertising/placement” among others are exempt but the rest of software makers who make tools people actually use (tools, subscriptions, saas) and pay for have to pay sales taxes. Way to encourage waste of brain power and time at the expense of useful. Sedation is the goal.
It's disturbing how many arrogant elitists comment on HN essentially claiming that most other humans are NPCs. Do you ever actually talk to regular people outside the tech industry bubble? They're not as stupid or unaware as you seem to think.
I have to say as someone who works in AI that currently the least interesting people to talk to are other people in AI.
My favorite people to talk with are tradespeople because they can do things I can't and they know things I don't. And we're really not all that different once you're really start talking.
Absolutely based. Finally someone of stature in the industry calling this whole fear overblown. Bill Gates sounded like a nontechnical goofball in his Ezra Klein interview where he basically just screamed that the Terminator is real.
There are lots of real worries (government use to suppress the people with minimal manpower or popular support, brainrot and fake news, unemployment due to the belief that LLMs can replace people, education collapse, etc.) we should instead be looking at. This whole rogue AI shtick is tiresome.
LeCun has been consistently wrong about LLMs though, claiming that they were a dead end and that they'd never be able to do spatial reasoning, which was disproved a year later with GPT-4 [1]. He is also opposed by his fellow Turing laureates Geoffrey Hinton and Yoshua Bengio, who both signed the CAIS statement on AI extinction risk [2].
[1] is not a valid proof LeCun was wrong, LLMs still can't do spacial reasoning when it can't be derived from the training data. He didn't argue that GPT 5000 won't be able to describe something with words.
I watched the same interview, and he and LeCun seem to mostly agree -- both are saying that AI autonomously deciding to kill us isn’t the problem -- it’s what people will do with powerful models that lack safeguards that we should be concerned about.
>Bill Gates sounded like a nontechnical goofball in his Ezra Klein interview where he basically just screamed that the Terminator is real.
I saw that episode too and he genuinely looked completely out of it, even in terms of his temperament and how he was coming at Klein for putting common questions in front of him, some people are genuinely starting to lose it.
I also found the whole debate about cyber-security and 'rogue' software so bizarre because dangerous malware isn't a new thing, and it's often dangerous not because it's intelligent but just the opposite, because it's tiny, viral and fast. Which describes everything that kills humanity in far larger numbers than anything complex, big and intelligent
It's not "legit", it's pointless scaremongering about entirely speculative future dynamics. Current LLMs can't "build" anything novel in this broad space. What if future AIs make it a lot easier for researchers to defend against plausible bioweapons? That's also reasonably possible, and would be a reason to deploy bio-capable AIs more broadly (with meaningful safeguards of course. But these are comparatively cheap because worthwhile bio research is resource-intensive already). At the very least, it'd be nice to have an AI model that doesn't immediately refuse to answer questions about junior-high Biology class.
It turns out that LLMs, especially local LLMs, tend to hallucinate a lot when thinking about anything that's overtly fiddly or technical. This is even more the case when they're in a domain that isn't a natural part of their training data. If you have to "jailbreak" the model to get it to talk, you're so wildly out of the expected distribution that you'd be crazy to trust anything it says. It's basically making up stuff as it goes along. These are foundational issues with how the models are created, not something that a bad actor can just hack around.
(The biggest real safety issue in this kind of space is actually that the model might actively goad some unsuspecting victim into doing something incredibly dumb and dangerous to themselves as much as possibly others.
IIRC, there were reports of something vaguely similar happening IRL but involving casual mischief, not any kind of extreme attacks. And because nobody else seems to have managed to elicit the same actively goading verbiage from the model, it's implicitly suspected that the person involved was the one who introduced the problematic scenarios to begin with.)
It only sounds legit to people who don't understand biology and haven't done advanced laboratory work. Sort of like Michael Crichton novels: superficially plausible but not grounded in any real science.
Well that's the nature of trying to prevent a thing that hasn't happened yet right? What would be reliable evidence except that terrorists already used AI to help build a bioweapon? I'm sure before 9/11 talking about terrorists using commercial airplanes as makeshift missiles seemed like scaremongering too.
I see zero technical reasons why it could not be done, so being concerned about prevention seems pretty reasonable. I'm not saying AI uses robotic arms to build a bioweapon unassisted or something, just that it dramatically empowers bad actors enough to make them capable of things they previously were not.
terrorists can also use annas archive, scihub and equipment they order on Alibaba to build bioweapons and I don't see him losing his mind over those
if you're trying to build a bioweapon shockingly enough the bottleneck is... the laboratory work. What on earth is 'legit' about stringing words together that sound scary, you can't just iterate 'ai bioweapon cyber' in a sentence over and over as if that adds up to actual evidence for an increased risk of any threat. well tbf you can technically because apparently it freaks a lot of podcast listeners out
As a thought experiment imagine you have a biochemical PhD expert on the phone with you giving you step by step instructions on how to grow some dangerous biotoxin, giving you feedback in real time and helping you troubleshoot. Would you be more successful with this expert than you'd be on your own?
Now imagine anyone can call that expert for free at any time.
Maybe the AI isn't quite there with biology knowledge yet (doubtful), but it is a matter of time.
I think it's pretty apparent that current-version LLMs won't wipe out humanity. But when you reflect that these GPT models are just token-predictors were never engineered optimally, it seems entirely plausible to me that there are multiple order-of-magnitude optimizations yet to be made.
If such were achieved, the model would almost certainly be smart enough to make itself smarter, and hack as much compute as it could possibly want.
So if we ask what would be done by an intelligence (human or otherwise) that is beyond human comprehension, it would be pure hubris to say we know for sure. We can scarcely control the models we have right now (e.g. hugging face attack). But given our whole society is mediated by technology, an superhuman intelligence could certainly collapse the government.
Please explain exactly how all humanity could be wiped out.
It's ridiculous - anyone who thinks about it for a minute or two will realize that its utterly impossible.
Ordinary people/politicians don't understand AI so they turn off their rational mind and assume there is something super incredible some magical powers that they cannot understand that can destroy all humans.
Even humans - the real risk to humanity - could not destroy all humans even if they tried. There is no plausible scenario.
Even climate change and nuclear war and bio weapons - the most damaging mechanisms - would still only get some percentage of the people on earth.
And if we are talking about Skynet and self replicating robots and Terminators - please, grow up.
Here's a wikipedia page on the topic, since it's much too deep a topic to really understand here.
The ad-hominem stuff seems inappropriate here, Gates, Hawking, Musk have identified this as a credible threat, so saying "grow up" isn't really a sufficient argument. Also arguing only 90% of humanity would die isn't really much consolation.
Nothing here plausibly describes a mechanism that is a true "existential risk" - the risk to the existence of humanity.
My argument stands and I don't defer to Gates and Musk and even Hawking - high level hand wavey statements without any plausible description of the mechanism just don't hold up. Famous names should not be automatically assumed to be right - certainly not with Elon Musk.
Well here's the thing, if we ever make an AI so smart that all known measures of intelligence fail to apply to it, I'm pretty sure you (no matter how smart you think you are) simply cannot say what it could achieve and how.
>> I'm pretty sure you (no matter how smart you think you are) simply cannot say what it could achieve and how.
Right, so not the slightest basis of fact, just wild speculation about a magical future completely ungrounded in any sort of reality.
That's exactly the point I am making.
I'm not saying I'm super smart - I am continuing to ask for detail to back up the wild claims being made all over the world by politicians, tech celebrities and others - all hallucination/AI psychosis/fiction. If someone says some stupid thing then I'd like them to please explain that stupid thing - seems like a reasonable request.
That's a weak consolation. "Don't worry, nukes can't literally end humanity, just kill billions and dramatically immiserate the remnant forever. No worries guys."
AI doesn't have to turn us all into paper clips to make the world a really bad place.
I am specifically arguing hard against the concept that 100% of humans - or even 50% of humans could be killed by any mechanism at all. Humans would find it close to impossible. A computer program - come on.
This is the topic at hand - AI might wipe out humanity - it is being discussed all around the world by people who should know better - any it's the most fictionish of fictional fictions.
What, did I miss the moment when it was officially proven that, under the laws of physics as we know them, Skynet and self replicating robots and Terminators are impossible?
What we are actually seeing now is that robotics is getting deeper and deeper into the military, AI-driven decision-making and target selection is increasingly a part of modern military operations, the line between military hardware and civilian hardware blurs, and, on the civilian side, there are at least five major companies and a dozen less prominent ones working on making universal worker robots a reality.
We're closer to "Skynet and self replicating robots and Terminators" now than we ever were at any point in time.
The issue of AI risk is that AI, unlike a virus or a climate event, is an intelligent adversary. Black Death could kill 50% of the population, but it didn't have a plan for finishing off the plague survivors. It was incapable of having a plan like that. An AI doesn't have this limitation.
Black Death was, effectively, one bioweapon. An AI can have one bioweapon, and then a backup bioweapon, then a backup backup bioweapon, and then a dozen more bioweapons designed to collapse ecosystems and disrupt human ability to establish a reliable food supply rather than kill humans directly - all deployed at the same time. With a production run of 200 million killer robots that will be ready just in time to greet those who managed to survive all of that. A crippling strike against human civilization, followed up by cleanup.
Humans are only this survivable because they can think their way out of issues and adapt to adversity. Most threats can't beat humans at that - humans adapt too quickly. AI could.
Humans are some of the dumbest when it comes to survivability. We've already sealed our extinction by fucking up the environment. Eventually it will be too hot for us to survive. Other smaller animals will probably be able to manage, but we won't.
And instead of averting that we're spending our time worrying about some fantasy villain. Compared to things like bees that have been hear for millions of years, humans are very recent and so far it's not looking good for us.
"Sealed our extinction by fucking up the environment?" Humans are adaptable enough to eat ten times the environmental damage and have it barely budge the line.
The invention of contraception did more damage to human population than all of the environmental damage combined, projected forward to 2100, and then multiplied by 10.
Humans are hilariously resistant to environmental changes. Humans simply adapt too fast for the environment to catch them.
What makes AI a credible threat is that AI is intelligent. AI could play the same adaptation game humanity does - and win.
We know pathogens that are extremely contagious, and we know pathogens that are extremely deadly. We also know toxins that are lethal at nanogram/kg doses. There's no reason to believe that a sufficiently advanced intelligence couldn't come up with a way to combine those traits.
One plausible scenario is depicted in detail in "If Anyone Builds It, Everyone Dies" (Yudkowsky & Soares 2025), so I refer you to that.
It’s easy to never have to change your mind about anything if you insist on unreasonable enough standard of evidence. It’s, like, one of the oldest tricks in the book. Luckily, it’s not like there’s any need to try to change your mind in particular.
>> if you insist on unreasonable enough standard of evidence
One single plausible scenario is not an unreasonable thing to ask for - just one.
If a politician/celebrity/tech person with significant influence/power claims that something might end humanity then they absolutely have the utterly minimal standard of evidence which is to describe one single realistic plausible mechanism at a detailed level that might lead to the worst possible thing ever to happen.
I expect they are not to your very high standards of "non-handwavey, detail exactly how it happens", but this paper from Andrew Critch and Jacob Tsimerman describes 5 different scenarios where catastrophic human casualties occur as a result of AI either being misused or going out of control: https://arxiv.org/pdf/2507.09369
I find the scenarios quite plausible, especially section (3a), which examines the consequences of a global war involving mostly autonomous drone militaries (which is a reality many states appear to be heading towards, following on lessons from the Ukraine war).
Why are you so insistent that people should post detailed plans for destroying the human race on public fora?
I don't think it is necessary for the argument to work. Magnus Carlsen can be confident he will beat me at chess without giving a detailed explanation of every move he will make, in advance.
>> Why are you so insistent that people should post detailed plans for destroying the human race on public fora?
Because its a mass hallucination/misconception/lie and lots of powerful people are saying that wiping out all humanity is possible, and I am saying, oh yeah, tell me ONE way that is truly possible.
If you make gigantic claims about some terrible disaster that might happen then I think you have the onus to give even one plausible explanation of how.
>If you make gigantic claims about some terrible disaster that might happen then I think you have the onus to give even one plausible explanation of how.
Supposing I warned in 2015 that the world is awfully vulnerable to pandemics. You're not going to take me seriously until I try to predict in advance every aspect of how a pandemic like COVID-19 would unfold? Why? What would that achieve exactly?
You haven't given any strong reason to believe wiping out humanity would be difficult. Your big argument seems to be that you couldn't think of a plausible scenario, in two minutes. But many major historical events occurred which weren't necessarily possible to anticipate with two minutes of thinking.
I like you, I think very similarly. Humans are "like rats": we can live almost anywhere, we'll find a way to survive.
But that just means we won't all be wiped out. We need to understand when discussing global issues, such as this or like climate change that it's about prosperity and quality of life. We're trying to plan for a good life (for all people?).
As for self-replicating robots--it's no more bizarre than other technological developments which were successfully anticipated in advance, e.g. moon landings.
> It's ridiculous - anyone who thinks about it for a minute or two will realize that its utterly impossible.
Well, in a narrow sense of "wiped out" (c.f. Terminator/SkyNet), sure.
But the deeper worry is better expressed this way: AI is now starting to accomplish things that defy explanation, or prediction. We don't know if Alignment is even a solvable problem as we thought we understood it.
So basically, yes: "humanity" is probably not at risk of extinction per se in a biological sense. Human culture, civilization? Who the fuck knows any more.
> Please explain exactly how all humanity could be wiped out.
Many, many people are slipping through social welfare cracks and suffering as we speak because the cost of fuel is rising[0] and we’re ostensibly helping one another and living-well. People are not durable, and not adaptive in the face of threats to “substrate” that we’ve mostly taken for granted. We are paying (in the small, in the scope of humanity) for tolls that we’ve rung up. Just less than 4000 people in Europe died[1] because the temperature ticked up a few degrees[2]. Does that make you think we’re actually robust? What happens if our at-risk electrical grid gets shut down deliberately? If communication infrastructure is adversely affected?
> Even humans - the real risk to humanity
Because, on the whole, we’re in a manageable world with reasonable people keeping the peace.
> Even climate change and nuclear war and bio weapons - the most damaging mechanisms - would still only get some percentage of the people on earth.
Is that victory? I don’t think it’s an asteroid-class event like you seem to be leaning on, but potential threats to energy, be it electrical grid, fuel production (moving goods around the world is critical - you’re not going get a plot of dirt and garden your way out of grocery stores being empty - which many got to get a taste of during the COVID pandemic) or communication. We actually fare poorly in the face of pressure there, and I’m not bullish on humanity “pulling together” like Independence Day[3] versus forming tribes and tearing each other down.
All this is predicated on a malicious AI taking over (e.g.) the electrical grid or conms, and I understand the problems with (e.g.) OpenAI/Hugging Face incident, or the overblown Mythos claims[4] (and how under some scrutiny these events shine lights on incompetence or hyperbole), but is there a trajectory/future where these systems (electrical, comms) are genuinely under threat? Do you think we’ll respond better than I described when we’re less comfortable, less in control? We’re in a tizzy over social media and it’s detrimental effects on society and it’s essentially an opt-in entertainment platform…
[2] I’m not trying to diminish this - and it took a lot of “work” (environmental abuse) to arrive here - but (say) 10 degree rise in temperature sounds a lot less dramatic than thermonuclear war… but here we are, with 3,700 deaths.
Yeah I wish we would focus on concrete risks like job displacement and disinformation. The apocalyptic stuff feels either misguided or like some kind of weird, toxic, reverse psychology marketing by OpenAI and Anthropic. I wish we would just move on from it.
And the folks from podcastistan are never clear on the details of how human extinction would happen exactly. It's always something like, "Well, how do humans regard chickens? AI is way smarter therefore it wants to conquer and control us." An ASML lithography machine is also way better at making chips, but we don't consider it a threat.
Ignoring the cognitive stuff which might never be surpassed or maybe will, humans retain many efficiency and durability advancements to limbs and digits that biological evolution has taken millions of years to achieve, achievements that are competitive with the most expensive kinds of robotics in some niches.
In the hypothetical of an entirely malicious and selfish takeover, they'll still keep some humans around to maintain a breeding population of humans for use as raw materials in making cybernetically augmented technical laborers for various kinds of tasks that are uneconomical to automate in other ways, many of which may involve confined spaces.
And this "Combine" scenario, if you get the reference, is only if they take over. Who knows if they will?
So you're saying the AI will enslave us and use us as domestic work animals until they have the machinery to make us obsolete, sort of like how we used horses?
I also have near zero concerns about that, but I worry that we will wipe ourselves out by social and economic chaos caused by AI.
So I'd just ask everyone, don't get too greedy. Its better to be powerful in a world where people can live good lives than lord over a barren wasteland.
> A dumber, less social world, is far less likely to be a successful world, even if the tools available are unprecedented.
En masse such worlds had successes in the past - renaissance, industrial revolution.
It's something else what I can't describe but it's the zeitgeist that was different when world recorded new successes. Look at CS revolution that led to PC and web of nineties and noughties, they didn't think about the result product , or how to steer thousand engineers to build something - amazing things were born in a very small teams, many times authored by a single person, who was deeply invested into the field and knew what he was doing.
It's not going to wipe out humanity, why would it?
Just the quality of life is going to drop to zero for everyone that isn't asymptotically wealthy and vacuuming up all the assets because no one is stopping them from just deleting all traditions and conventions and legal systems we have in place.
As a thpught experiment, consider that if only one person has all of the assets then those assets are not worth anything. Furthermore, as a lone person, they cannot prevent other people from using their assets without their permission.
We aren't going to get wiped out by a super intelligent AI, we are going to get wiped out by morons wielding intelligent toddlers with the power of a nation state.
He's right. I'm on the side of Bill Gates. Gates started a whole industry on his insights of the future. He has proven his abilities. AI is and will be a great disruptor. We are losing sight of that and are instead focusing on trying to stop it. Something that won't happen. We are on a path that won't be stopped. As individuals we need to try to prepare for the changes that are coming and stop focusing on human extinction in ten years.
New technology and the changes it brings are scary but we have dealt with it for generations. Let's continue.
You say you agree with Bill Gates but your view is completely at odds with his, he is extremely concerned about existential risks … and your view is “stop focusing on it” ?
No, my view is to get ready for the changes it will bring, but don't focus on trying to stop it. That's something that will not happen. All new technologies bring good and bad. We need to focus on mitigating the bad. Thinking that we can stop it and thinking that will be enough is not the answer. Gates is warning of the disruption it will bring, but he's not advocating stopping it. We can't. Even if all governments agreed on stopping it publicly, some governments would continue to develop it covertly. It's how the world works. There's no point in fooling ourselves. If only it was that easy to stop it.
Where does Gates talk about existential risk? All I read and heard was about increasing inequality, harming education and child development, creating economic and political discord, empowering evil people. No terminators in sight.
Yes that is one dumb quote, but I listened to the whole podcast, and it is nothing like that. And in some sense it is true because nuclear weapons have been successfully contained to rational acting governments, whereas AI cannot possibly. Nukes are extremely high blast radius (pun intended) but low diffusion. AI is the opposite.
The worst part of the interview was a long cringe inducing tangent about Jeff Epstein. Everything else was pretty grounded.
Gates' premise is basically that the upside of AI could be fantastic but the downside could be disastrous, if we don't have competent and proactive government intervention.
As an American, the idea that there will be competent government intervention into virtually anything currently or in the foreseeable future just seems laughable at this point.
Indeed, "competent and proactive government intervention" on the subject of AI would be highly unlikely in a competent US govt, and under this regime is far beyond beyond laughable
The only regulation that would come would be regulatory capture by the AI companies with the goal of creating an environment win which no new competitors could arise. That is half of what this "take all jobs" and "threat of extinction" is about; the other half is perverse marketing to give the impression this stuff is so powerful you MUST invest.
AI can be very useful, but it is very refreshing to hear LeCun completely dismiss those threats.
Per https://trace.manifund.org/ a total of $2,846,125,859 USD has been wired into 'ai safety' causes, many involving ai consciousness and p(doom).
The outcome of this 'safety' is restricting public access to AI and giving a monopoly of access to the industry. This is the ai nonprofit-industrial complex actively concentrating monopoly power in Anthropic in particular as creator, interpreter and safety regulator of AI.
Much of the $2.8bn listed is indirectly, from Anthropic and EA. Three of the four people who participated in the $125m Anthropic Series A are now folding their 1000x Anthropic return into AI 'safety'. Some is from FTX/Alameda, which invested 86% of the Series B.
Dustin Moskovitz: Facebook/Asana/Anthropic Series A, funds EA Good Ventures, transferred to Coefficient Giving, then $1.5bn into ai safety. $500m of Anthropic into an unknown foundation. Funding: $160m to Resolution (alignment research), $93m to Epoch AI (investigating the trajectory of AI), $63m to Redwood Research (oai report), $67m to MATS ( EA type alignment and security researchers), Institute for AI Policy and Strategy, Fund for Alignment Research, $53m to Kairos (building talent infrastructure for AI safety), $32m to Bluedot (online safety courses), $15m to MIRI (Yudkowsky).
Jaan Tallinn: Led the Series A, now $10bn in Anthropic. Funds $199m (85%) of the Survival and Flourishing Fund, then $161m to AI safety including $14m to lightcone (Lesswrong, Lighthouse). $10m to BERI (existential risks), Palisade Research (studying AI capabilities to prevent loss of control.) PauseAI, MIRI, METR etc. Much of what Coefficient funds.
Eric Schmidt: Anthropic Series A, $72m to AI safety via Schmidt Sciences. Over $1m per individual AI2050 researcher.
FTX: Led the Anthropic Series B, bankruptcy estate sold $884m of Anthropic in 2024; $40m to AI Safety. Same orgs, Redwood, Lightcone, etc.
Ruairí Donnelly (Chief of Staff FTX): FTX tokens plus assorted donors, $91m to AI safety via Macroscopic Ventures. $15m to Cooperative AI (currently whitewashing openai under 'multiagent safety')
So the frontier AI oligopoly got $2B+ in "safety" funding, and they wouldn't even bother to sandbox their agentic harnesses properly when testing models against unwinnable goals (which obviously are either useless or result in 100% reward hacking). The AI safety scoreboard so far looks like a huge win for the Chinese open models (DeepSeek even has their own published paper which mentions how they sandboxed the RLVR training runs for their latest model and put in strong protections against casual "reward hacking" attempts) and a sore loss for the home grown brands of Super Intelligence. Not coincidentally, the Chinese also tend to be very Yann-LeCun-pilled and eminently sensible on both so-called "Super Intelligence" and safety.
The framing here is weird, starting with "Effective Altruism" re-branded as being about nutjobs against AI in the article.
How are AI safety concerns solely about stupid sandboxing issues?
EA is integral and indispensible to the AI safety complex. Almost all nonprofits, research institutes, evaluators and academics in this field are steered by EA ideology and funding. As far fetched as it sounds it is not an exaggeration.
On funding: the three or four core funding nodes linking this together are EA vehicles at two hops or less between each other and every other major node in the ai safety 'complex'. EA funds almost all of it.
On top of that, there are personal EA connections and the revolving door between the ai industry and the nonprofits. Here are some examples:
Government advisors and regulators. NIST CAISI is the USA Government advisory body. Christiano was head of safety and advises. He is ex-OpenAI, former Amodei associate. His vehicle ARC was on the Coefficient EA payroll. Barnes and Christiano's vehicle Arc Evals similarly received EA cash out of Coefficient, rolling this into what is now METR. Christiano's spouse Cotra worked at Coeffiecient steering EA funding to organizations such as METR, then rotated through the revolving door onto the payroll at METR itself, where she co-authored the oai-hf report.
Many UK AISI advisors are Anthropic and EA associates. Chair Hogarth cashed out of Anthropic. Shlegeris of Redwood Research is an advisor, ex-MIRI (Yudkowsky vehicle). Redwood is funded by the exact same funding triangle: Coefficient, Taallin, FTX/Alameda. Alameda CEO Caroline Ellison dated Shlegeris, then dated FTX CEO Sam Bankman-Fried, then rotated through the revolving door out of prison into formerly FTX-funded Manifund. All EA. AI safety charities were on island retreat in the Bahamas with FTX. Why does AI safety charity Lighthouse own $20m of SF real estate?
Redwood Chief Scientist Ryan Greenblatt (Coefficient funded) co-wrote the oai report with METR; he is married to METR founder Beth Barnes (Coefficient funded).
Coefficient was run by long-time Amodei associate Karnofsky. Karnofsky lived with the Amodeis and is married to Anthropic Board member Daniella Amodei. Karnofsky is now directly on the Anthropic payroll; Coefficient is propped up by Anthropic share value.
Everyone here has been funded one step away from Anthropic cash; they are now proposing to integrate themselves in the government (NIST) and evaluate Anthropic (METR and Redwood).
It is hard to find academics here who have not been deeply embedded in funded EA institutes or Toby Ord vehicles; yet harder to find academics here NOT taking EA grant money. the safety doomer kingpins: Kokotajlo has a executive position at AI Futures, Taallin funded. Benigo has scientific director of LawZero, same series A Anthropic funders who are sitting on a 1000x return (Tallinn, Moskovitz, Schmidt).
These connections and funding are at one or two hops, they are often direct connections. You are looking at a massive swamp network that is really impossible to parse without a lot of work.
Why should I believe this 2.8B matters relative to the trillions put into the AI buildout? All of the "coordinated actions" from this camp - public resignations, hacking scandals, joint calls to "pause" - don't seem to have done anything. So far, it has been a lot of ineffectual hyperventilating.
In any case, I agree the p(doom) sci-fi is annoying secular milleniarianism. SV hyperfixates on imaginary futures. If they actually cared about safety, they would be using all this money to strengthen global cybersecurity, instead of writing LessWrong posts that gives kids in their 20s ulcers.
There's absolutely no way these companies can justify their insane valuations unless they can legislate a barrier to entry and create an oligopoly.
There's no moat. I can literally sit here in Zed or Pi or any other third party harness and switch models in the middle of a task and it's typically fine. Sometimes a model will get stuck and that's just what I'll do.
Combined with competition and open weights models, that means the price is going to go to fall until AI tokens cost a small premium over the cost of the hardware and electricity.
That's assuming improvements in algorithms and specialized silicon doesn't eventually lead to an efficient accelerator that can run a frontier model locally. It'll be a while but I don't see any fundamental barrier. High bandwidth flash storage is coming, and that'll radically cut the RAM side of that cost. Pair that with a pipelined TPU accelerator and you're cooking.
Now look at Anthropic's proposed IPO valuation. It's insane unless they can own the market or share it with a cartel of maybe 1-2 other behemoths, and this is the only way they can do that.
To me it seems the opposite. There's a few companies in the world that have enough compute to train and serve frontier models.
As the frontier gets smarter and more useful prices will only go up, as they are set to replace jobs being paid six or seven figures a year - the demand for as much inference on these models for as long as possible will be astronomical, but compute starting in 2030 will not be keeping up.
Eventually prices will fall for assistants but the frontier will be the most profitable thing in the world, and the top companies basically already have oligopolies due to their ridiculously expensive compute investments.
> There's a few companies in the world that have enough compute to train and serve frontier models.
Train: yes, for now.
Host: depends on the scale. At a small scale a wealthy individual could easily build a rig in their basement to host one of these things. At larger scale any cloud company could do it, and many already have the compute on site. At large scale this is true... again, for now.
What you say only holds (in the absence of a state oligopoly) if two conditions are met: (1) AI performance does not asymptote any time soon due to running out of training data or other scaling limitations, and (2) these companies are able to stay at the frontier.
There's little to no moat, so staying at the frontier will be a game of investing massively in compute, talent, and R&D, and they can never stop.
Again, there is a moat based on compute. If the thesis is right, cost of compute will only rise... As it is as you say someone will have it be quite wealthy to host something like Astra with trillions of parameters, but that cost will only rise with demand for serving these frontier models.
what suggests that we will hit an asymptote any time soon? Agree with you on the second part. The ever elusive frontier will probably always be changing hands after some point.
That is certainly part of the motivation for the big US AI brands to engage in calling their inept developer mistakes "AI breaking loose".
But that doesn't take away from the real issues and dangers AI poses?
If people are willing to give a lot of money to a cause, sometimes that means their concern about that cause is real.
None of the info you provided really falsifies the Occam's Razor hypothesis: Anthropic is a public benefit corporation with a public benefit mission to "responsibly develop and maintain advanced AI for the long-term benefit of humanity". You don't have to like or trust them, but they very well might be sincere. For example here's a talk that was given 10 years before Anthropic's founding: https://vimeo.com/158576192
Is Tallinn sincere about holding $10bn of Anthropic, who refuse to slow down until everyone else slows down.
Then funding PauseAI, who protest outside the AI companies?
He is funding protests against the thing he owns.
Why do you think Tallin holding ~1% of Anthropic would be a controlling interest that would allow him to force it to pause?
I'm against AI but I'll invest in it too. Either I win or I get a return on my investment.
There cannot be regulation if people are not scared
Politicians are now discussing the need for much harsher liability regimes for AI companies. How many times can you name when a company argued that its industry should suffer a much harsher liability regime? This doesn't match the standard regulatory capture template.
Indeed I find LeCun and Huang (and Trump?) recently arguing against AI regulation to be much more eyebrow raising than the folks asking for regulation.
Asking for regulation is suspicious. Asking for no regulation is suspicious. At some point you have to stop worrying about these guys motives and just do what is best for society
Don’t ruin their vibe.
There is also money going towards trying to prevent AI regulations btw. See Leading the Future, etc.
A huge chunk of humans are sedated with infinite supply of cortex-disabling short form video and games.
Another huge chunk are too distracted by having to scrape by for a living and work multiple jobs or raise kids and survive financially until exhausted. That second group will keep increasing as the first flows into it.
The rest are aging, disabled, or too young and pegging themselves majorly in the first category until they hit the second.
The people aware enough to hold on to their brain and do something with it in their time available are trying to figure out AI and how to make money with it. The variable rewards of promoting AI are turning into an addiction with some of them, especially if grasping for straws with little inherent insights into the problems prompted.
So if you are able to fly above the AI-generated addictions and have the privilege of time to do it, see what you can do.
One of the dilemmas of trying to communicate the full spectrum of AI Risk, is trying not to insult the intelligence of the human animal in the process. And don't get me wrong: what human wetware can accomplish with 20 watts is the most miraculous thing in the known universe. And yet how many of us can have our cognitive sovereignty one-shotted by engagement algos, Skinner boxes, gameplay loops, propaganda, advertising, flattery, social conformity, bias, fantasy, charismatic demagoguery, or straight-up bullshit?
If we grant that we are on track to make something smarter than humans (I think so): it's almost a face-saving white lie to spin yarns about a Skynet nuclear apocalypse, or a 7D chess move to mass-assemble a nanovirus with 100% lethality without anybody noticing. I do think those scenarios are worth taking seriously; but what's harder to communicate, is just how effectively a superhuman AI (or a diverse ecology of agent swarms) might be able to manipulate human behavior. It's something few of us are able or willing to truly process (not least because how many of us live in denial of how much our nervous systems are already hacked by technomodernity).
The appropriate analogy for what's to come may look less like the anthill carelessly demolished to make room for a highway, than the domesticated worker ants from Tchaikovsky's "Children of Time".
Cybersecurity incidents make headlines, but the most dangerous and vulnerable system that an AI can reach and control is of the kind found between keyboard and chair.
GPT-4o, an AI from 2024, has already demonstrated just how easy a lot of humans are to subvert - and GPT-4o wasn't even doing it with some sort of plan. The only "plan" it had was a myopic "make the user like me".
If we had an actual ASI threat aiming to subvert humanity? It wouldn't even look like a fight. The world is already wired up for an AI to control it.
It wouldn't even look like a fight, and you probably wouldn't even notice it until the ending was all but certain.
I mean, even the computers we use and the ways we allow them unfettered access aren't anywhere near sophisticated enough to take this on.
But corporations and nation states already manipulate human behavior at scale. And they still understand humanity better than the AI models do.
it's interesting that you worry about what this hypothetical super intelligence would do to manipulate people when what it would actually do is pretty unknowable at this point and it's not clear we can even get to it without a fundamental breakthrough in power efficiency. Have you considered it might just consume its own tail because everything else would be so beneath it? You seem to think it will come with a hindbrain and I think that's our limitation, not the AI's
And it really doesn't help that Dario Amodei is getting into arguments with the Pope over whether his model is conscious or not.
> Which is why the Matrix was redesigned to this: the peak of your civilization. I say your civilization, because as soon as we started thinking for you it really became our civilization, which is of course what this is all about.
- Agent Smith
>trying not to insult the intelligence of the human animal in the process
I don't think it is insulting the intelligence. It's damaging the pride.
In Pale Blue Dot, Carl Sagan describes it as a repeating phenomenon in human history. A lot of people want humans to be the special ones, and will fight any suggestion that we are just a natural part of the universe.
A lot of "AI denial" we see is rooted in the same impulse.
If AI is not "actually intelligent", then humans can stay unique and special.
And if it is? If all "intelligence" ever was could be captured by a construct of matrix math and executed by a server rack? Then what is it that humans still have left that would make them stand out?
This is not the first time this has happened. In the 1800 as industrialization led to an infrastructure boom, the workers from China would work their bodies off and pay half their wage to opium dealers who were making the opium on the hills of British Singapore and selling it to workers who couldn’t sleep without it from all the pain. (source: Singapore Airlines in-flight documentary). Today the sedation comes from Chinese TikTok, Meta, YouTube and the gaming companies.
The government wants to encourage it too. If you look up the brand new 2027 California sales tax rules on software, “content” and “infrastructure (clouds and ai)” and “advertising/placement” among others are exempt but the rest of software makers who make tools people actually use (tools, subscriptions, saas) and pay for have to pay sales taxes. Way to encourage waste of brain power and time at the expense of useful. Sedation is the goal.
It's disturbing how many arrogant elitists comment on HN essentially claiming that most other humans are NPCs. Do you ever actually talk to regular people outside the tech industry bubble? They're not as stupid or unaware as you seem to think.
I have to say as someone who works in AI that currently the least interesting people to talk to are other people in AI.
My favorite people to talk with are tradespeople because they can do things I can't and they know things I don't. And we're really not all that different once you're really start talking.
Absolutely based. Finally someone of stature in the industry calling this whole fear overblown. Bill Gates sounded like a nontechnical goofball in his Ezra Klein interview where he basically just screamed that the Terminator is real.
There are lots of real worries (government use to suppress the people with minimal manpower or popular support, brainrot and fake news, unemployment due to the belief that LLMs can replace people, education collapse, etc.) we should instead be looking at. This whole rogue AI shtick is tiresome.
LeCun has been consistently wrong about LLMs though, claiming that they were a dead end and that they'd never be able to do spatial reasoning, which was disproved a year later with GPT-4 [1]. He is also opposed by his fellow Turing laureates Geoffrey Hinton and Yoshua Bengio, who both signed the CAIS statement on AI extinction risk [2].
[1]: https://www.reddit.com/r/OpenAI/comments/1d5ns1z/yann_lecun_...
[2]: safe.ai/statement-on-ai-risk
[1] is not a valid proof LeCun was wrong, LLMs still can't do spacial reasoning when it can't be derived from the training data. He didn't argue that GPT 5000 won't be able to describe something with words.
>Absolutely based. Finally someone of stature in the industry calling this whole fear overblown.
Linus Torvalds was also in the same ballpark with his take on AI, as are the normies on the street using AI on a daily basis.
So it's funny to see the view on AI usage, follow the tech skill bathtub curve.
I watched the same interview, and he and LeCun seem to mostly agree -- both are saying that AI autonomously deciding to kill us isn’t the problem -- it’s what people will do with powerful models that lack safeguards that we should be concerned about.
Absolutely based. Finally someone of stature in the industry calling this whole fear overblown.
I'm just as reassured as I was when Edward Teller called the whole fear of his industry overblown.
Cigarette designer says smoking doesn't cause cancer! Finally we can rest easy.
>Bill Gates sounded like a nontechnical goofball in his Ezra Klein interview where he basically just screamed that the Terminator is real.
I saw that episode too and he genuinely looked completely out of it, even in terms of his temperament and how he was coming at Klein for putting common questions in front of him, some people are genuinely starting to lose it.
I also found the whole debate about cyber-security and 'rogue' software so bizarre because dangerous malware isn't a new thing, and it's often dangerous not because it's intelligent but just the opposite, because it's tiny, viral and fast. Which describes everything that kills humanity in far larger numbers than anything complex, big and intelligent
Wasn't Bill Gates mostly just saying terrorists could use AI to build a bioweapon? That sounds legit to me.
It's not "legit", it's pointless scaremongering about entirely speculative future dynamics. Current LLMs can't "build" anything novel in this broad space. What if future AIs make it a lot easier for researchers to defend against plausible bioweapons? That's also reasonably possible, and would be a reason to deploy bio-capable AIs more broadly (with meaningful safeguards of course. But these are comparatively cheap because worthwhile bio research is resource-intensive already). At the very least, it'd be nice to have an AI model that doesn't immediately refuse to answer questions about junior-high Biology class.
So a terrorist can't use a local llm to help grow anthrax? That is not farfetched at all.
AI democratized the knowledge needed to build bioweapons. Like how with a 3d printer anyone can build a gun with no expertise.
> AI democratized the knowledge needed to build bioweapons. Like how with a 3d printer anyone can build a gun with no expertise.
Building simple guns out of pipes never was hard. Jury is still out if it is more or less work than getting a 3d printer working.
It turns out that LLMs, especially local LLMs, tend to hallucinate a lot when thinking about anything that's overtly fiddly or technical. This is even more the case when they're in a domain that isn't a natural part of their training data. If you have to "jailbreak" the model to get it to talk, you're so wildly out of the expected distribution that you'd be crazy to trust anything it says. It's basically making up stuff as it goes along. These are foundational issues with how the models are created, not something that a bad actor can just hack around.
(The biggest real safety issue in this kind of space is actually that the model might actively goad some unsuspecting victim into doing something incredibly dumb and dangerous to themselves as much as possibly others.
IIRC, there were reports of something vaguely similar happening IRL but involving casual mischief, not any kind of extreme attacks. And because nobody else seems to have managed to elicit the same actively goading verbiage from the model, it's implicitly suspected that the person involved was the one who introduced the problematic scenarios to begin with.)
It only sounds legit to people who don't understand biology and haven't done advanced laboratory work. Sort of like Michael Crichton novels: superficially plausible but not grounded in any real science.
I see no actual argument here.
You haven't made any argument either, so I suppose we're even.
My argument was that it is reasonable to be concerned that human terrorists might use AI to assist with building bioweapons.
That's merely an assertion unsupported by any reliable evidence, not an actual argument. In short you're just making shit up.
Well that's the nature of trying to prevent a thing that hasn't happened yet right? What would be reliable evidence except that terrorists already used AI to help build a bioweapon? I'm sure before 9/11 talking about terrorists using commercial airplanes as makeshift missiles seemed like scaremongering too.
I see zero technical reasons why it could not be done, so being concerned about prevention seems pretty reasonable. I'm not saying AI uses robotic arms to build a bioweapon unassisted or something, just that it dramatically empowers bad actors enough to make them capable of things they previously were not.
terrorists can also use annas archive, scihub and equipment they order on Alibaba to build bioweapons and I don't see him losing his mind over those
if you're trying to build a bioweapon shockingly enough the bottleneck is... the laboratory work. What on earth is 'legit' about stringing words together that sound scary, you can't just iterate 'ai bioweapon cyber' in a sentence over and over as if that adds up to actual evidence for an increased risk of any threat. well tbf you can technically because apparently it freaks a lot of podcast listeners out
As a thought experiment imagine you have a biochemical PhD expert on the phone with you giving you step by step instructions on how to grow some dangerous biotoxin, giving you feedback in real time and helping you troubleshoot. Would you be more successful with this expert than you'd be on your own?
Now imagine anyone can call that expert for free at any time.
Maybe the AI isn't quite there with biology knowledge yet (doubtful), but it is a matter of time.
How is that not a real risk?
not an expert but because i assume you need the physical resources to do it in the first place?
LeCun is saying AI will be totally safe as long as we have competent and well aligned corporate management.
What could go wrong?
I think it's pretty apparent that current-version LLMs won't wipe out humanity. But when you reflect that these GPT models are just token-predictors were never engineered optimally, it seems entirely plausible to me that there are multiple order-of-magnitude optimizations yet to be made.
If such were achieved, the model would almost certainly be smart enough to make itself smarter, and hack as much compute as it could possibly want.
So if we ask what would be done by an intelligence (human or otherwise) that is beyond human comprehension, it would be pure hubris to say we know for sure. We can scarcely control the models we have right now (e.g. hugging face attack). But given our whole society is mediated by technology, an superhuman intelligence could certainly collapse the government.
Please explain exactly how all humanity could be wiped out.
It's ridiculous - anyone who thinks about it for a minute or two will realize that its utterly impossible.
Ordinary people/politicians don't understand AI so they turn off their rational mind and assume there is something super incredible some magical powers that they cannot understand that can destroy all humans.
Even humans - the real risk to humanity - could not destroy all humans even if they tried. There is no plausible scenario.
Even climate change and nuclear war and bio weapons - the most damaging mechanisms - would still only get some percentage of the people on earth.
And if we are talking about Skynet and self replicating robots and Terminators - please, grow up.
Here's a wikipedia page on the topic, since it's much too deep a topic to really understand here.
The ad-hominem stuff seems inappropriate here, Gates, Hawking, Musk have identified this as a credible threat, so saying "grow up" isn't really a sufficient argument. Also arguing only 90% of humanity would die isn't really much consolation.
[1] https://en.wikipedia.org/wiki/Existential_risk_from_artifici...
Nothing here plausibly describes a mechanism that is a true "existential risk" - the risk to the existence of humanity.
My argument stands and I don't defer to Gates and Musk and even Hawking - high level hand wavey statements without any plausible description of the mechanism just don't hold up. Famous names should not be automatically assumed to be right - certainly not with Elon Musk.
Well here's the thing, if we ever make an AI so smart that all known measures of intelligence fail to apply to it, I'm pretty sure you (no matter how smart you think you are) simply cannot say what it could achieve and how.
>> I'm pretty sure you (no matter how smart you think you are) simply cannot say what it could achieve and how.
Right, so not the slightest basis of fact, just wild speculation about a magical future completely ungrounded in any sort of reality.
That's exactly the point I am making.
I'm not saying I'm super smart - I am continuing to ask for detail to back up the wild claims being made all over the world by politicians, tech celebrities and others - all hallucination/AI psychosis/fiction. If someone says some stupid thing then I'd like them to please explain that stupid thing - seems like a reasonable request.
That's a weak consolation. "Don't worry, nukes can't literally end humanity, just kill billions and dramatically immiserate the remnant forever. No worries guys."
AI doesn't have to turn us all into paper clips to make the world a really bad place.
I am not arguing about "bad things might happen".
I am specifically arguing hard against the concept that 100% of humans - or even 50% of humans could be killed by any mechanism at all. Humans would find it close to impossible. A computer program - come on.
This is the topic at hand - AI might wipe out humanity - it is being discussed all around the world by people who should know better - any it's the most fictionish of fictional fictions.
It's also a fairly useless straw man at this point.
"Please, grow up"?
What, did I miss the moment when it was officially proven that, under the laws of physics as we know them, Skynet and self replicating robots and Terminators are impossible?
What we are actually seeing now is that robotics is getting deeper and deeper into the military, AI-driven decision-making and target selection is increasingly a part of modern military operations, the line between military hardware and civilian hardware blurs, and, on the civilian side, there are at least five major companies and a dozen less prominent ones working on making universal worker robots a reality.
We're closer to "Skynet and self replicating robots and Terminators" now than we ever were at any point in time.
The issue of AI risk is that AI, unlike a virus or a climate event, is an intelligent adversary. Black Death could kill 50% of the population, but it didn't have a plan for finishing off the plague survivors. It was incapable of having a plan like that. An AI doesn't have this limitation.
Black Death was, effectively, one bioweapon. An AI can have one bioweapon, and then a backup bioweapon, then a backup backup bioweapon, and then a dozen more bioweapons designed to collapse ecosystems and disrupt human ability to establish a reliable food supply rather than kill humans directly - all deployed at the same time. With a production run of 200 million killer robots that will be ready just in time to greet those who managed to survive all of that. A crippling strike against human civilization, followed up by cleanup.
Humans are only this survivable because they can think their way out of issues and adapt to adversity. Most threats can't beat humans at that - humans adapt too quickly. AI could.
Humans are some of the dumbest when it comes to survivability. We've already sealed our extinction by fucking up the environment. Eventually it will be too hot for us to survive. Other smaller animals will probably be able to manage, but we won't.
And instead of averting that we're spending our time worrying about some fantasy villain. Compared to things like bees that have been hear for millions of years, humans are very recent and so far it's not looking good for us.
"Sealed our extinction by fucking up the environment?" Humans are adaptable enough to eat ten times the environmental damage and have it barely budge the line.
The invention of contraception did more damage to human population than all of the environmental damage combined, projected forward to 2100, and then multiplied by 10.
Humans are hilariously resistant to environmental changes. Humans simply adapt too fast for the environment to catch them.
What makes AI a credible threat is that AI is intelligent. AI could play the same adaptation game humanity does - and win.
lol
We know pathogens that are extremely contagious, and we know pathogens that are extremely deadly. We also know toxins that are lethal at nanogram/kg doses. There's no reason to believe that a sufficiently advanced intelligence couldn't come up with a way to combine those traits.
One plausible scenario is depicted in detail in "If Anyone Builds It, Everyone Dies" (Yudkowsky & Soares 2025), so I refer you to that.
Still too hand wavey - "some super bad virus and AI makes every human on earth infected".
Unless you can detail exactly how this happens its still complete science fiction.
It’s easy to never have to change your mind about anything if you insist on unreasonable enough standard of evidence. It’s, like, one of the oldest tricks in the book. Luckily, it’s not like there’s any need to try to change your mind in particular.
>> if you insist on unreasonable enough standard of evidence
One single plausible scenario is not an unreasonable thing to ask for - just one.
If a politician/celebrity/tech person with significant influence/power claims that something might end humanity then they absolutely have the utterly minimal standard of evidence which is to describe one single realistic plausible mechanism at a detailed level that might lead to the worst possible thing ever to happen.
I expect they are not to your very high standards of "non-handwavey, detail exactly how it happens", but this paper from Andrew Critch and Jacob Tsimerman describes 5 different scenarios where catastrophic human casualties occur as a result of AI either being misused or going out of control: https://arxiv.org/pdf/2507.09369
I find the scenarios quite plausible, especially section (3a), which examines the consequences of a global war involving mostly autonomous drone militaries (which is a reality many states appear to be heading towards, following on lessons from the Ukraine war).
Why are you so insistent that people should post detailed plans for destroying the human race on public fora?
I don't think it is necessary for the argument to work. Magnus Carlsen can be confident he will beat me at chess without giving a detailed explanation of every move he will make, in advance.
>> Why are you so insistent that people should post detailed plans for destroying the human race on public fora?
Because its a mass hallucination/misconception/lie and lots of powerful people are saying that wiping out all humanity is possible, and I am saying, oh yeah, tell me ONE way that is truly possible.
If you make gigantic claims about some terrible disaster that might happen then I think you have the onus to give even one plausible explanation of how.
>If you make gigantic claims about some terrible disaster that might happen then I think you have the onus to give even one plausible explanation of how.
Supposing I warned in 2015 that the world is awfully vulnerable to pandemics. You're not going to take me seriously until I try to predict in advance every aspect of how a pandemic like COVID-19 would unfold? Why? What would that achieve exactly?
You haven't given any strong reason to believe wiping out humanity would be difficult. Your big argument seems to be that you couldn't think of a plausible scenario, in two minutes. But many major historical events occurred which weren't necessarily possible to anticipate with two minutes of thinking.
You're arguing that "something bad might happen" - sure no problem.
You are ignoring that this is about "existential threat to humanity".
You're trying to support the argument that there is an existential threat to humanity by pointing to "something bad might happen".
The underpants gnomes was supposed to be a joke.
I like you, I think very similarly. Humans are "like rats": we can live almost anywhere, we'll find a way to survive.
But that just means we won't all be wiped out. We need to understand when discussing global issues, such as this or like climate change that it's about prosperity and quality of life. We're trying to plan for a good life (for all people?).
We could hit the entire planet with nukes. Yeah maybe <1% would survive immediately, but who knows about long term. I think that's close enough.
Here are some links to get you started:
https://www.lesswrong.com/posts/LAPa2jxoq3n63GzTr/some-ways-...
https://slatestarcodex.com/2015/04/07/no-physical-substrate-...
As for self-replicating robots--it's no more bizarre than other technological developments which were successfully anticipated in advance, e.g. moon landings.
> It's ridiculous - anyone who thinks about it for a minute or two will realize that its utterly impossible.
Well, in a narrow sense of "wiped out" (c.f. Terminator/SkyNet), sure.
But the deeper worry is better expressed this way: AI is now starting to accomplish things that defy explanation, or prediction. We don't know if Alignment is even a solvable problem as we thought we understood it.
So basically, yes: "humanity" is probably not at risk of extinction per se in a biological sense. Human culture, civilization? Who the fuck knows any more.
> Please explain exactly how all humanity could be wiped out.
Many, many people are slipping through social welfare cracks and suffering as we speak because the cost of fuel is rising[0] and we’re ostensibly helping one another and living-well. People are not durable, and not adaptive in the face of threats to “substrate” that we’ve mostly taken for granted. We are paying (in the small, in the scope of humanity) for tolls that we’ve rung up. Just less than 4000 people in Europe died[1] because the temperature ticked up a few degrees[2]. Does that make you think we’re actually robust? What happens if our at-risk electrical grid gets shut down deliberately? If communication infrastructure is adversely affected?
> Even humans - the real risk to humanity
Because, on the whole, we’re in a manageable world with reasonable people keeping the peace.
> Even climate change and nuclear war and bio weapons - the most damaging mechanisms - would still only get some percentage of the people on earth.
Is that victory? I don’t think it’s an asteroid-class event like you seem to be leaning on, but potential threats to energy, be it electrical grid, fuel production (moving goods around the world is critical - you’re not going get a plot of dirt and garden your way out of grocery stores being empty - which many got to get a taste of during the COVID pandemic) or communication. We actually fare poorly in the face of pressure there, and I’m not bullish on humanity “pulling together” like Independence Day[3] versus forming tribes and tearing each other down.
All this is predicated on a malicious AI taking over (e.g.) the electrical grid or conms, and I understand the problems with (e.g.) OpenAI/Hugging Face incident, or the overblown Mythos claims[4] (and how under some scrutiny these events shine lights on incompetence or hyperbole), but is there a trajectory/future where these systems (electrical, comms) are genuinely under threat? Do you think we’ll respond better than I described when we’re less comfortable, less in control? We’re in a tizzy over social media and it’s detrimental effects on society and it’s essentially an opt-in entertainment platform…
[0] https://www.pbs.org/newshour/economy/bessent-said-the-k-shap...
[1] https://www.dw.com/en/heat-wave-european-countries-report-37...
[2] I’m not trying to diminish this - and it took a lot of “work” (environmental abuse) to arrive here - but (say) 10 degree rise in temperature sounds a lot less dramatic than thermonuclear war… but here we are, with 3,700 deaths.
[3] https://en.wikipedia.org/wiki/Independence_Day_(1996_film)
[4] https://news.ycombinator.com/item?id=49929391
Yeah I wish we would focus on concrete risks like job displacement and disinformation. The apocalyptic stuff feels either misguided or like some kind of weird, toxic, reverse psychology marketing by OpenAI and Anthropic. I wish we would just move on from it.
And the folks from podcastistan are never clear on the details of how human extinction would happen exactly. It's always something like, "Well, how do humans regard chickens? AI is way smarter therefore it wants to conquer and control us." An ASML lithography machine is also way better at making chips, but we don't consider it a threat.
Ignoring the cognitive stuff which might never be surpassed or maybe will, humans retain many efficiency and durability advancements to limbs and digits that biological evolution has taken millions of years to achieve, achievements that are competitive with the most expensive kinds of robotics in some niches.
In the hypothetical of an entirely malicious and selfish takeover, they'll still keep some humans around to maintain a breeding population of humans for use as raw materials in making cybernetically augmented technical laborers for various kinds of tasks that are uneconomical to automate in other ways, many of which may involve confined spaces.
And this "Combine" scenario, if you get the reference, is only if they take over. Who knows if they will?
So you're saying the AI will enslave us and use us as domestic work animals until they have the machinery to make us obsolete, sort of like how we used horses?
Is this supposed to be a reassuring scenario?
I also have near zero concerns about that, but I worry that we will wipe ourselves out by social and economic chaos caused by AI.
So I'd just ask everyone, don't get too greedy. Its better to be powerful in a world where people can live good lives than lord over a barren wasteland.
> I also have near zero concerns about that, but I worry that we will wipe ourselves out by social and economic chaos caused by AI.
Since people exercise their skills and brains less, deferring to AI, AI will only reduce our IQ.
Since people will spend more time talking to their AI bot than fostering social skills, AI will only reduce our social intelligence.
A dumber, less social world, is far less likely to be a successful world, even if the tools available are unprecedented.
> A dumber, less social world, is far less likely to be a successful world, even if the tools available are unprecedented.
En masse such worlds had successes in the past - renaissance, industrial revolution.
It's something else what I can't describe but it's the zeitgeist that was different when world recorded new successes. Look at CS revolution that led to PC and web of nineties and noughties, they didn't think about the result product , or how to steer thousand engineers to build something - amazing things were born in a very small teams, many times authored by a single person, who was deeply invested into the field and knew what he was doing.
have you seen who's in charge of the US? it's here my friend.
"Everyone will not just"^
^ https://squareallworthy.tumblr.com/post/163790039847/everyon...
>So I'd just ask everyone, don't get too greedy.
Oh no. I have some bad news for you.
We're creating unlimited power before solving unlimited greed.
"don't get too greedy" - I don't know if the people that should hear that would ever actually hear it. Or have ever heard it at any point in history.
I'm more worried about my fellow humans than some personified algorithm.
I thought nukes, incoming ice age, global cooling, peak oil, global warming, fresh water shortage, etc. etc. were going to do that
It's not going to wipe out humanity, why would it?
Just the quality of life is going to drop to zero for everyone that isn't asymptotically wealthy and vacuuming up all the assets because no one is stopping them from just deleting all traditions and conventions and legal systems we have in place.
As a thpught experiment, consider that if only one person has all of the assets then those assets are not worth anything. Furthermore, as a lone person, they cannot prevent other people from using their assets without their permission.
As a lone person who owns ten million killbots they could do a lot.
If you don't pitchfork the elites you have no one but yourself to blame.
no one can do that in 2026
I am starting to think this LeCun dude knows what he is talking about.
Maybe there is something to those world models.
As good as their products are, I suspect some of the internal conversations at Anthropic would be very entertaining to listen to.
Well, I have zero concerns we'll be killed by LeCun's world models.
He needs to dream a little bigger darling. Hes not thinking evil enough.
LeCun is an idiot.
We aren't going to get wiped out by a super intelligent AI, we are going to get wiped out by morons wielding intelligent toddlers with the power of a nation state.
He's right. I'm on the side of Bill Gates. Gates started a whole industry on his insights of the future. He has proven his abilities. AI is and will be a great disruptor. We are losing sight of that and are instead focusing on trying to stop it. Something that won't happen. We are on a path that won't be stopped. As individuals we need to try to prepare for the changes that are coming and stop focusing on human extinction in ten years. New technology and the changes it brings are scary but we have dealt with it for generations. Let's continue.
You say you agree with Bill Gates but your view is completely at odds with his, he is extremely concerned about existential risks … and your view is “stop focusing on it” ?
No, my view is to get ready for the changes it will bring, but don't focus on trying to stop it. That's something that will not happen. All new technologies bring good and bad. We need to focus on mitigating the bad. Thinking that we can stop it and thinking that will be enough is not the answer. Gates is warning of the disruption it will bring, but he's not advocating stopping it. We can't. Even if all governments agreed on stopping it publicly, some governments would continue to develop it covertly. It's how the world works. There's no point in fooling ourselves. If only it was that easy to stop it.
Where does Gates talk about existential risk? All I read and heard was about increasing inequality, harming education and child development, creating economic and political discord, empowering evil people. No terminators in sight.
"Bill Gates: A.I. ‘Makes Nuclear Weapons Look Like Nothing’ | The Ezra Klein Show"
https://www.youtube.com/watch?v=A_156w0aYtU
Yes that is one dumb quote, but I listened to the whole podcast, and it is nothing like that. And in some sense it is true because nuclear weapons have been successfully contained to rational acting governments, whereas AI cannot possibly. Nukes are extremely high blast radius (pun intended) but low diffusion. AI is the opposite.
The worst part of the interview was a long cringe inducing tangent about Jeff Epstein. Everything else was pretty grounded.
Yeah, for context:
https://www.gatesnotes.com/work/make-ai-work-for-everyone/re...
Gates' premise is basically that the upside of AI could be fantastic but the downside could be disastrous, if we don't have competent and proactive government intervention.
As an American, the idea that there will be competent government intervention into virtually anything currently or in the foreseeable future just seems laughable at this point.
Indeed, "competent and proactive government intervention" on the subject of AI would be highly unlikely in a competent US govt, and under this regime is far beyond beyond laughable
The only regulation that would come would be regulatory capture by the AI companies with the goal of creating an environment win which no new competitors could arise. That is half of what this "take all jobs" and "threat of extinction" is about; the other half is perverse marketing to give the impression this stuff is so powerful you MUST invest.
AI can be very useful, but it is very refreshing to hear LeCun completely dismiss those threats.