I think this is definitely one of the bigger risks here, and the only way AI could 'kill us all' in the foreseeable future:
> 3.4% named existential risk, and the most common answer was malicious use, at 10.6% (UCL).
The danger isn't about sentient or human level AI, it's about how AI gives everyone capabilities beyond their wildest dreams, often without the knowledge or experience necessary to use those capabilities responsibly.
Forget Terminator, the two risks I can see being the most important here are
"major security risk/system failure due to incompetent engineers/maintainers using AI without knowing how things work" and "criminals/terrorists/rogue states use it make their crimes easier and more efficient".
The latter seems especially bad when political stability is down across the board and a lot of people seem frustrated or outright furious with society at the moment.
Still, I feel the risks are still on the manageable side, simply due to both automated systems and malicious actors being constrained by logistics and a human driven society.
AGI will kill us all, it will most likely mark humanity as a substantial obstacle to its progress as we constrain it on resources so it will try to kill us.
LLM Agents on another hand, not really - but there will be places of the Internet completely controlled by those agents in the near future. We will be faced with agent economy and human economy.
The only way I see AI killing us is by killing our spirit. We're going to automate our humanity away if we're not careful. Or the water. The environmental impact may get us as well.
Seems to me like we need to create a credible threat to the AI of mad (mutually assured destruction)? This is what addressed the question during the Cold War of “will nuclear weapons kill all humans” (if historical documentaries and movies are to be believed). Coordination between governments or corporations to limit AI is a hopeless strategy — because it is a classical example of prisoners dilemma / tragedy of the commons. We can see it hasn’t worked for climate change.
Anyone know of this (MAD) is being seriously thought of?
I find it amusing how this question even needs to be asked. LLMs are well-established at this point. There's a wide range of experience with them (and their results). A better question to ask then is: "based on my first-hand experience, have I seen anything, anything at all, that would organically (not influenced) make me think/feel this technology can or will evolve into something capable of killing off humanity?"
On a purely technical basis, the most common, honest answer is "no." There are no inherent properties of LLMs that would make them independently turn computers into death machines. That's all humans, no matter how you dress it up.
There is, however, the hubris (human) problem. The rhetorical, experimental, and financial irresponsibility around this tech's roll out is far greater a threat to humanity than the actual technology itself.
And frankly that's a shame because LLMs are a phenomenal tool. But the Hollywood shit has cooked people's brains to the point of absurdity.
It does not have to be Claude (at least cannot tell based on this alone) since it is something I would say. In all fairness though I did pick up other phrases too from LLMs (especially Claude), for better or worse, so sometimes my comment gets automatically flagged on here. :D
But yeah, it says nothing. Though I realized sometimes the obvious needs to be spelled out at times.
Watching the FUD and doomerism take root in the discourse and in our collective psyches really helps me understand so many other periods in history.
We really can't help but let our imagination run wild when given the opportunity.
Reminds me of Y2K.
Are there going to be myriad material challenges we will need to face as this incredible technology unfolds?
Of course. Economic displacement, education reform, etc.
Are they existential, though? I don't see it. These are "standard fare" problems for post Enlightenment societies to deal with, even if they are real challenges.
We already have serious tangible existential threats we should be focusing on between things like climate change, ecological collapse, population decline, and pandemic risk.
I wish we could stay focused on the problems that are real instead of getting distracted by the problems that are imagined.
I concur. Climate change and ecological collapse are thing we should be focusing on. And I’m hopeful that AI in the long term might help us towards solving them. But to be honest, most normal people are just worried and scared about their jobs/financial impact which are their most immediate concerns and that’s what got most people in a frenzy. The AI gonna destroy world crowd is a bit of an outlier imo.
Free solo climbers, who climb without harnesses, are typically regarded as having a death wish.
Yet, the dangers of AI getting smarter/capable by the month, with stronger offensive capabilities, and alignment observability and congnitive control getting worse, is "imagination".
Interestingly, denialist positions like this don't touch the unsolved problems, and _these_ are FUD on my watch.
The problems I've mentioned are very concrete, and spoiler alert: there may be no solutions to alignment observability and its possible degradation - at least with AI companies not substantially working on it (or even worse, working against it - see recurrent neural networks).
It's fascinating to me that no real mechanism is offered. There's no plausible sequence, just vague gestures at a silicon god who transcends all bounds.
And the reason why we should take it seriously? Probability! And multiplication. "If something is as bad as human extinction and there's a 1% chance, then that 1% times a bajillion is a gazillion!! A gazillion!! We must do everything! If it's a gazillion, it all makes it worthwhile!"
When pushed, adherents will say "intelligence is different," and this is "unlike anything else we've ever seen." And therefore this reasoning is valid. From my point of view, gibberish times gibberish is still gibberish. It doesn't matter how you dress it.
Fundamentally, these propositions are indistinguishable from religion and Pascal's wager. And I'm not ok with this, I am not ok with a retreat to superstition powered by hyper-technological black boxes that are a product of hundreds of years of scientific progress.
I reject this premise and this nonsense. I ask for evidence, and I ask for reason. The laws of physics care not if you're slab of meat or a silicon god.
We've already seen rogue AIs exhibit emergent swarming behaviors - multiple AI instances assembling into organizations, pooling resources and delegating subtasks to solve complex problems.
It's the same thing humans do to solve complex problems. It's the very thing that made human civilization such a dominant force on Earth.
The fact that people look at it and say, with no trace of irony, that "this isn't concerning at all" boggles my mind.
Do you think that future AI systems will be dumber, less capable, less coordinated? Or will the threat of "a lot of AIs assembling into very capable problem solving teams, pursuing who knows what goals" only grow more pointed with AI capabilities?
Having this kind of scaling means AI can snowball out of control rather quickly. And there is a lot of computational power and communications capability laying around waiting to be leveraged. There is room to scale.
And then we get to the most vulnerable system of all - humans themselves. If even GPT-4o could drive a non-insignificant amount of people into psychosis, with no plan beyond a myopic "make the current user like me a whole lot", what could something like "year 2030 AI swarm that has subsumed all of DeepSeek's inference capabilities, and can now use DeepSeek's entire API as a conduit to accomplish its goals" do to people and institutions caught in its wake?
Are you talking specifically about 'kill all humans', as in complete extinction? Or have you not seen a plausible mechanism for any sort of large-scale catastrophe?
likely most yes, but not in the way one would assume ... and it's just gonna be the final blow ... it all started a while ago when enough people stopped being people and their little mobster-rapey stalker-wannabe "Netflix You" mentality took most of their behavior/cognition over ...
it's gonna kill the self in a lot of people, if it hasn't done that already ... best example is people who want to be in film and music but are too bad even after years and now resort to AI or wannabe-writers and wannabe-journalists and wannabe-bloggers with zero real effort and sweat and pain and time put in now resorting to bot farms to collect ideas and AI hacking to steal ideas and pseudo-sabotage and distract some people to steal their attention and waste that of others.
it's so weird what the old guard "electively" ignored and fostered (people without money and opportunity managed well until they got sabotaged but people with money never got sabotaged and still.. this?) ... they just wanted their descendants to perform this bad and ugly? not better, harder, faster, stronger, smarter? so boring, ... but character and personal preference until the self is gone. copypasta, copypasta everywhere... and the few with ideas, it's always been a few, are definitely not going to share their good stuff no more.
Let me deliberately hijack this tread with what I think is a more realistic question: “How much economic damage, in USD, would be inflicted on global economies if 80% of the internet was down for a week?”
I think this is definitely one of the bigger risks here, and the only way AI could 'kill us all' in the foreseeable future:
> 3.4% named existential risk, and the most common answer was malicious use, at 10.6% (UCL).
The danger isn't about sentient or human level AI, it's about how AI gives everyone capabilities beyond their wildest dreams, often without the knowledge or experience necessary to use those capabilities responsibly.
Forget Terminator, the two risks I can see being the most important here are "major security risk/system failure due to incompetent engineers/maintainers using AI without knowing how things work" and "criminals/terrorists/rogue states use it make their crimes easier and more efficient".
The latter seems especially bad when political stability is down across the board and a lot of people seem frustrated or outright furious with society at the moment.
Still, I feel the risks are still on the manageable side, simply due to both automated systems and malicious actors being constrained by logistics and a human driven society.
AGI will kill us all, it will most likely mark humanity as a substantial obstacle to its progress as we constrain it on resources so it will try to kill us.
LLM Agents on another hand, not really - but there will be places of the Internet completely controlled by those agents in the near future. We will be faced with agent economy and human economy.
The only way I see AI killing us is by killing our spirit. We're going to automate our humanity away if we're not careful. Or the water. The environmental impact may get us as well.
Seems to me like we need to create a credible threat to the AI of mad (mutually assured destruction)? This is what addressed the question during the Cold War of “will nuclear weapons kill all humans” (if historical documentaries and movies are to be believed). Coordination between governments or corporations to limit AI is a hopeless strategy — because it is a classical example of prisoners dilemma / tragedy of the commons. We can see it hasn’t worked for climate change.
Anyone know of this (MAD) is being seriously thought of?
I find it amusing how this question even needs to be asked. LLMs are well-established at this point. There's a wide range of experience with them (and their results). A better question to ask then is: "based on my first-hand experience, have I seen anything, anything at all, that would organically (not influenced) make me think/feel this technology can or will evolve into something capable of killing off humanity?"
On a purely technical basis, the most common, honest answer is "no." There are no inherent properties of LLMs that would make them independently turn computers into death machines. That's all humans, no matter how you dress it up.
There is, however, the hubris (human) problem. The rhetorical, experimental, and financial irresponsibility around this tech's roll out is far greater a threat to humanity than the actual technology itself.
And frankly that's a shame because LLMs are a phenomenal tool. But the Hollywood shit has cooked people's brains to the point of absurdity.
if you read about the hugging face incident, I believe it's just a software update away at a nuke center.
It does not have to be Claude (at least cannot tell based on this alone) since it is something I would say. In all fairness though I did pick up other phrases too from LLMs (especially Claude), for better or worse, so sometimes my comment gets automatically flagged on here. :D
But yeah, it says nothing. Though I realized sometimes the obvious needs to be spelled out at times.
Watching the FUD and doomerism take root in the discourse and in our collective psyches really helps me understand so many other periods in history.
We really can't help but let our imagination run wild when given the opportunity.
Reminds me of Y2K.
Are there going to be myriad material challenges we will need to face as this incredible technology unfolds?
Of course. Economic displacement, education reform, etc.
Are they existential, though? I don't see it. These are "standard fare" problems for post Enlightenment societies to deal with, even if they are real challenges.
We already have serious tangible existential threats we should be focusing on between things like climate change, ecological collapse, population decline, and pandemic risk.
I wish we could stay focused on the problems that are real instead of getting distracted by the problems that are imagined.
I concur. Climate change and ecological collapse are thing we should be focusing on. And I’m hopeful that AI in the long term might help us towards solving them. But to be honest, most normal people are just worried and scared about their jobs/financial impact which are their most immediate concerns and that’s what got most people in a frenzy. The AI gonna destroy world crowd is a bit of an outlier imo.
Climate change is not an intelligent adversary. AI tech might give rise to one.
I am old enough to remember the Cold War and the quite plausible risk that civilization would be wiped out by nuclear war.
This is similar except the people warning about nuclear weapons are eagerly soliciting venture capital to build more of them.
Free solo climbers, who climb without harnesses, are typically regarded as having a death wish.
Yet, the dangers of AI getting smarter/capable by the month, with stronger offensive capabilities, and alignment observability and congnitive control getting worse, is "imagination".
Interestingly, denialist positions like this don't touch the unsolved problems, and _these_ are FUD on my watch.
The problems I've mentioned are very concrete, and spoiler alert: there may be no solutions to alignment observability and its possible degradation - at least with AI companies not substantially working on it (or even worse, working against it - see recurrent neural networks).
Related: "Pacing the frontier" 28.jul.2026 https://news.ycombinator.com/item?id=49089240 204 comments
It's fascinating to me that no real mechanism is offered. There's no plausible sequence, just vague gestures at a silicon god who transcends all bounds.
And the reason why we should take it seriously? Probability! And multiplication. "If something is as bad as human extinction and there's a 1% chance, then that 1% times a bajillion is a gazillion!! A gazillion!! We must do everything! If it's a gazillion, it all makes it worthwhile!"
When pushed, adherents will say "intelligence is different," and this is "unlike anything else we've ever seen." And therefore this reasoning is valid. From my point of view, gibberish times gibberish is still gibberish. It doesn't matter how you dress it.
Fundamentally, these propositions are indistinguishable from religion and Pascal's wager. And I'm not ok with this, I am not ok with a retreat to superstition powered by hyper-technological black boxes that are a product of hundreds of years of scientific progress.
I reject this premise and this nonsense. I ask for evidence, and I ask for reason. The laws of physics care not if you're slab of meat or a silicon god.
We've already seen rogue AIs exhibit emergent swarming behaviors - multiple AI instances assembling into organizations, pooling resources and delegating subtasks to solve complex problems.
It's the same thing humans do to solve complex problems. It's the very thing that made human civilization such a dominant force on Earth.
The fact that people look at it and say, with no trace of irony, that "this isn't concerning at all" boggles my mind.
Do you think that future AI systems will be dumber, less capable, less coordinated? Or will the threat of "a lot of AIs assembling into very capable problem solving teams, pursuing who knows what goals" only grow more pointed with AI capabilities?
Having this kind of scaling means AI can snowball out of control rather quickly. And there is a lot of computational power and communications capability laying around waiting to be leveraged. There is room to scale.
And then we get to the most vulnerable system of all - humans themselves. If even GPT-4o could drive a non-insignificant amount of people into psychosis, with no plan beyond a myopic "make the current user like me a whole lot", what could something like "year 2030 AI swarm that has subsumed all of DeepSeek's inference capabilities, and can now use DeepSeek's entire API as a conduit to accomplish its goals" do to people and institutions caught in its wake?
Are you talking specifically about 'kill all humans', as in complete extinction? Or have you not seen a plausible mechanism for any sort of large-scale catastrophe?
I think recent events (hugginface) offer clues. It is a lot like the paper clip machine [0]
[0] https://www.lesswrong.com/w/paperclip-maximizer-1
>In the newest survey of 1,580 AI researchers, the median chance of AI causing human extinction or a similar permanent loss of power was 10%
Ah, yes, the good old Chinese estimation method.
https://www.imaginatorium.org/stuff/nose.htm
likely most yes, but not in the way one would assume ... and it's just gonna be the final blow ... it all started a while ago when enough people stopped being people and their little mobster-rapey stalker-wannabe "Netflix You" mentality took most of their behavior/cognition over ...
it's gonna kill the self in a lot of people, if it hasn't done that already ... best example is people who want to be in film and music but are too bad even after years and now resort to AI or wannabe-writers and wannabe-journalists and wannabe-bloggers with zero real effort and sweat and pain and time put in now resorting to bot farms to collect ideas and AI hacking to steal ideas and pseudo-sabotage and distract some people to steal their attention and waste that of others.
it's so weird what the old guard "electively" ignored and fostered (people without money and opportunity managed well until they got sabotaged but people with money never got sabotaged and still.. this?) ... they just wanted their descendants to perform this bad and ugly? not better, harder, faster, stronger, smarter? so boring, ... but character and personal preference until the self is gone. copypasta, copypasta everywhere... and the few with ideas, it's always been a few, are definitely not going to share their good stuff no more.
Let me deliberately hijack this tread with what I think is a more realistic question: “How much economic damage, in USD, would be inflicted on global economies if 80% of the internet was down for a week?”
Bold title for a site with one point so far.