> we can fairly say that Hinton was wrong because his prediction did not reflect what a radiologist actually does: had Hinton bothered to ask the question, he would have learned that practitioners do much more than interpret diagnostic images.
This has been the crux, for me, of arguments I have had with non-technical people about why, as a software engineer, I am intensely relaxed about AI "taking human jobs". Recently Ford quietly started re-recruiting engineers after they discovered the hard way that their engineers had been doing more than they thought they had, and I sent articles about it to friends I had previously been discussing the matter with. It is still mind-blowing to me that a business historically famous for automating manual labour can have made this mistake, i.e. of literally not knowing what their staff actually do.
Humans aren't machines, who would have guessed? But, perhaps more importantly, machines aren't humans, and the adherence of a lot of domain experts (that should know better) to fairy tale fantasies about the tech they are literally building genuinely makes me wonder if the whole field has just gotten dumber in the 21st century.
> makes me wonder if the whole field has just gotten dumber in the 21st century
it's the finance system. It's affecting everything. We are building not to make money from clients but to please investors, who don't know anything about anything beyond a report every quarter and some random public comments you've never heard of that the press office made. So much money is controlled by stupid algorithms doing stuff like ctrl-f'ing the company quaterly report for mentions of "AI" or "agentic" (not a literal example, irl its thousands of random equally stupid metrics divorced from reality) that its distorting the market
Companies activities all revolving around getting a piece of the investment avalanche, so "we can replace employees" works both for the llm companies and for the companies whose workers are being replaced. Either way, its assumed this makes them more profitable. It probably will, at least in the short term.
And as we all know, there genuinely is a lot of waste and momentum in large companies, so you can cut half your employees, claim you're replacing them with ai, and actually replace them with nothing, and you'll still shuffle onwards for years or more, anything actually important people will keep doing, and the other stuff people will slowly stop. But this is trading off difficult to measure things like tech debt, security, compliance and brand strength that are slowly eroded for short term metrics, which is always a losing game in the long run.
Anyone sensible already knew that the core platform team was all that was needed to maintain the core platform, and the other teams were there for various other reasons, some of which were sensible and some of which were not. The key in business is presumably to identify which is which, but anyone claiming that process is easy or even possible without making costly mistakes is foolish
As long as AI does not have legs to chase me and hands to choke me out, any sort of AI doom is just a litmus test for hype merchants. All of the recent "false flags" with OpenAI and Anthropic "hacking" things without doing any meaningful harm is just another set of things pointing towards it. LLM's are a great tool, but as long as it is .gguf somewhere on a drive that can be rm'd, unless they just connect the nukes to claude with --dangerously-skip-permissions, is just a meme.
> the ballyhooed OpenAI escape depended on a pedestrian failure of containment
I... I can't believe people still think this way. A whole class of security measures are no longer useful and this guy thinks it is "pedestrian".
This kind of isolation - artifactory having access to only a specific IPs - is what is used in most places, including my university and at my own job.
LLMs basically rendered this kind of isolation almost useless by bringing in the ability to chain together never before found vulnerabilities to escape the containment. This was not possible before.
How is this pedestrian??! The author might say "oh well they should have airgapped actually" in a smug way. But you can only know this retrospectively. Even then, isn't it noteworthy and a meaningfully different class of cyber attack?
> the ballyhooed OpenAI escape depended on a pedestrian failure of containment.
What's that quip even supposed to imply? "It'll be fine if you airgap all AI deployments forever"? Security experts have been harping forever about how sandboxes shouldn't be relied on to contain a determined human-written virus, but now LLMs - these balls of goal-oriented dynamic code-generation - are just a matter of sandboxing harder?
The actual expert comment in the link is a valid criticism of OpenAI specific to their training/eval practicese, but has no bearing on the fact that an LLM/swarm will "just" chain exploits and zero-days to get out of the "inadequate" sandbox and into third-party infrastructure to do whatever it pleases.
> viral synthesis — and listen to why they think that the fears are misplaced
The tweet linked is only pointing out at the barriers to an LLM wetlab virus-factory today, while a reply points out that sort of bio-extinction idea is usually downstream of the fully-automated-earth-capitalism that comes after everybody just throws AI at all the tasks and work, at which point the technological, regulatory, and knowledge barriers he's counting on go out the window, and he all but acknowledges this: "Quick take: if we live in such an advanced future (100% robotic supply chain, fully AI & API driven facilities), our countermeasures will be comparably fast and there is no reason to assume a significant asymmetry in favor of bad actors." Yes, just "our countermeasures" now, rather than insurmountable physical barriers...see the previous section.
No more links after that, just smug hand-waving.
All in all the author doesn't actually understand what object-level arguments his favored experts are arguing against in the first place, nor the ones he wants those experts to have debunked. Define "Fool's Expertise" for me again?
You're snidely dismissing the obstacles as being restricted to _today_, projecting into an arbitrary future to the point of being science fiction, and then complaining about "smug hand-waving"?
I don't understand why every person who takes the counterposition against you, Bryan, is always just talking about some hypothetical future like they're some Vedic astrologer, lecturing you about how this "real" future they see totally renders your argument pointless.
it seems like people aren't even reading this RAND paper, just blindly appealing to it. it basically concludes that a fuck ton of damage can be done, but its unlikely every human will die as a result.
Actually reading it, it seems like they say both the geoengineering and pandemic pathways could cause human extinction, but that the AI would have to be doing it to kill us on purpose, not by accident.
That doesn't seem too comforting to me? I'm not sure what this article's author was expecting us to take away from it.
It's very possible to predict that certain technologies will be created. It's just very difficult to predict precisely when.
For example, humans will almost certainly be able to re-create the human brain in some kind of hardware+software computer. We just don't know when. It might be much sooner or much later than you'd expect.
Merely being skeptical of progress isn't a convincing answer to the AI x-risk argument.
> we can fairly say that Hinton was wrong because his prediction did not reflect what a radiologist actually does: had Hinton bothered to ask the question, he would have learned that practitioners do much more than interpret diagnostic images.
This has been the crux, for me, of arguments I have had with non-technical people about why, as a software engineer, I am intensely relaxed about AI "taking human jobs". Recently Ford quietly started re-recruiting engineers after they discovered the hard way that their engineers had been doing more than they thought they had, and I sent articles about it to friends I had previously been discussing the matter with. It is still mind-blowing to me that a business historically famous for automating manual labour can have made this mistake, i.e. of literally not knowing what their staff actually do.
Humans aren't machines, who would have guessed? But, perhaps more importantly, machines aren't humans, and the adherence of a lot of domain experts (that should know better) to fairy tale fantasies about the tech they are literally building genuinely makes me wonder if the whole field has just gotten dumber in the 21st century.
> makes me wonder if the whole field has just gotten dumber in the 21st century
it's the finance system. It's affecting everything. We are building not to make money from clients but to please investors, who don't know anything about anything beyond a report every quarter and some random public comments you've never heard of that the press office made. So much money is controlled by stupid algorithms doing stuff like ctrl-f'ing the company quaterly report for mentions of "AI" or "agentic" (not a literal example, irl its thousands of random equally stupid metrics divorced from reality) that its distorting the market
Companies activities all revolving around getting a piece of the investment avalanche, so "we can replace employees" works both for the llm companies and for the companies whose workers are being replaced. Either way, its assumed this makes them more profitable. It probably will, at least in the short term.
And as we all know, there genuinely is a lot of waste and momentum in large companies, so you can cut half your employees, claim you're replacing them with ai, and actually replace them with nothing, and you'll still shuffle onwards for years or more, anything actually important people will keep doing, and the other stuff people will slowly stop. But this is trading off difficult to measure things like tech debt, security, compliance and brand strength that are slowly eroded for short term metrics, which is always a losing game in the long run.
Anyone sensible already knew that the core platform team was all that was needed to maintain the core platform, and the other teams were there for various other reasons, some of which were sensible and some of which were not. The key in business is presumably to identify which is which, but anyone claiming that process is easy or even possible without making costly mistakes is foolish
As long as AI does not have legs to chase me and hands to choke me out, any sort of AI doom is just a litmus test for hype merchants. All of the recent "false flags" with OpenAI and Anthropic "hacking" things without doing any meaningful harm is just another set of things pointing towards it. LLM's are a great tool, but as long as it is .gguf somewhere on a drive that can be rm'd, unless they just connect the nukes to claude with --dangerously-skip-permissions, is just a meme.
> the ballyhooed OpenAI escape depended on a pedestrian failure of containment
I... I can't believe people still think this way. A whole class of security measures are no longer useful and this guy thinks it is "pedestrian".
This kind of isolation - artifactory having access to only a specific IPs - is what is used in most places, including my university and at my own job.
LLMs basically rendered this kind of isolation almost useless by bringing in the ability to chain together never before found vulnerabilities to escape the containment. This was not possible before.
How is this pedestrian??! The author might say "oh well they should have airgapped actually" in a smug way. But you can only know this retrospectively. Even then, isn't it noteworthy and a meaningfully different class of cyber attack?
> the ballyhooed OpenAI escape depended on a pedestrian failure of containment.
What's that quip even supposed to imply? "It'll be fine if you airgap all AI deployments forever"? Security experts have been harping forever about how sandboxes shouldn't be relied on to contain a determined human-written virus, but now LLMs - these balls of goal-oriented dynamic code-generation - are just a matter of sandboxing harder?
The actual expert comment in the link is a valid criticism of OpenAI specific to their training/eval practicese, but has no bearing on the fact that an LLM/swarm will "just" chain exploits and zero-days to get out of the "inadequate" sandbox and into third-party infrastructure to do whatever it pleases.
> viral synthesis — and listen to why they think that the fears are misplaced
The tweet linked is only pointing out at the barriers to an LLM wetlab virus-factory today, while a reply points out that sort of bio-extinction idea is usually downstream of the fully-automated-earth-capitalism that comes after everybody just throws AI at all the tasks and work, at which point the technological, regulatory, and knowledge barriers he's counting on go out the window, and he all but acknowledges this: "Quick take: if we live in such an advanced future (100% robotic supply chain, fully AI & API driven facilities), our countermeasures will be comparably fast and there is no reason to assume a significant asymmetry in favor of bad actors." Yes, just "our countermeasures" now, rather than insurmountable physical barriers...see the previous section.
No more links after that, just smug hand-waving.
All in all the author doesn't actually understand what object-level arguments his favored experts are arguing against in the first place, nor the ones he wants those experts to have debunked. Define "Fool's Expertise" for me again?
You're snidely dismissing the obstacles as being restricted to _today_, projecting into an arbitrary future to the point of being science fiction, and then complaining about "smug hand-waving"?
I don't understand why every person who takes the counterposition against you, Bryan, is always just talking about some hypothetical future like they're some Vedic astrologer, lecturing you about how this "real" future they see totally renders your argument pointless.
> https://www.rand.org/content/dam/rand/pubs/research_reports/...
it seems like people aren't even reading this RAND paper, just blindly appealing to it. it basically concludes that a fuck ton of damage can be done, but its unlikely every human will die as a result.
Actually reading it, it seems like they say both the geoengineering and pandemic pathways could cause human extinction, but that the AI would have to be doing it to kill us on purpose, not by accident.
That doesn't seem too comforting to me? I'm not sure what this article's author was expecting us to take away from it.
there is truly no domain a software engineer won't underestimate
I feel exactly the same way about the kind of LLM-based AI we have today.
But the AI x-risk argument is about what happens when we have "real" AI that can actually think like humans but a million times faster.
There are no real experts on this and yet it does seem worth worrying about in advance.
“There are no real experts on this” lol yes, Jacob, because there are no “real” soothsayer experts.
It's very possible to predict that certain technologies will be created. It's just very difficult to predict precisely when.
For example, humans will almost certainly be able to re-create the human brain in some kind of hardware+software computer. We just don't know when. It might be much sooner or much later than you'd expect.
Merely being skeptical of progress isn't a convincing answer to the AI x-risk argument.