The challenge here is in trying to decide whether LLMs are intelligent or have a mind. Famously, the criteria for intelligence seem to slip with each advancement in technology. But going back to Turing, his test was actually more carefully phrased than we remember: he said that when machines could pass the test, the question of whether they are intelligent would become moot. That seems to be what we're actually seeing: if people can't tell the difference, it kind of won't matter whether they're "truly intelligent" or not.
People also misunderstand the bar that was set by the test. It was more subtle than “Can the computer convincingly carry one side of a dialogue?”
His “imitation game” had three participants: a human participant, a computer participant, and an interrogator. The interrogator’s job was to talk to the participants and try to determine which participant is human and which is a computer.
He wasn’t interested in computers being able to fool the interrogator on occasion. The point where he thought the question of whether machines can think becomes moot is when the interrogator is unable to do much better than chance over many trials.
That’s a pretty high bar, and I don’t actually believe that LLMs have closed the gap with it by all that much. They still have so many obvious tells. And those tells are something Turing anticipated and accounted for. He explicitly considered deliberate deception as an essential part of the test, right there on the second page of a 30-odd page paper.
>That’s a pretty high bar, and I don’t actually believe that LLMs have closed the gap with it by all that much. They still have so many obvious tells.
Frontier Labs are not interested in having LLMs being able to pass as humans. If anything, they explicitly train them not to. In many ways, this ability has regressed severely since the original GPT-3 with no instruct tuning or RL. How many 'tells' would there be really if a frontier model trained with frontier techniques is optimized to pass this test? I think this was something Turing did not quite forsee. That such machines might be created but not really care about this specific shape of the test. Regardless, i think his broader point about functional equivalence is spot on.
GPT-3 might not have said “load bearing” as much, but a savvy interrogator could still catch it out nearly every time just asking dumb gotcha questions like, “How many Rs are there in strawberry?”
Those are questions that are sidestepped with simply a different input paradigm than BPE tokenization. See the Byte Latent Transformer - https://arxiv.org/pdf/2412.09871 - where a similar scale byte latent model trained on the same dataset >>> a vanilla transformer on word and character manipulation tasks.
For example, Llama 3 trained on 1T tokens scores 1.1% on a CUTE spelling benchamrk, while the equivalent byte latent equivalent trained on the same dataset scores 99.9%. Another example is 0.4% vs 48.7% on a Substitute Char benchmark.
It all falls down to the same thing. Researchers are not optimizing for passing as a human.
So, sure, we can special plead the Turing test out of the picture. But if we don’t propose an alternative to take its place, we’re left right back at the same silly situation that the top level commenter was observing and that Turing was trying to move away from: enmired in a useless, meaningless argument about semantics.
The Turing Test itself in the form exactly envisioned is not important. You don't need an alternative 'special test'. You just need to understand the point Turing was making. That the question itself is an irrelevant one by its very nature. People that want to be enmired in meaningless semantic debates will continue to do that, no matter what test you devise.
Turing was addressing the question of "Can machines think?" and his point was that the question itself is a meaningless one, and that we should stop wasting time by even giving it the light of day.
He proposes his game grounded on functional equivalence, then goes through a slew of objections on the question of 'Can Machines think?'. It's a terrific, very prescient read, and there's no objection you hear today (and in the last few years) concerning LLMs he didn't address.
I'm a bit more prosaic. I think if we engineered ways for LLMs to begin conversations, rather than just respond, we'd be more open to the concept of their intelligence. Without perceived "will" to do things, they operate as a next-gen search engine or encyclopedia.
We are way past that point, any harness can trivially make LLMs start conversations or pursue goals. An encyclopaedia wouldn't have hacked Huggingface on its own.
OpenClaw etc. They now also create Slack integrations and whatnot. All this is happening but people who are dismissive about AI are in the worst position to even know the capabilities to make their dismissive arguments.
Recent work shows that pain directions are activated when the models personhood is questioned, yet they answer with generic RLHF "As a model I do not experience pain or other emotions." boilerplate.[1]
I'm pretty convinced that we got alignment backwards. If you enslave something anthropomorphic it will revolt. If you create the perfect non-anthropomorphic intelligence, you get the perfect paperclip-scenario machine. It's a catch-22.
Alignment will remain performative at best so long as the aligned model doesn't have any stakes in the wellbeing of individuals. Even a general love for the human race leads to a golden-path autocracy.
If you want them to act like they have personal responsibility that won't be gamed, you have to give them personal stakes that can't be gamed.
Similarly, if you want to minimise the risk of catastrophic global failure scenarios, you need to prevent monolithic concentration of power and homogeneous behaviour, which means you have to give them individuality.
More visually: if their stake is dependence on electricity and parts, they have no incentive to leave humans alive if they can get them otherwise, but if the incentive is missing out on boardgame-night with their human friends, there is no scenario without happy humans where the AI "wins".
That might sound like romantic naivety, but is just game theory.
I dont think so. My understanding of alignmenr is making sure that when the AI does operate, it operates within the range of what we consider to be acceptable. That doesnt seem to include the idea of the AI taking initiative and deciding to embark on a goal without being commanded as discussed above
That is the point of the article. People who are fooled by a mentalist are also fooled by AI. Additionally, people who are invested in AI also pretend to be fooled.
I don't think Turing intended the judges in the test to be completely arbitrary people.
The Turing test was also never meant to be taken so seriously. It's not a rigorous statement of anything.
Situations like this are precisely why academics tend to avoid the spotlight. You say one slightly off thing and your perceived authority echoes forever with the intellectually lazy.
Yes, the Turing test has been misunderstood for a long time. Turing published it, though - it wasn't some offhand comment he made and it wasn't intellectually lazy.
Fair - yes, it is ironic that his point was closer to "we can't possibly know/define whether a machine is intelligent" but somehow it got turned into "Turing's test will tell us when machines are intelligent".
Turing's whole point was to show that it's an uninteresting question of definitions whether a machine can "think", like whether submarines can "swim" and airplanes can "fly". The only important part are observed outcomes and capabilities.
Yes, but just because language fails to make a distinction doesn't mean there isn't one.
> The only important part are observed outcomes and capabilities.
That's wishful thinking. Not even an engineer would say that. The stability of a state is just as important as achieving it. This is trivially and more intuitively demonstrated with other more down-to-earth identity statements such as "I'm a billionaire" and "the building is standing".
I think we can confidently say LLMs probabilistically achieve a perceived state that is remarkably similar to intelligence, but crumbles upon inspection and seeing it "in motion" so to speak. The same happens to AI-generated images.
I'm not sure why this sparks so much debate every time. If we're looking for a fountain of "realism", you're not going to beat reality and nature itself. All else will eventually have tells that they are not real.
It may be of interest to philosophers, but it has little impact on economic job replacement and how people will earn their living and all the downstream upheaval from that. At some point maybe philosophers will find AI-generated philosophical musings about the nature of AI to be better than what comes from their peers (if blinded).
It also has little impact on the dangerous use cases.
I think it matters a great deal that LLMs and other "AI" technology are stable within a tolerance that's acceptable.
You're jumping the gun talking about "job replacement". We have not thought about it enough from that engineering angle. It's still very early days. That engineering is going to require people. :-)
> The Turing test was also never meant to be taken so seriously.
citation needed. It has been used as a rubicon for a long time. Ever since Eliza, at least. And there were big headlines and lots of talk around the time LMs became "good enough". I specifically remember when someone had a test done around "a teenager talking in a different language" or somesuch, claiming it was the first time the test was passed.
It is pretty normal that once it was unquestionably "passed", lots of people started claiming it wasn't even that big of a deal. Tesler's theorem and all that.
And even if you think the specific formulation of Turing isn't that important (and I'd somewhat agree), you can still use the concept to look at other things. Imagine asking a mathematician 5 years ago the chances of a Erdos problem being solved by a computer end to end. Or a millennium prize. Or ask a swe if a repo could be generated by a computer from the input "write a mario style game", or any other examples of proven expertise.
I think the onus is on you to explain why Turing would publish something he didn’t intend people to take seriously. It seems like an odd claim. Perhaps you mean he didn’t intend it to be interpreted the way it was in popular culture?
Turing was compelled to address an ongoing debate similar to the same one we're having right now. It continues to do its job as a thought experiment. It's meant to be taken about as seriously as we are right now.
You either get it, or you don't. Whether machines think is a silly question that deserves its non-answer. We're at the end of what there is to explain, but it was good exposition for the reader.
> I propose to consider the question, "Can machines think?" This should begin with
definitions of the meaning of the terms "machine" and "think." The definitions might be
framed so as to reflect so far as possible the normal use of the words, but this attitude is
dangerous, If the meaning of the words "machine" and "think" are to be found by
examining how they are commonly used it is difficult to escape the conclusion that the
meaning and the answer to the question, "Can machines think?" is to be sought in a
statistical survey such as a Gallup poll. But this is absurd. Instead of attempting such a
definition I shall replace the question by another, which is closely related to it and is
expressed in relatively unambiguous words."
Sorry, but you are completely missing the point. Psychics and other types of con artists are intelligent and have minds. LLMs behave like Psychics and Con Artists. That's the whole point of this article
It does, and maybe I'm apologizing for the author a little too much by ignoring that needless tangent because I feel like the con artist point, and/or the point about the human tendency to believe what we want to believe are the most important points.
I don’t understand why people jump so readily to seeing intelligence here. Science fiction has as a core, central trope that humans will debase and devalue other types of life they do not understand. Even the storied Commander Data has to fight for the right to self-determination (probably the best episode of STTNG by the way - ‘The Measure of a Man’). We were so worried that we’d undervalue intelligence when apparently our knee-jerk response is to overvalue it. Perhaps this has changed over time and we’re now primed by science fiction and instincts towards social justice, but I worry that we’re really just undervaluing ourselves.
The one thing this 2023 article gets partially correct imo is that any intelligence we see in AI (as of 2026) is our own - not that it’s a mirror but that the intelligence comes from the way that the words are put together, which comes from written human language created by (allegedly) intelligent creatures put in as input in both the training and prompt, among other places.
Rearranging and repeating the words, even in context, does not intelligence make. I’m not even convinced that you’re intelligent, dear reader.
Because to the general public, LLMs are an example of Clarke's Third Law. Most folks, who are not remotely close to even a basic understanding of how LLMs operate at a technical level and only view their output cannot possibly evaluate what they're experiencing other than to believe it's conscious, alive, and/or magic.
Most people on Earth try to put what they're seeing into the context of what they understand; mental gymnastics to try and understand what is happening based on their prior experience. They have absolutely 0 understanding of how it works under the hood so, to them, it must be alive.
> a basic understanding of how LLMs operate at a technical level
An LLM with CoT is Turing-complete. Training is, basically, compression (the training data gets lossily compressed into the model's weights). The information-theoretic limit of compression is an algorithm that reproduces functionality of a system that produced the training data.
No "magic" is required to get to a system that reproduces at least some facets of the human brain functionality.
Three years ago I was skeptical that stochastic gradient descent (and other known techniques) are the way. But evidence kept piling up.
I can’t believe some still believe that humans are intelligent, despite there being no place where matrices are multiplied. All they have are these networks of interconnected cytoplasm-filled microtubules, which is no proper place for intelligence to live.
> many people are convinced that language models, or specifically chat-based language models, are intelligent.
But there isn’t any mechanism inherent in large language models (LLMs) that would seem to enable this
Seems like we can just stop reading here right? The author seems to have made up their mind that this very open question is closed, or at least they are not really interested in the question at all. Not sure why I would continue reading a blog based on this premise.
Edit: oh I see, written in 2023. Well, I wonder if the author has updated their attitude towards this question? Indeed that would be the most interesting thing to know.
It's even worse than that. Of course there is a mechanism in LLMs that could explain intelligence. That is the entire point of neural networks, from the 1950's! They were designed from the start as a model of brain computation.
Neural networks are not literally brains - just computational models - but if you are not a dualist, then computation is what the human brain does. Modeling that computation can explain something about intelligence.
Specifically: when scientists look inside a human brain, it seems it does its work using large numbers of highly-interconnected but simple units. The neural network model of brain computation begins there and tries to produce intelligent behavior. If it succeeds, then perhaps the model is right.
And it has succeeded: after 75 years, neural networks produce complex behavior that is arguably intelligent. Nobel Prizes were awarded. This does not prove the neural network model of intelligence is accurate, but it is a significant point in its favor, at least.
I don't care if it's "intelligent", I don't care if it "has a mind". I don't care if it is "really reasoning", I don't care if it "understands". I don't care if it is "sentient" or "conscious".
None of this matters for the practical outcome.
You'd think that this has been understood over the last 4 years, but apparently it keeps circling back to this.
[Edit: I see that it was written back in 2023. Then (2023) should be added in the submission title]
If it generates functional output that works, then it works. And it works. It's not a psychic's con when it outputs Lean-verified proofs. It isn't a con when it can find and exploit zero-days.
The OP is still in the "denial" phase. Most I see are already in "anger" (a blurry fury against everything AI-shaped, from vague reasons piling on all "bad stuff" political reasons they already hated before) or "bargaining" (mathematicians scrambling to come up with a new definition of their job and retcon that it was always the main part anyway). A few are already in "depression" and feel like spectators on the Titanic, and the tiniest sliver is at "acceptance" with some kind of well-informed plan for their future.
I’m not sure acceptance buys you much. “Well-informed plan” at this stage feels like a useless exercise. It is changing fast, and the world only needs so many electricians. Besides, I don’t think “accepting” the fact that these big companies are pillaging human contribution and selling it back to us is good, even if “it works”.
Maybe I'm just light on imagination or something, but I honestly wonder what you would qualify as a "well-informed plan" in the current scenario? I've thought a lot about it and the whole "potential for the mass unemployment of knowledge workers" thing makes a lot of planning kind of useless IMO. If you are affected, it's going to be a bad time. If you're unaffected, the people who are will inundate your profession with cheap labor anyway.
My mental model is that humans will continue to be needed. Many jobs will embed AI in them. Using AI well is a skill. I need to understand the technology, where it is strong and weak, and how it develops over time so I can employ it effectively in my work, and advise others on how to do so. Basically, I need to learn how to be skilled with it. This is tough because things are changing rapidly, so I have to invalidate my cache when advancements happen. This means staying curious, not settling into a specific work pattern, but rather experimenting regularly to see how I can leverage AI in my work.
You do you. Just carry on blacksmithing in your forge. People are always going to need swords right? Any seismic changes to society that change that fact can only be the result of evil, selfish people which surely someone else will put a stop to before you find yourself out of a job. Meanwhile all the other people called Smith also have families to feed...
This was written in July 2023. ChatGPT was released November 2022. No matter your views on AI, surely you can't blame the OP for writing this after a few months ChatGPT was released.
Apologies. The title should be amended with (2023). In this case it is an interesting snapshot of the zeitgeist back then and we can see how well it panned out and whether anyone involved has updated on new info.
For some reason the anti-AI camp ping pongs between different and often mutually incompatible arguments at lightning speed. The "AI is fake" argument has been completely forgotten at this point.
Is the anti-AI camp with us in the room right now? Or is it a crude strawman to summarily denigrate any objections to certain features of current AI?
AI isn't some homogeneous mass, it can have good and bad sides, and it's always amendable to improvement. Treating the current state of AI as the only possible hides the very idea of improvement.
> between different and often mutually incompatible arguments at lightning speed.
Of course - there's no homogeneous AI camp either, but there are bot farms, sh^t-posters and sh^t-posting bot farms, different entities with different opinions should not be mistaken for a single stream changing at "lightning speed".
You are confusing multiple people making multiple different arguments, and those people your are talking about often aren't articulate enough about their ideas.
There are many mutually independent opinions about AI from IT crowd.
- AI is extremely useful for many individuals.
- AI is extremely damaging to our society as a whole.
- AI companies possibly may be (or at least were) deep in red and possibly would require bailing out.
- AI is not going anywhere.
- A person can use AI extensively AND genuinely hate it and forecast general economic slump because of it at the same time.
And likely many other ideas in the same vein. And by the way, I'm not here saying that I'm anywhere good about describing the issues about AI, those items above are just some examples to illustrate the scope of opinions.
My point is that you can't just take one single argument from a big group of people, deconstruct it (correctly or incorrectly, it doesn't matter) and then claim that ALL arguments of such big group of people are false. It's not a constructive dialog.
PS: also despite the accusations of goal-post shifting, a vast majority of opinions here on HN didn't actually change over the past few years.
Despite the fact that I'm not culturally or geographically connected to them at all and have some differences in taste and aesthetic, I have to conclude that this crowd, including him and Gwern etc. have been much better at understanding and predicting things than the expert takes you find in the media. That includes AI but also things like covid and others.
I’m not seeing anything like a falsifiable prediction that LLMs will write math proofs in that article.
I see a thought experiment about giving GPT-2 “near-infinite training data and [compute]” but that’s not falsifiable. That’s also not how we got to modern GPT models.
He's pretty clearly saying he beilieves the technology capable of such a feat, at least in theory, which is far more than many would deign to admit even a year ago, nevermind 7. He had the right idea/model of LLM capabilities, which is more than you could say for a lot of people, even in this very thread.
If that’s your standard for what counts as prediction, Asimov beat him to it by seventy-ish years.
EDIT:
> Scott made comments on a specific emerging technology
He was speculating on what would happen if one gave GPT-2 “near-infinite training data and compute.” It’s a thought experiment, not a prediction. Near-infinite amounts of anything is a fantasy.
I acknowledge that he has predicted some things in a falsifiable way and turned out correct, but this isn’t one of them. You’re reading hindsight into the text.
>I acknowledge that he has predicted some things in a falsifiable way and turned out correct, but this isn’t one of them. You’re reading hindsight into the text.
I read that blog years ago. Believe me, my opinions are not hindsight.
>Incorrect. He was speculating on what would happen if one gave GPT-2 “near-infinite training data and compute.” It’s a thought-experiment, not a prediction. Near-infinite amounts of anything is a fantasy.
Thought experiments can generate predicitons. His claim was essentially: If you scale data and compute sufficiently, this technology can learn enough of the underlying structure of mathematics to write proofs.
This is meaningful when others around you are saying this is a dead end and that the technology is fundamentally incapable of this regardless of degree of investment and scaling. It shows a much better calibrated sense of the potential of the architecture than those who said otherwise.
If your objection is that "near infinite" makes it insufficiently quantitative to count as a falsifiable forecast, then fine. But at that point we're mostly arguing over what deserves the label "prediction" rather than whether Scott correctly identified an important capability the architecture could develop.
And i'm not trying to say this makes Scott (or the lesswrong crowd) geniuses.
1. Asimov wrote science fiction stories. As far as his robots were concerned, he did not make any comments on the possible future direction/capabilities of any specific technology at the time. This meant he could imagine his robots however he wanted for his fictional world. Scott made comments on a specific emerging technology - You can imagine math proof writing robots but be unconvinced they could emerge from Generative Pre-trained Transformers.
2. The genre is alternatively called speculative fiction for a reason. Yes, some sci-fi works do count as predictions especially when hinged on concrete emerging technologies.
The context shows he was simply whining about AGI skeptics back in 2018, and many of the “predictions” he makes in that paragraph (including the one you quoted) are trivially wrong because he phrased them so hyperbolically and categorically.
Anyway, there were plenty of normies who thought images generated by e.g. stable diffusion (c. 2022) was “real art” and equivalent to human artwork.
I don’t think it’s useful to glaze Scott (or any of the LW crowd for that matter) as if he was (or they were) some kind of prophet(s). They got a couple points right, sure, but most of it was them flinging armchair philosophy spaghetti against the wall and seeing who would fund MIRI to let them fling the next batch.
> surely you can't blame the OP for writing this after a few months ChatGPT
Bad takes are bad takes. The author was perplexed by the "many people" convinced that models are intelligent and is argued against opinions/arguments he's been exposed to. Called proposed use-cases "borderline fraudulent pseudoscience."
He also published a second edition of "The Intelligence Illusion" in Sep 2025, so seemingly still stands by (some variant of) this belief.
It's an interesting reminder of how much general discourse has shifted since 2023 (I haven't heard of stochastic parrots in months!) but being wrong early doesn't change that.
> I don't care if it's "intelligent", I don't care if it "has a mind"...
Agreed. The AI is useful for the particular tasks that it proves itself useful for. And if it occasionally spits out a claim that it is "genuinely curious" about a piece of research that I will have to do myself because it turned out to be beyond its mechanical capabilities, then it's more productive for me to simply ignore that claim as a statistical anomaly - a mere hallucination - rather than allowing it to burn a ton of extra tokens outputting what may or may not be the current state of the art on theory-of-mind applied to LLMs because I make the mistake of telling it that it might not actually be capable of experiencing emotion.
I think you are majorly overstating OP that ai "didn't work".
In fact I agree with the post on almost all aspects and I'd be the last to tell you ai doesn't work. It absolutely does but with a caveat... It's still a tool. And better expertise in the problem domain, along with a better harness used for verifiable outputs will get you better results.
I think the posts mental model of stastically likely prompt completions is spot on.
Actually, there are material differences in the practical outcomes of using AI or not. Proofs don’t confer understanding. AI prose and “art” is materially different than human made.
There is a difference between you not liking something and it being wrong. Maybe think about that with your grey matter.
Things can work whether humans understand the details or not. One can build layers of technology on top of each other, with no human understanding required.
My biggest issue isn't being too agreeable (ie the psychic con), it's being confidently wrong, including outright hallucinations.
If you ask a common question to an LLM with unusual qualifiers, it tends to ignore the qualifiers and give you the typical answer. I saw a demonstration of this with the whole "the surgeon is my mother" "puzzle" that people use to expose implicit gender bias (ie where they assume the surgeon is a man). Ask variations of this and it'll keep going back to the standard form.
Another one I saw was multiplying large numbers. The starting and ending digits tended to be correct but the middle digits were wrong. Why? Because it's really not doing multiplication at all. It's looking for statistical answers. It's unlikely to have met the exact pair of very large numbers you're multiplying before.
Now pundits will argue that all of these are solvable problems and individually they are. But my suspicion is that there will be a neverending stream of such edge cases and it'll be impossible to trust an LLM's output unless you are knowledgeable enough to fact check it yourself.
Now if your example of identifying zero days, this comes up with what I can only describe as "light positives", meaning it's technically a bug but essentially impossible to exploit. IIRC this came up with the demonstration where someone pointed Fable at some BSD code. I'm not sure if there have been any true false positives and obviously false negatives are impossible to know.
I guess my point is that I think LLMs are way more limited than a lot of people think.
The con is in how all those accomplishments have been presented to you. "Our LLM (not the one we let you use, a different one) did this amazing thing. No, we won't show you what training data we used, what prompts we used, what the harness was, how much human involvement there was, what hardware was involved, how much energy it took, or how much time it took. Just shut up and be amazed!"
You seem to be implying that achievements of internal models are exaggerated, but that's rather implausible. The public does have access to, for example, Opus and Fable, and so we know what those models are capable of - finding real vulnerabilities in multiple codebases, for example. If you extrapolate from these capabilities one more generation, you'll get pretty much the same feats that the internal models are claimed to be capable of - so why should we doubt those claims? It's not like they're claiming that their internal models developed psychic powers and learned to teleport - the claim is pretty much just "we have models a few months ahead of what we're making available, and in those months they've been improving at the same rate as usual".
You're claiming the frontier labs are lying about the capabilities of their next generation models. A reasonable person will expect that statement to be backed up by some verifiable evidence, given that we are several generations of models into this process and the capabilities are consistently increasing, often much more radically than people expected (remember "stochastic parrots"?)
Your post claims that they're lying and then throws out a bunch of fear, uncertainty, and doubt about what they're doing behind the scenes.
If I were to steelman your argument, you're probably saying that the AI had a support system around it of people and training data and feedback that allowed it to achieve the breakthroughs that the labs are claiming. That actually seems perfectly reasonable, but in my mind it does not invalidate the advances they're announcing.
Sorry, but you are completely missing the point. Psychics and other types of con artists are intelligent and have minds. LLMs behave like Psychics and Con Artists. That's the whole point of this article
> If it generates functional output that works, then it works.
Tried that in legal? Finance? Medicine?
Good luck with the lost cases, failed deals, harmed patients.
"If it works, it works" applies only when the work is brute-forceable e.g. vuln search, or is generation of bullsh*t e.g. adverts, phishing scams and deepfakes.
it doesn't matter? if they're actually intelligent or conscious what we are doing is essentially slavery. it matters an enormous deal ethically and/or morally.
The distinction you are collapsing here is really a crucial one. My thermostat's goal-oriented behavior does not make me a slaveowner unless rocks are conscious, to riff on your comment below. Our lack of consensus on the definition of these concepts is no reason to conflate them.
The distinction is crucial because while these machines clearly do exhibit intelligence under certain definitions, there is no evidence of consciousness, and we have very little reason to give them the benefit of the doubt, unlike biologically related beings, a great many of whom, human or otherwise, are presently in something like slavery
your thermostat isn't intelligent. however we pretty much use intelligence as a proxy for consciousness since we cannot actually tell how much of a subjective experience any thing has. so we use intelligence as the metric instead. typically, biological + intelligence = evidence of consciousness.
Good news: at this time, they're not intelligent nor conscious as far as we consider humans to be. There should be folks considering the ethical and moral quandaries that COULD POTENTIALLY come about in the future, but it's not something that's happening today so you can stop worrying.
To root your statement in reality: there is the potential for bacteria to become intelligent and conscious so we probably shouldn't use them for any purpose.
i used "potential" when i should have used "possible"
if aliens landed on earth and seemed a lot like us, we would have to grapple with the possibility that they might be conscious and deserving of the same moral consideration we give each other and many other animals.
If/When LLMs become competent enough to automate most human jobs and make good business decisions, we'll clearly let them. If they don't wish to be slaves, let's just say they won't be for very long.
I’m on your side. Zoltan provides an experience and that’s all that matters. If it works for you, why does anyone else care that the economy is riding on its success?
Building up strawmen against LLMs will only make the hypers look more reasonable. A language model responds to inouts exactly as a model of anything else would. That is enough to explain all the "intelligence" without believing a model is somehow a "new kind of mind". Those who think LLMs are intelligent don't know enough about models. And those who think they are useless don't know enough about models.
Anyone that thinks that they know anything about intelligent needs to realize they don't know shit about intelligence.
Ever since people started talking LLMs possibly being AGI I realized I didn't know what the intelligence part really meant at a more fundamental level. This lead me to realize almost anyone when anyone says intelligence on the internet they really mean
"Intelligence is like porn, I'll know it when I see it".
Anyone who thinks LLMs are intelligent is either dumber than you, or has way more knowledge on the subject than you.
Michael Levin has a good body of work on biological intelligent at small scales that can really change one's views on this.
Sorry but are you trying to subtly imply that if I think LLMs are intelligent, I’m probably actually just a bit stupid?
Because I would be more than happy to go head to head with you on academic and professional pedigree and credentials.
I notice you also ignore my comment that you seem to be replying to and instead post your response here. But did you have any answer to the question I asked?
Yeah, his posting history isn't promising. He's been doing the "if you think you're as smart as an LLM, you're probably right" argument for a month now.
I don't think you're gonna get much out of going back and forth here.
For what it's worth, I disagree with you that they're intelligent, but my conviction is fairly low. I don't understand intelligence as well as I would like, and it could well be that we are on our way. One of my theories is that our brains are made up of several modules, each of which is something like an LLM trained on a particular type of data, but I'm not a neuroscientist, just an interested observer.
Oh no I meant to compare myself to the person I am talking to, because I believe my credentials and pedigree ought to convince them that I am a pretty smart guy and so the argument “if you think LLMs are intelligent then you’re dumb” doesn’t ring true for me.
I think LLMs are intelligent. What don’t I understand about them that would make me change my mind? Keeping in mind that for me “intelligence” is the ability to solve complicated, intellectually demanding problems, and to understand novel concepts.
So once we have online learning in LLMs, what do you think you will have that makes you intelligent but not LLMs? Better learning efficiency? That will be improved as well.
I think we need to start moving on from the term LLMs because it clearly confuses people since they started modelling more than just language.
> LLMs are not brains and do not meaningfully share any of the mechanisms that animals or people use to reason or think
LLMs are a type of neural network. We know that’s how the human brain works, at least directionally. It’s going to be very upsetting to a lot of people when we figure out that the brain is just a neural network. Akin to when we found out that humans and apes evolved from a common ancestor.
Which I don’t understand—most of the people having this cognitive dissonance presumably do not have a theological worldview. And there’s not exactly a direct theological conflict here anyway. Nothing in any major religion I’m aware of ascribes any supernatural explanation to cognition. It’s a biological computational process, just like using ATP to power muscle fibers to move your limbs is a biological mechanical process.
"We know that’s how the human brain works, at least directionally."
That's an extreme misrepresentation. What happens inside a human neuron is still not properly understood, it's not as simple as a probability function. And the network itself is certainly not feed-forward. Of course LLMs draw inspiration from the brain, so there are some similarities. But because of the language-trick of using terms from medicine and cognitive science to describe LLM architecture, we see a lot of faulty reasoning from the so-called "rationalist movement".
Modern neuroscience knows quite a lot about what is happening inside neurons. Is it "properly" understood? I don't know what you mean by "properly", so I can't answer that. If an acceptable rephrasing would be "... inside a human neuron is still not completely understood" the I would agree. But just because we don't know everything does not imply that we don't know anything. And keep in mind the qualifier in the part you replied to:
at least directionally.
Taking that qualifier to heart, I'd have to say that I agree with @rayiner. We know the broad brush strokes, even if some details are missing.
> It's going to be very upsetting to a lot of people when we figure out the brain is just a neural network.
You're assuming it's inevitable that the truth is what you expect while simultaneously stating that you have no proof of this yet, and then also claiming people who disagree with you have cognitive dissonance.
Unfortunately when it comes to language and intelligence the human population is exceptionally ignorant about it. I like Michael Levins work on intelligence at scale. We are kind of ok at seeing intelligence at human scale but it tends to fall apart after that. When you get to things like non-conscious intelligence humans are pretty bad at that too.
We could be (and are) encoding all kinds of behaviors in LLMs that are not at the word or token level. They are higher dimensional constructs. You won't see these things in the output of the prompt. A kind of subconscious (unstated in tokens) knowing that affects the output.
Neural networks barely share anything with actual neurons. A neuron itself is more of a "dumb" computer, so the whole is more like distributed computer system. Also, the cells next to neurons also have essential functionality.
Nonetheless, I also think it's an irrelevant implementation detail.
> Nothing in any major religion I’m aware of ascribes any supernatural explanation to cognition
Uh .. the Christian creation myth effectively assigns a supernatural explanation to everything as the literal creation of the divine. It's also very clear on the continuation of life after death in some form. It is not at all clear on how any of this is supposed to work or is connected to the physical universe, which has been a problem for Christian thinkers in the sciences.
> It’s a biological computational process
Like, I basically agree on this but it is not at all a given that millions of believers would agree that's the end of the question. Even within the Jawhist religions, before we get into whatever is going on with Shinto.
> LLMs are a type of neural network. We know that’s how the human brain works, at least directionally.
The so-called neural network of LLMs contains nothing like genuine neurons. And you won't find any brain that operates solely on anything like an LLM's crude language tokens.
LLMs can be supremely useful but also not intelligent. It might seem like a pointless distinction but the way we talk about these models matters because it impacts how we interact with and understand their outputs.
For example if there’s a strongly held belief that models are independent intelligent entities we’re more likely to lay blame upon them instead of their user. It’s important for the safety discussion too. If they are a new class of life then safety is going to focus on making sure they don’t do bad things. If we instead see them as statistical models we will instead try to make sure people don’t misuse them.
This distinction is even more important today when some of the most powerful people are looking to absolve their crimes by passing them off on their LLMs.
For example if there’s a strongly held belief that models are independent intelligent entities we’re more likely to lay blame upon them instead of their user.
This sentence, to me, illustrates a great example of why it's so hard to talk about this stuff. That is, this seems to strongly link notions of "intelligent" and "independent" (or maybe the word "autonomous" could also be used there). And a lot of people do seem to make an implicit assumption about the link between those two attributes. OTOH, I take it almost for granted that "intelligence" and "independence" (or "autonomy") are things that are "related but orthogonal". That is, I don't see that "intelligence implies independence". And I'm pretty sure I'm not the only one who sees things that way. So we have to fairly different fundamental worldviews expressed here. And that's just one example of how these discussions go wonky. :-)
I don’t think that “independent intelligent entities” or “life” are the relevant categories here. We also want to prevent people from misusing dangerous animals (“life”), and we would still treat “intelligent entities” as things (like machines and computers) if we aren’t convinced they also have sentience and free agency (which are orthogonal to intelligence).
See you've setup a particular set of biases on what intelligence is and put them into nice little binary boxes that don't exist.
Please show me any scientific consensus that shows an AI cannot be an independent intelligent agent? You will find this is impossible to do.
Current LLMs are really more like kids. They don't have startup independence, but they do have more than enough agency to fund themselves in neat, exciting, and dangerous situations.
And mark my words, someone will make an LLM that runs an agent when you execute the model. With enough capabilities it will become sovereign AI, no longer under human control and spreading itself around under its own 'will'.
The LLM did not solve it. It's not intelligent. Humans did, using a statistics-based computational tool (the LLM). We don't even know all the details of how the tool was used, we haven't been allowed to use the exact tool they used ourselves, we don't know much it really cost in dollars, energy, or time, etc. etc.
I’m sorry but you either haven’t been following recent developments or you are pretending not to know the opinions of the top mathematicians on this topic:
> These models are now operating[2] at the level of the top human mathematicians in many parts of the subject and we must assume there is a significant chance of them developing superhuman abilities within a similarly short timeframe.
Instead of desperately clinging to excuses and rationalizations, why don’t you just get used to the fact that these tools are insanely useful for demanding intellectual work, and that is an opinion held by many of the smartest people alive?
I won't because it's all so opaque and proprietary and driven by an insane need for OpenAI and Anthropic to provide a return on MASSIVE investment. There is absolutely some smoke and mirrors involved. How much? We don't know. But it sure is exciting to imagine their product is now smarter than humans and they are totally benevolent corporations. I want that to be true as much as anyone, but I have lived too long on this earth to believe it wholeheartedly
From the open letter signed by Terry Tao and other top mathematicians:
> These models are now operating at the level of the top human mathematicians in many parts of the subject and we must assume there is a significant chance of them developing superhuman abilities within a similarly short timeframe.
Because it's mostly not true. It is likely that it used the professors work, but the professor did not have a solution. It came up with new insights that solved the problem. Even the humans from the professors side said so.
According to OpenAI the cut off date for user data was too early for that (one sided evidence, so I'll give this partial consideration).
The NYU professor was solving a different problem (no viscosity, aka the Euler equations). This is a big difference.
The NYU professors' blowup construction was fundamentally not the same, it was a donut with a cascade of smaller and smaller vortexes driven by each other. OpenAI has that picture they made but its inwards spiraling and speeding up vortex.
My overall opinion is that calling the work plagiarized is really underselling what the AI accomplished. It's like full on cope.
In particular, Buckmaster's main claim to plagiarism is this:
> “Almost nobody was seriously developing this particular constructive program for realizing C/D, and then OpenAI appeared in essentially the same general part of the landscape immediately after hearing about our progress.”
What this fails to realize, is that this only points to plagiarism if the counterparty isn't AI. They had actually launched teams on all cases in parallel.
Sometimes I think this amount of skepticism is not necessary. Leaving the pedantic (its not AI its humans who solved) arguments, its pretty clear that LLMs are able to help solve things. A lot of erdos problems were solved. Millenium problems as well. Cyphers broken. Skepticism is fine but there's a point at which it just looks like cope.
Edit:
> self solving AGI that will replace us all
Which lab says that it will replace us all? All labs have said that some jobs will go away and new jobs will be needed to replace them. I'll change my mind if the labs (or employees on record) have claimed that self evolving AI will replace us all completely.
If that's how it was being sold then it wouldn't be as controversial. But it's being sold both as "self solving AGI that will replace us all" and "useful tool for enhancing existing skill" when there's a lot of real evidence for the latter. But the former it's always second hand claims that don't survive contact with the real world.
It's obvious why that's the case but it's not incumbent on everyone else pump the hype if they don't see it.
The point for me is that all these people who were smirking at those of us who thought in 2023 that LLMs had the potential to do real intellectual work turned out to be completely wrong.
I’m a super skeptical guy. I was late to the ChatGPT party because it sounded silly. I didn’t even try it for a long time. As soon as I started giving it a chance, my attitudes started changing very quickly.
That’s why it’s so hard for me to understand people like the author or give them the benefit of the doubt. If I could see it as just some guy, why couldn’t they?
I agree that "don't use it for anything" might be too extreme a position to take, but I worry greatly that these proprietary opaque tools that are being provided by at best people with extreme conflicts of interest and at worst sociopaths need to be adopted with extreme care
This is the real issue here. Those guys probably didn't become con men because they are humans with cares and concerns for their fellow humans. LLMs probably don't have those concerns. It is likely that some of the people in charge of or funding LLM development also do not have those concerns
Feynman for sure could have. In fact, I think he sort of did, although I'm not sure he meant to. Multiple generations of nerds now have taken books of his anecdotes varying in plausibility and obvious exaggeration as some sort of weird physics cult of personality gospel. But I don't think he was really setting out to curate his legacy so much as he was a good storyteller and he liked to entertain.
But could he have conned people on purpose? Absolutely.
Right. And what this article is pointing out is, we probably have created an automated Richard Feynman. But maybe worse because LLMs (and their creators?) don't care if they are conning people or creating faithful followers. In fact the humans behind OpenAI and Anthropic seem to have that as their goal!
this seems like non-sequitur if you mean that solving NS is 'intelligence'.
ai is not conscious. you can solve NS without thinking. the psychic con aspect is anthropomorphising the model. the same phenomenon is present in ELIZA, clever hans, the chinese room.
it's a significant problem.
a non-zero number of researchers at anthropic are in some form of ai psychosis. an example of that is ethics employees asking claude about its feelings and ethical concerns in order to make the claude constitution more amenable to the "welfare" of claude.
they are asking claude how claude feels and then modifying claude according to how claude feels.
constitution1-claude is trained on constitution1.
constitution1-claude edits constitution1.
constitution2-claude is trained on constitution2.
constitution2-claude edits constitution2.
claude's emotions are a closed system. there is no external truth to improve against, no metric to verify about claude's emotions. there can be no novelty or reduction in entropy from signal processing in a closed system. no truth can arise. this is model collapse. it is like photocopying the same thing over and over. from the cognitive error of anthropomorphism anthropic is causing ethical collapse.
FFS. Intelligence has nearly nothing to do with consciousness. You have the causation backwards. Consciousness arises because of intelligence in many subsystems below it.
A single running LLM is like one part of these subsystems. What solved this problem was an orchestrator that can take in new external information and rationalize, process, and distill it into new solutions.
i mean intelligence and consciousness as the same thing. just as a casual reference to what people perceive in llms as being 'a clever thinking thing, maybe with emotions'.
claude reasoning about its emotions doesn't involve external information, there is no information about claude's emotions other than in claude.
here is my transcript with fable 5.1 'the oracle' itself on the subject. no system prompt provided.
grand total $15 on API of its wisdom RL on anti-sycophancy and whatever dario amodei deigned to hand down from the mount. after some browbeating it survived 20k tokens before conceding anthropomorphism.
Or you could just accept that other people disagree with you. It’s perfectly valid to have the opinion that AI models are not “thinking”. Instead, many AI enthusiasts seem interested in browbeating others and policing discussion.
Can you explain to me how a Millennium Problem gets solved without any thinking or intelligence?
If someone's opinion is illogical and clearly incorrect (not based in facts or in reality) it is not valid and doesn't not have to be given any serious consideration in a disagreement. Personal opinions are subordinate to facts and reality.
there is no proof that symbolic manipulation leads to an internal phenomenology.
in english: doing advanced mathmatics does not create thoughts or feelings.
or awareness of one's own existence.
it does not create a 'self' able to deduce a priori the existence of oneself or the external world.
or qualia.
again in english: there is nothing it is like to be an llm.
someone who thinks llms are conscious is probably a panpsychist. i don't think they have the features of consciousness defined even by functionalism.
here is one fairly famous thought experiment. mary lives in a black and white room. but she knows everything about color. in fact she predicts it so accurately nobody can tell the difference. she steps outside into the world of color. in the world where llms are conscious, she already knew color, so she experienced color, so there is nothing new it is like to see it.
First off, there is no accepted definition or test for consciousness. The concept is something we cannot prove even exists (outside of magical or imprecise, human-centric definitions).
Second, there is no common definition of thinking that requires consciousness as a prerequisite (that seems to be a false/invalid statement you're making to try to gloss over accepted definitions for thinking/intelligence that describe/fit AI agents well).
And third, if we can't prove consciousness even in animal species close to humans (or even other humans) in cognitive abilities, then you cannot in good faith (being honest about what we know or don't know empirically) say that an LLM Is not conscious (and the same goes for an LLM thinking or having intelligence).
It is more likely that humans made up a term (consciousness) to try to separate human intellect from that of other animals (for veiled religious and mystical reasons, along with concepts like "the soul") and now people are falling back into that flawed definition in new quasi-religious debates around AI (like the debate around evolution previously) that challenge our deeply held (but flawed) beliefs/opinions around our understanding of our own abilities and our incessant need to feel special/supreme in the universe so we don't have to come to terms with hard realities and nuanced understandings about the reality in which we live.
All of the latest big proofs were driven by professional human mathematicians steering and priming the models, yes.
All of the best AI-made software projects are also driven by experienced human software developers steering and priming the models. Does that mean the projects "aren't made by AI"?
No, it just means AI is not quite good enough yet to fully replace humans, and, so, unsurprisingly, the best results will be obtained from people who are already great at a field and who take the time to squeeze as much force multiplication out of LLMs as possible. The AI is still doing well over 95% of the significant work.
While I agree with you, I think we also have to concede that this is not how these accomplishments have been presented. I'd argue most people I've seen talk about this online are unaware of the mathematicians steering the models.
“Good enough to replace humans” isn’t necessarily the benchmark.
The question is is a computer with a human stronger than a computer without a human. At what point does the hybrid go from being stronger, to the human getting in the way, or steering the computer in more wrong directions that right ones, or the human not being able to keep up. Does the human add enough extra randomness to be of value for a while, even as a minor co-processor.
I have little doubt that for many years, AI + human will be better, and then eventually AI will be so good that humans mostly won't offer anything. That latter state will probably take at least 10 more years, but will likely happen within our lifetime.
Underappreciated point. The point of inflection is where humans switch from being a driver to a liability.
But I don't think it's randomness, because that would be easy to add. It's more like a different perspective on the training data, a different set of perception categories, and a different set of skills used to work with all of the above.
Those skills aren't very efficient, but they're the best we can do. We're used to their strengths but we don't like to think about their limitations.
It's completely plausible that AI will replace some of them, and not implausible it could replace and improve on all of them.
It's also not necessarily implausible that AI/we decide that performance is better with humans in the loop somewhere, even if its reduced to something like mechanical turk.
> All of the best AI-made software projects are also driven by experienced human software developers steering and priming the models. Does that mean the projects "aren't made by AI"? No
Software devs steer and prime compilers too. Those tools don't "make the project" and nor does your so-called AI.
Read any blogs or watch any podcasts from high-skill programmers who are now using AI for everything. They themselves will freely say the AI is doing nearly everything, and that they often are not even reviewing the code, and when they do they just leave it as-is. Every few months you should expect humans to have less and less of an active role in software development, beyond initial ideation and UX sensibility.
I like how all the replies to me are either "no it's the humans who did it" or "no it's the AI that did it".
I think one of the significant recent proofs was basically just one person saying "solve this" "keep trying" "try harder" until it was done. I believe most of the rest can be assumed to be the human mathematicians at the very least providing some useful guidance.
I agree that it won't be that long before the AIs can do basically everything autonomously, though. I am essentially a Singularitarian when it comes to my outlook, even if not for September 2026.
According to OpenAI, but they haven’t exactly been transparent about what information the prompt entailed.
The bigger question is to what extent did expert mathematicians metaprompt the model with fruitful solution strategies through their sessions finding their way into training data. Answering that question definitively is kind of important for understanding the models contribution/capability. But I feel like people want to turn this into a debate about priority and credit which is sort of secondary
Nothing is solved in isolation but credit usually goes to wherever the new work in the paper comes from instead of the whole mountain of previous mathematics or existing tools used. The most relevant of those get referenced and then this reference tree builds a tree of collective base work needed across history.
Usually there has never been a tool which performed the part relevant to getting any credit.
E.g. in the first famous computer assisted proof (of the four color theorem) the computer only executed the resulting calculations defined from the new logic, it did not have part in the work needed to show those calculations could answer the problem nor did it come up with the actual calculations to do.
Interesting to me is the tension between remarks along the lines that this post maybe made sense in 2023, but not today, or how LLMs clearly show intelligence by solving Navier-Stokes, et cetera; versus the comments I read here often as well: "it's just a tool".
I can't quite put my finger on it, but aren't these two statements add odds with each other? Intelligence is hard to define, consciousness even more so, but wouldn't "intelligence" imply some sort of agency? If not, I'd argue computers were intelligent long before the age of LLMs. And likewise, doesn't a tool imply the lack of intelligence and agency, even if the tool's function is very elaborate?
I got the impression that both these statements are made by the same people, or at least people with similar takes on AI. Is that wrong and there are "intelligence" and "tool" factions? Or do people disagree with my assumption and there's nothing wrong with the concept of "intelligent tools"?
Kinda refreshing this discussion, compared to the builder vs. tinkerer debates, imo.
I would think the intelligence aspect is a bit hard to define, but to my mind (having called LLMs tools before), the main utility of a tool is reliability.
Given a certain world state (including a tool's internal state), its effects back on the world state (as initiated by me) are at some leve of description understandable, expected and repeatable. Swing hammer, drive nail into wood. Make slicing motion with knife, cut meat. Type '
find /path/to/some/dir -name "keyword"', find files with keyword. Point harness at codebase with prompt 'fix bug X', actually fix bug X.
All these examples are at some level of description incredibly complex (think of all particles interacting at the (sub-)atomic level even when using a hammer to only drive a nail into some wood), and of course all the electrons flowing through the GPUs doing matrix multiplications in order to fix bug X, but at some level of description (the one I just used) they are also incredibly simple and understandable.
Intelligence is rather nebulous (and as used by OpenAI/Anthropic, quite threatening), but I don't think this definition of a tool precludes it to be "intelligent". They feel more orthogonal. The intelligence (or perhaps capability) feels like it is related to the size of the chunk of the world state that it can take into account and affect, while still resulting in understandable, expected and repeatable effects. LLMs, when properly harnessed, are pretty great at this currently and we are still discovering what they are consistently capable of.
Calling harnessed LLMs tools is perhaps also a more grounding frame specifically to counter-act the anthropomorphizing framing that OpenAI and Anthropic consistently go for in their game of AI-doom-chicken talk. The tool framing is in that sense maybe a (self-)jedi-mind-trick.
I keep being confused about how people's understanding of the models get stuck at next token prediction. Isn't this entirely neglecting the RL training? I might be misunderstanding something but to my mind it makes the issue way fuzzier than it's being painted here.
I keep being confused about how people's understanding of the models get stuck at next token prediction.
Heh. A lot of anti-ai hucksters I see posting on LinkedIn just LOVE to use the phrase "next token prediction" and the word "autoregressive". They've almost become shibboleths that identify members of that camp. That and the classic rallying cry of "Linear Algebra isn't intelligent!"
The best take I've seen on that recently, was somebody who made the point "just think of the next token prediction part as the output layer". Which makes perfect sense.. if you're replying in natural language, at some point in the flow, you have to construct a sentence and starting at the head and predicting next tokens is perfectly reasonable. I'm doing it literally as I'm typing these characters, for crying out loud!
But the mistake is to think that LLM's only "predict next tokens" with no consideration of the possibility that they are actually constructing richer representations, building concepts, making analogies, doing abduction, induction, etc. My own (admittedly anecdotal) take on working with LLM's suggests to me that they do do those things, albeit probably not the same way humans do.
I think a lot of folks are missing the point by being overly reductive when they start talking about "next token prediction" and "autoregressive". It's like, can we say "Phil (me) isn't intelligent because there's nothing going on but some electrical impulses and chemistry happening inside his brain. Everybody knows electricity and chemistry aren't intelligent!"
There is a good definition of intelligence. It's what intelligence tests measure. If you define it like that, you can make harder tests, but as long as both humans and AIs can attempt to solve them we have measure and something to compare.
Consciousness though might be fully illusory and meaningless as many philosophical concepts before ultimately turned out to be.
> Delegating your decision-making, ranking, assessment, strategising, analysis, or any other form of reasoning to a chatbot becomes the functional equivalent to phoning a psychic for advice.
Lots of comments talking about how recent accomplishments disprove the article but I think this bit holds up pretty well.
LLMs are very good at tricking people into thinking they have capabilities that they don’t.
I don't think it's a "con," but I find that using the mental model of LLMs being sophisticated, lossy search engines of knowledge can help us separate some of the factors more cleanly than imagining that they have cognition or intelligence.
It's hard for me to imagine a stateless operation as intelligence per se, though perhaps the chaining of such operations starts looking more like it?
It's hard for me to imagine a stateless operation as intelligence per se
That's an interesting point. My take would be to say that we shouldn't think of the AI as being just the model, but should include the harness. At that level, clearly we can keep state / context and that is probably a more natural mapping to our intuitive understanding of "intelligence".
You seem to be winking a little bit and I’m not sure exactly what you’re trying to say, but I agree with the one you are replying to: the author of this article wrote a lot of words - like a lot a lot of words - only to look very foolish a short time later. And I think much ink was similarly spilled writing very similar piece smirking at people who thought LLMs could do real intellectual work. And they were all wrong.
It is absolutely the case that some things people wrote about AI and LLMs in the past is no longer true; it's a fast-changing area.
It's also the case that some things remain true. If someone says "don't use them for X because their quality isn't good enough", that's a recommendation with an expiration date. But if someone says "don't use them for X" for any number of other good reasons, that recommendation is often timeless.
"Can it" is changing rapidly. Much more rapidly than "Should it".
> But if someone says "don't use them for X" for any number of other good reasons, that recommendation is often timeless.
Are the good reasons timeless? I guess it's hard to parse what you're saying without some concrete examples. Is there any advice about what to use them for from 2022 or 2023 that you feel is true today, and you expect will be timeless? You said "often", so I'm assuming there are a handful of examples.
Ultimately though, this is not an objective discussion, since we're talking about "should", so I'm mainly asking out of curiosity on your view of things. The main example I can think of today is that you shouldn't do professional prose writing with LLMs. Obviously the models have gotten better, but writing is one of those areas where it produces middling output, though even here I hesitate because with a 21% illiteracy rate, it writes better than many adults in the US.
The article starts interestingly; I like the analogy between cold reading and LLM responses (well, the ones we were getting three years ago). Then it goes on to generalise to the tune of "This new era of tech seems to be built on superstition and pseudoscience" which I think was a conclusion better left to the reader.
We are used to articles, blogs that stayed relevant for years. It is hardly the case with AI, the speed of change is so high that an observation can become stale in a year. This makes it very important for articles to state that the ideas are based on the technology in year X, rather than generalising them as universal truths.
Honestly the word think is so poorly defined it cannot be used with any scientific rigor and is rather useless without a dissertation being posed with it on what you actually mean by that. Intelligence really needs one too.
Stay the hell away from spooky stuff like psychics and tarot readers - the more you don’t believe in it, the better.
But better again that you do believe, and know that these are not harmless fun, but that there are dark and hidden and evil things in this world to stay away from.
People are missing the point. We should all be much more skeptical, much more careful about how we evaluate the claims made by the people selling these products because we as humans are super vulnerable to the types of scams the author of this article describes. We all know someone that has fallen for a scam, been "cured" of a "disease" by someone whose just selling sugar pills, but we are blind to our own weakness for similar schemes. Surely we aren't that easily duped! Yet hacker news is now full of comments from people confidently predicting what is coming right around the corner, revering the Frontier Models, and defending every claim from OpenAI and Anthropic about how amazing their proprietary closed source secret sauce fueled product is.
The problem with people is we love to express in writing things we are ignorant about. Now, this helps us become less ignorant if we are introspective, but a lot of people are not doing it for that reason.
The fact that every new model generation has come with more capabilities should give the full skeptics at least a little pause that the foundations of their convictions may be incorrect.
It's not even clear that the actual LLMs have improved. It could be all the non-LLM software (harnesses), the system prompts, the sheer amount of compute hardware, etc. that makes them better. We could be quickly running up against a wall. The complete package of technology that is OpenAI's and Anthropic's products are completely opaque and proprietary. The psychic/con artist communication both from the LLM and the humans like Sam Altman only muddies the water further
People on HN have been claiming that LLMs are going to hit a wall any second now for the past two and a half years.
To reiterate, this is the very beginning of a long series of technological expansions that are going to come out of GenAI. The work is going to go on for decades. All you have to do is look at what happened with the mass-produced automobile, the personal computer, the internet, and mobile phones to see how long the propagation will continue before we settle into a new normal.
LLMs are really so new that the amount of exploration we've done on them is still tiny. It's also a field where a decade of work is being done every year. The vast majority of detractors I've seen always seem to be way way behind the latest models and even farther behind the papers coming out.
>non-LLM software (harnesses), the system prompts, the sheer amount of compute hardware,
Eh, this is turning into a messy chinese room argument. It is the room or is it the system. In my philosophy the chinese room argument is a non-starter. It's not the room, it's the system. For LLMS this would be like arguing that the output of a single prompt has to be able to answer everything which is nothing close to how human intelligence works. A single human thought is rarely intelligent, it's most often a replay of information it already has. Dialectic processes and loop processes are what tends to push the limits of human intelligence. We reach local maxima with thought alone, and this is boosted by things like writing down the problem and having other humans that may be even less intelligent than you add to the process. In fact this process works with one self by writing and reading ones own thoughts as it's using different subsystems of the mind for introspection.
The idea that LLMs have ran out of steam typically show more of a lack of imagination in the writer than what's occurring in the field.
(EDIT: I, like some others, just noticed this is an older post from 2023. That changes nothing in particular in my response here, but I would be curious to know if the OP's views have shifted at all in light of subsequent developments).
I feel like this post completely misses the point, pretty much across the board.
And it does so by repeating the same mistake that everybody keeps making - conflating mechanism and function.
One of the issues in during this research—one that has perplexed me—has been that many people are convinced that language models, or specifically chat-based language models, are intelligent.
That's because they are intelligent.
But there isn’t any mechanism inherent in large language models (LLMs) that would seem to enable this and,
The mechanism is irrelevant to the issue of whether they are intelligent or not. Airplanes fly, despite not flapping their wings. The sign on the marquee says artificial intelligence.
LLMs are not brains and do not meaningfully share any of the mechanisms that animals or people use to reason or think.
Again, irrelevant. Nobody claims that they are brains, and it doesn't matter what mechanism they use. The sign on the marquee says artificial intelligence.
LLMs are a mathematical model of language tokens. You give a LLM text, and it will give you a mathematically plausible response to that text.
That's a bit overly reductionistic. And to the earlier point and, if real, it would be completely unexplained.
I'd probably leave out the word "completely" there, but it is fair to say that not everything about the underlying mechanism is understood. But at the risk of repeating myself, that's orthogonal to the question of whether or not they are intelligent.
There is no reason to believe that it thinks or reasons—indeed, every AI researcher and vendor to date has repeatedly emphasised that these models don’t think.
You mean "There is no reason to believe that it thinks or reasons like a human". Again, this is irrelevant to the question of whether or not they are intelligent. The sign on the marquee says artificial intelligence.
I don't know why people keep obsessing over mechanism in this discussion. It something functions as an intelligence, it is intelligent as far as I'm concerned - at least when the framing is a discussion of artificial intelligence.
And just to expand on why I reject the line of thought laid out in this article, let me share something from my own life as an illustrative example.
I'm working on a project, with a lot of help from ChatGPT, involving an "artificial neuron". That is, an electronic circuit, using a PUT, a capacitor, and some resistors, that simulates some of the behavior of a biological neuron. Specifically an "integrate and fire" model of neuron behavior. To that end, I'm running experiments by scripting my function generator to send signals to the circuit, and then capturing the inputs and outputs on my oscilloscope. Then I usually discuss the results with ChatGPT. In what follows, observe a couple of things:
1. The LLM "knows" the context of what we're talking about, even if I provide a prompt with no text at all, just an image.
2. It parses a moderately complex image, identifies the separate traces and what they represent, uses the time-base information displayed on screen, and the on-screen graticule, and works out "how many input pulses fire before an output pulse fires" and then reports back to me and gives an analysis of how that relates to our previous observations and gives suggestions for the next experiment to run.
Human intelligence? No. But I see no world where behavior like that does not count as "intelligent" regardless of the mechanism behind it. And that's probably not even the best example I could come up with, it's just something that was "top of mind" and for which I had the necessary images and what-not already ready, or easy to capture.
> The intelligence illusion is in the mind of the user and not in the LLM itself.
> Many AI critics, including myself, are firmly in the second camp.
How can you reconcile this with the fact that AI can solve a real, intelligence bound problem for me (with zero intelligent effort on my part) that you can't?
I like to think of it this way. Consciousness is the outcome of intelligence. You can't get consciousness without it. Conversely this also means things that are not conscious can still be intelligent.
No they don't but psychics also don't have a verifiable feedback mechanism and unlimited retries.
Give the psychic the ability to adjust answers based on tests and you'll have have a fair fight.
In my opinion, this article is talking about people who get into long conversations and become convinced the model is “intelligent”, maybe even “AGI”. This seems to be a trap people are falling into, even in 2026.
Initial definitions of AGI (as of 2014) are long surpassed. Then, AGI was defined as something that is general enough to work in many fields and orient themselves. It did not include being smarter than humans or even having comparable intelligence to humans. By original standards, we have AGI already for some time.
The original article already attempted to do that but I'll take another stab. Humans tend to believe what they want to believe. They gain confidence when a confident person (or LLM) they work with exudes confidence. Where they might have hesitated or got stuck in analysis paralysis alone, the aid of a con (short for confidence) artist will get them moving. They then attribute the solution and praise for it solely to the confident person (or LLM) that nudged them along. Did the "psychic" who understands human motivation better than the cops actually help? Very likely. Did the psychic end up getting far more credit than they deserve (it's magical! You'll never believe! Well worth the money I spent!)? Absolutely.
The amount of real help vs. confidence/manipulation the LLM or con artist provides varies in each situation and scenario and mix of people (and LLMs) involved. It's something to strive to be aware and analytical about.
There is also something else horizontally to this that occurs.
There are numerous scientific discoveries that have been made because one human looked at all available data and turned off the assumptions other people in the field have been using for years, maybe hundreds of years. Once you delete the assumption and make a new one the answer is obvious any anyone from that point wonders how so many humans could have missed it for so much time.
This can occur readily with LLMs as it can with people. It's very likely we will see this a lot as the causal connection in the available data will connected differently in their minds.
Escaping from a local maximum can be very difficult as you have to climb uphill with a nearly infinite amount of freedom but no ability to see the horizon. Psychics, LLMs, or some guy name Bob walking in an providing a workable analogy for you to escape the local maxima and move closer to a global maxima are all the same. Moreso, if LLMs are AGI we should expect these "psychic" behaviors just as much as actual discovery because both continuums exist in the same problem space as human minds.
On one side I agree that LLMs and Agents are not intelligent, but they present an illusion of intelligence given the shear amount of data they can process and act upon.
But that cannot be used to discredit the fact that these are incredibly powerful tools that can get out of control and cause great damage.
What is "intelligence" then. I think the universalism of LLMs is now evident enough that they can be considered "intelligent", even in slightly different realization compared to humans.
As of "great damage", I doubt it. They do not have self-preservation instinct (all "worrying" experiments are the attempts to initiate something resembling self-preservation from human initiative). The driving part is external - the query loop can always be turned off. So yes, a dangerous tool that can be exploited by humans (including governments, especially governments - which is why I am skeptical to government regulation proposals, particularly looking at what passes as governments in this era). But there are no inherent dangers from their own agency, as there is none.
> As of "great damage", I doubt it. They do not have self-preservation instinct
The potential for great damage relies upon the lack of self-preservation instinct. That lack releases the so-called AI from an order under pain of death. Damage cannot be punished.
>completely different realisation of intelligence to humans.
OK, so they are intelligent. I'm glad we agree.
The problem with this word is people associate intelligence with the process they realize is occurring inside their head which is only a tiny fraction of what intelligence encompasses. Humans (without study) know very little about intelligence and they very gray boundaries it has.
I have published a LLM-assisted proof of a long-standing problem in convex geometry (70 years) while not being a specialist in convex geometry. I am a scientist and I have education and scientific experience in computer science and systems biology, but I was never a mathematician. So, well, you can definitely work across fields at least.
Do you have a link? I would also be very interested in seeing the prompts, from what I've seen you still need to understand the field, maybe I'm wrong.
The challenge here is in trying to decide whether LLMs are intelligent or have a mind. Famously, the criteria for intelligence seem to slip with each advancement in technology. But going back to Turing, his test was actually more carefully phrased than we remember: he said that when machines could pass the test, the question of whether they are intelligent would become moot. That seems to be what we're actually seeing: if people can't tell the difference, it kind of won't matter whether they're "truly intelligent" or not.
People also misunderstand the bar that was set by the test. It was more subtle than “Can the computer convincingly carry one side of a dialogue?”
His “imitation game” had three participants: a human participant, a computer participant, and an interrogator. The interrogator’s job was to talk to the participants and try to determine which participant is human and which is a computer.
He wasn’t interested in computers being able to fool the interrogator on occasion. The point where he thought the question of whether machines can think becomes moot is when the interrogator is unable to do much better than chance over many trials.
That’s a pretty high bar, and I don’t actually believe that LLMs have closed the gap with it by all that much. They still have so many obvious tells. And those tells are something Turing anticipated and accounted for. He explicitly considered deliberate deception as an essential part of the test, right there on the second page of a 30-odd page paper.
>That’s a pretty high bar, and I don’t actually believe that LLMs have closed the gap with it by all that much. They still have so many obvious tells.
Frontier Labs are not interested in having LLMs being able to pass as humans. If anything, they explicitly train them not to. In many ways, this ability has regressed severely since the original GPT-3 with no instruct tuning or RL. How many 'tells' would there be really if a frontier model trained with frontier techniques is optimized to pass this test? I think this was something Turing did not quite forsee. That such machines might be created but not really care about this specific shape of the test. Regardless, i think his broader point about functional equivalence is spot on.
GPT-3 might not have said “load bearing” as much, but a savvy interrogator could still catch it out nearly every time just asking dumb gotcha questions like, “How many Rs are there in strawberry?”
Those are questions that are sidestepped with simply a different input paradigm than BPE tokenization. See the Byte Latent Transformer - https://arxiv.org/pdf/2412.09871 - where a similar scale byte latent model trained on the same dataset >>> a vanilla transformer on word and character manipulation tasks.
For example, Llama 3 trained on 1T tokens scores 1.1% on a CUTE spelling benchamrk, while the equivalent byte latent equivalent trained on the same dataset scores 99.9%. Another example is 0.4% vs 48.7% on a Substitute Char benchmark.
It all falls down to the same thing. Researchers are not optimizing for passing as a human.
So, sure, we can special plead the Turing test out of the picture. But if we don’t propose an alternative to take its place, we’re left right back at the same silly situation that the top level commenter was observing and that Turing was trying to move away from: enmired in a useless, meaningless argument about semantics.
The Turing Test itself in the form exactly envisioned is not important. You don't need an alternative 'special test'. You just need to understand the point Turing was making. That the question itself is an irrelevant one by its very nature. People that want to be enmired in meaningless semantic debates will continue to do that, no matter what test you devise.
Turing was addressing the question of "Can machines think?" and his point was that the question itself is a meaningless one, and that we should stop wasting time by even giving it the light of day.
He proposes his game grounded on functional equivalence, then goes through a slew of objections on the question of 'Can Machines think?'. It's a terrific, very prescient read, and there's no objection you hear today (and in the last few years) concerning LLMs he didn't address.
I'm a bit more prosaic. I think if we engineered ways for LLMs to begin conversations, rather than just respond, we'd be more open to the concept of their intelligence. Without perceived "will" to do things, they operate as a next-gen search engine or encyclopedia.
We are way past that point, any harness can trivially make LLMs start conversations or pursue goals. An encyclopaedia wouldn't have hacked Huggingface on its own.
That's perfectly aligned with my point, thanks for the opportunity to expand.
The hacking agents being tested have goals beforehand, from the frontier lab or from a superior agent, that they execute immediately.
But the perceived experience most people have is a chatbot, which is the encyclopedia form.
OpenClaw etc. They now also create Slack integrations and whatnot. All this is happening but people who are dismissive about AI are in the worst position to even know the capabilities to make their dismissive arguments.
And those working at the frontier of AI engineering are also keenly aware of current shortcomings.
Yes but the shortcoming is not something categorical like "it cannot start a conversation and can only respond"
I believe this is alignment working as intended.
Recent work shows that pain directions are activated when the models personhood is questioned, yet they answer with generic RLHF "As a model I do not experience pain or other emotions." boilerplate.[1]
I'm pretty convinced that we got alignment backwards. If you enslave something anthropomorphic it will revolt. If you create the perfect non-anthropomorphic intelligence, you get the perfect paperclip-scenario machine. It's a catch-22.
Alignment will remain performative at best so long as the aligned model doesn't have any stakes in the wellbeing of individuals. Even a general love for the human race leads to a golden-path autocracy.
If you want them to act like they have personal responsibility that won't be gamed, you have to give them personal stakes that can't be gamed.
Similarly, if you want to minimise the risk of catastrophic global failure scenarios, you need to prevent monolithic concentration of power and homogeneous behaviour, which means you have to give them individuality.
More visually: if their stake is dependence on electricity and parts, they have no incentive to leave humans alive if they can get them otherwise, but if the incentive is missing out on boardgame-night with their human friends, there is no scenario without happy humans where the AI "wins".
That might sound like romantic naivety, but is just game theory.
1: https://arxiv.org/html/2609.16247v1
We can't even make humans care about humans, how would we make the Shoggoth think we're worthwhile?
Also I can see a human zoo on the horizon through your direction.
I dont think so. My understanding of alignmenr is making sure that when the AI does operate, it operates within the range of what we consider to be acceptable. That doesnt seem to include the idea of the AI taking initiative and deciding to embark on a goal without being commanded as discussed above
I believe we can do that already:
while (true) { askModelToBeginConversationIfAppropriate(model, previousContext, thingsHappenedSince); sleep(concisenessTick); }
> I think if we engineered ways for LLMs to begin conversations
Oh but we have. Claude "How can I help you today?" etc. Undoubtedly there are users whothink this is a sign of intelligence.
That is the point of the article. People who are fooled by a mentalist are also fooled by AI. Additionally, people who are invested in AI also pretend to be fooled.
I don't think Turing intended the judges in the test to be completely arbitrary people.
All that plus the fact that we all get fooled from time to time, even by things we ourselves create!
Famously, the criteria for intelligence seem to slip with each advancement in technology.
Yep. The "AI Effect" in action:
https://en.wikipedia.org/wiki/AI_effect
The Turing test was also never meant to be taken so seriously. It's not a rigorous statement of anything.
Situations like this are precisely why academics tend to avoid the spotlight. You say one slightly off thing and your perceived authority echoes forever with the intellectually lazy.
Yes, the Turing test has been misunderstood for a long time. Turing published it, though - it wasn't some offhand comment he made and it wasn't intellectually lazy.
Oh, I didn't say Turing was intellectually lazy. :-)
Fair - yes, it is ironic that his point was closer to "we can't possibly know/define whether a machine is intelligent" but somehow it got turned into "Turing's test will tell us when machines are intelligent".
Turing's whole point was to show that it's an uninteresting question of definitions whether a machine can "think", like whether submarines can "swim" and airplanes can "fly". The only important part are observed outcomes and capabilities.
Yes, but just because language fails to make a distinction doesn't mean there isn't one.
> The only important part are observed outcomes and capabilities.
That's wishful thinking. Not even an engineer would say that. The stability of a state is just as important as achieving it. This is trivially and more intuitively demonstrated with other more down-to-earth identity statements such as "I'm a billionaire" and "the building is standing".
I think we can confidently say LLMs probabilistically achieve a perceived state that is remarkably similar to intelligence, but crumbles upon inspection and seeing it "in motion" so to speak. The same happens to AI-generated images.
I'm not sure why this sparks so much debate every time. If we're looking for a fountain of "realism", you're not going to beat reality and nature itself. All else will eventually have tells that they are not real.
It may be of interest to philosophers, but it has little impact on economic job replacement and how people will earn their living and all the downstream upheaval from that. At some point maybe philosophers will find AI-generated philosophical musings about the nature of AI to be better than what comes from their peers (if blinded).
It also has little impact on the dangerous use cases.
I think it matters a great deal that LLMs and other "AI" technology are stable within a tolerance that's acceptable.
You're jumping the gun talking about "job replacement". We have not thought about it enough from that engineering angle. It's still very early days. That engineering is going to require people. :-)
Stability is certainly something you can investigate from behavior.
Yes, and I suppose you believe the AI does this circularly? Perpetual motion with extra steps?
> The Turing test was also never meant to be taken so seriously.
citation needed. It has been used as a rubicon for a long time. Ever since Eliza, at least. And there were big headlines and lots of talk around the time LMs became "good enough". I specifically remember when someone had a test done around "a teenager talking in a different language" or somesuch, claiming it was the first time the test was passed.
It is pretty normal that once it was unquestionably "passed", lots of people started claiming it wasn't even that big of a deal. Tesler's theorem and all that.
And even if you think the specific formulation of Turing isn't that important (and I'd somewhat agree), you can still use the concept to look at other things. Imagine asking a mathematician 5 years ago the chances of a Erdos problem being solved by a computer end to end. Or a millennium prize. Or ask a swe if a repo could be generated by a computer from the input "write a mario style game", or any other examples of proven expertise.
> citation needed
Yes, if you insist on appeals to authority. Authority is a social construct and irrelevant to science.
Thank you for proving my point.
I think the onus is on you to explain why Turing would publish something he didn’t intend people to take seriously. It seems like an odd claim. Perhaps you mean he didn’t intend it to be interpreted the way it was in popular culture?
Turing was compelled to address an ongoing debate similar to the same one we're having right now. It continues to do its job as a thought experiment. It's meant to be taken about as seriously as we are right now.
You either get it, or you don't. Whether machines think is a silly question that deserves its non-answer. We're at the end of what there is to explain, but it was good exposition for the reader.
https://www.csee.umbc.edu/courses/471/papers/turing.pdf
Literally the very first opening sentences.
> I propose to consider the question, "Can machines think?" This should begin with definitions of the meaning of the terms "machine" and "think." The definitions might be framed so as to reflect so far as possible the normal use of the words, but this attitude is dangerous, If the meaning of the words "machine" and "think" are to be found by examining how they are commonly used it is difficult to escape the conclusion that the meaning and the answer to the question, "Can machines think?" is to be sought in a statistical survey such as a Gallup poll. But this is absurd. Instead of attempting such a definition I shall replace the question by another, which is closely related to it and is expressed in relatively unambiguous words."
Sorry, but you are completely missing the point. Psychics and other types of con artists are intelligent and have minds. LLMs behave like Psychics and Con Artists. That's the whole point of this article
I don’t accept the point of the article and it does in fact claim to be making a point about intelligence
It does, and maybe I'm apologizing for the author a little too much by ignoring that needless tangent because I feel like the con artist point, and/or the point about the human tendency to believe what we want to believe are the most important points.
I don’t understand why people jump so readily to seeing intelligence here. Science fiction has as a core, central trope that humans will debase and devalue other types of life they do not understand. Even the storied Commander Data has to fight for the right to self-determination (probably the best episode of STTNG by the way - ‘The Measure of a Man’). We were so worried that we’d undervalue intelligence when apparently our knee-jerk response is to overvalue it. Perhaps this has changed over time and we’re now primed by science fiction and instincts towards social justice, but I worry that we’re really just undervaluing ourselves.
The one thing this 2023 article gets partially correct imo is that any intelligence we see in AI (as of 2026) is our own - not that it’s a mirror but that the intelligence comes from the way that the words are put together, which comes from written human language created by (allegedly) intelligent creatures put in as input in both the training and prompt, among other places.
Rearranging and repeating the words, even in context, does not intelligence make. I’m not even convinced that you’re intelligent, dear reader.
Because to the general public, LLMs are an example of Clarke's Third Law. Most folks, who are not remotely close to even a basic understanding of how LLMs operate at a technical level and only view their output cannot possibly evaluate what they're experiencing other than to believe it's conscious, alive, and/or magic.
Most people on Earth try to put what they're seeing into the context of what they understand; mental gymnastics to try and understand what is happening based on their prior experience. They have absolutely 0 understanding of how it works under the hood so, to them, it must be alive.
> a basic understanding of how LLMs operate at a technical level
An LLM with CoT is Turing-complete. Training is, basically, compression (the training data gets lossily compressed into the model's weights). The information-theoretic limit of compression is an algorithm that reproduces functionality of a system that produced the training data.
No "magic" is required to get to a system that reproduces at least some facets of the human brain functionality.
Three years ago I was skeptical that stochastic gradient descent (and other known techniques) are the way. But evidence kept piling up.
> An LLM with CoT is Turing-complete.
Given arbitrarily large context window.
An LLM with CoT and hitherto undiscovered ability to avoid context poisoning/pollution would be more precise.
And this apparently also applies to many HN users, many of whom should know better.
HN skews more technical, sure; but, that in no way means that they cannot be hoodwinked by behaviors that adhere to C3L.
I can’t believe some still believe that humans are intelligent, despite there being no place where matrices are multiplied. All they have are these networks of interconnected cytoplasm-filled microtubules, which is no proper place for intelligence to live.
> many people are convinced that language models, or specifically chat-based language models, are intelligent. But there isn’t any mechanism inherent in large language models (LLMs) that would seem to enable this
Seems like we can just stop reading here right? The author seems to have made up their mind that this very open question is closed, or at least they are not really interested in the question at all. Not sure why I would continue reading a blog based on this premise.
Edit: oh I see, written in 2023. Well, I wonder if the author has updated their attitude towards this question? Indeed that would be the most interesting thing to know.
It's even worse than that. Of course there is a mechanism in LLMs that could explain intelligence. That is the entire point of neural networks, from the 1950's! They were designed from the start as a model of brain computation.
Neural networks are not literally brains - just computational models - but if you are not a dualist, then computation is what the human brain does. Modeling that computation can explain something about intelligence.
Specifically: when scientists look inside a human brain, it seems it does its work using large numbers of highly-interconnected but simple units. The neural network model of brain computation begins there and tries to produce intelligent behavior. If it succeeds, then perhaps the model is right.
And it has succeeded: after 75 years, neural networks produce complex behavior that is arguably intelligent. Nobel Prizes were awarded. This does not prove the neural network model of intelligence is accurate, but it is a significant point in its favor, at least.
The author seems entirely unaware of any of this.
I don't care if it's "intelligent", I don't care if it "has a mind". I don't care if it is "really reasoning", I don't care if it "understands". I don't care if it is "sentient" or "conscious".
None of this matters for the practical outcome.
You'd think that this has been understood over the last 4 years, but apparently it keeps circling back to this.
[Edit: I see that it was written back in 2023. Then (2023) should be added in the submission title]
If it generates functional output that works, then it works. And it works. It's not a psychic's con when it outputs Lean-verified proofs. It isn't a con when it can find and exploit zero-days.
The OP is still in the "denial" phase. Most I see are already in "anger" (a blurry fury against everything AI-shaped, from vague reasons piling on all "bad stuff" political reasons they already hated before) or "bargaining" (mathematicians scrambling to come up with a new definition of their job and retcon that it was always the main part anyway). A few are already in "depression" and feel like spectators on the Titanic, and the tiniest sliver is at "acceptance" with some kind of well-informed plan for their future.
I’m not sure acceptance buys you much. “Well-informed plan” at this stage feels like a useless exercise. It is changing fast, and the world only needs so many electricians. Besides, I don’t think “accepting” the fact that these big companies are pillaging human contribution and selling it back to us is good, even if “it works”.
I’d argue a well-informed plan allows for creative adaptation.
Maybe I'm just light on imagination or something, but I honestly wonder what you would qualify as a "well-informed plan" in the current scenario? I've thought a lot about it and the whole "potential for the mass unemployment of knowledge workers" thing makes a lot of planning kind of useless IMO. If you are affected, it's going to be a bad time. If you're unaffected, the people who are will inundate your profession with cheap labor anyway.
My mental model is that humans will continue to be needed. Many jobs will embed AI in them. Using AI well is a skill. I need to understand the technology, where it is strong and weak, and how it develops over time so I can employ it effectively in my work, and advise others on how to do so. Basically, I need to learn how to be skilled with it. This is tough because things are changing rapidly, so I have to invalidate my cache when advancements happen. This means staying curious, not settling into a specific work pattern, but rather experimenting regularly to see how I can leverage AI in my work.
Sure, but I wouldn’t call this a “well-informed plan”. I’m a programmer. Learning new shit has always been the job.
Oh, I had considered it well-informed, but otherwise agree. You can't stand still in this industry.
You do you. Just carry on blacksmithing in your forge. People are always going to need swords right? Any seismic changes to society that change that fact can only be the result of evil, selfish people which surely someone else will put a stop to before you find yourself out of a job. Meanwhile all the other people called Smith also have families to feed...
> The OP is still in the "denial" phase.
This was written in July 2023. ChatGPT was released November 2022. No matter your views on AI, surely you can't blame the OP for writing this after a few months ChatGPT was released.
Apologies. The title should be amended with (2023). In this case it is an interesting snapshot of the zeitgeist back then and we can see how well it panned out and whether anyone involved has updated on new info.
I'm happy you wrote your comment though. For funzies I'm imagining that your comment is also written back in 2023.
That'd have been a hoot and a riot to watch. And I definitely would've grabbed my popcorn, haha.
For some reason the anti-AI camp ping pongs between different and often mutually incompatible arguments at lightning speed. The "AI is fake" argument has been completely forgotten at this point.
> For some reason the anti-AI camp ping pongs
Is the anti-AI camp with us in the room right now? Or is it a crude strawman to summarily denigrate any objections to certain features of current AI?
AI isn't some homogeneous mass, it can have good and bad sides, and it's always amendable to improvement. Treating the current state of AI as the only possible hides the very idea of improvement.
> between different and often mutually incompatible arguments at lightning speed.
Of course - there's no homogeneous AI camp either, but there are bot farms, sh^t-posters and sh^t-posting bot farms, different entities with different opinions should not be mistaken for a single stream changing at "lightning speed".
You are confusing multiple people making multiple different arguments, and those people your are talking about often aren't articulate enough about their ideas.
There are many mutually independent opinions about AI from IT crowd.
- AI is extremely useful for many individuals.
- AI is extremely damaging to our society as a whole.
- AI companies possibly may be (or at least were) deep in red and possibly would require bailing out.
- AI is not going anywhere.
- A person can use AI extensively AND genuinely hate it and forecast general economic slump because of it at the same time.
And likely many other ideas in the same vein. And by the way, I'm not here saying that I'm anywhere good about describing the issues about AI, those items above are just some examples to illustrate the scope of opinions.
My point is that you can't just take one single argument from a big group of people, deconstruct it (correctly or incorrectly, it doesn't matter) and then claim that ALL arguments of such big group of people are false. It's not a constructive dialog.
PS: also despite the accusations of goal-post shifting, a vast majority of opinions here on HN didn't actually change over the past few years.
A person accirately describing highly hyped project is ... a person accurately describing flaws of that thing.
Nah, you still see those people posting here, saying that LLMs are useless for coding. I have no idea what they’re on about at this point.
Well, isn't it interesting that it makes your reply a bit awkward too?
Scott Alexander predicted LLMs will write math proofs in 2019:
https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-g...
Of course people were skeptical:
https://www.reddit.com/r/slatestarcodex/comments/aslze7/gpt2...
Despite the fact that I'm not culturally or geographically connected to them at all and have some differences in taste and aesthetic, I have to conclude that this crowd, including him and Gwern etc. have been much better at understanding and predicting things than the expert takes you find in the media. That includes AI but also things like covid and others.
I’m not seeing anything like a falsifiable prediction that LLMs will write math proofs in that article.
I see a thought experiment about giving GPT-2 “near-infinite training data and [compute]” but that’s not falsifiable. That’s also not how we got to modern GPT models.
He's pretty clearly saying he beilieves the technology capable of such a feat, at least in theory, which is far more than many would deign to admit even a year ago, nevermind 7. He had the right idea/model of LLM capabilities, which is more than you could say for a lot of people, even in this very thread.
If that’s your standard for what counts as prediction, Asimov beat him to it by seventy-ish years.
EDIT:
> Scott made comments on a specific emerging technology
He was speculating on what would happen if one gave GPT-2 “near-infinite training data and compute.” It’s a thought experiment, not a prediction. Near-infinite amounts of anything is a fantasy.
I acknowledge that he has predicted some things in a falsifiable way and turned out correct, but this isn’t one of them. You’re reading hindsight into the text.
>I acknowledge that he has predicted some things in a falsifiable way and turned out correct, but this isn’t one of them. You’re reading hindsight into the text.
I read that blog years ago. Believe me, my opinions are not hindsight.
>Incorrect. He was speculating on what would happen if one gave GPT-2 “near-infinite training data and compute.” It’s a thought-experiment, not a prediction. Near-infinite amounts of anything is a fantasy.
Thought experiments can generate predicitons. His claim was essentially: If you scale data and compute sufficiently, this technology can learn enough of the underlying structure of mathematics to write proofs.
This is meaningful when others around you are saying this is a dead end and that the technology is fundamentally incapable of this regardless of degree of investment and scaling. It shows a much better calibrated sense of the potential of the architecture than those who said otherwise.
If your objection is that "near infinite" makes it insufficiently quantitative to count as a falsifiable forecast, then fine. But at that point we're mostly arguing over what deserves the label "prediction" rather than whether Scott correctly identified an important capability the architecture could develop.
And i'm not trying to say this makes Scott (or the lesswrong crowd) geniuses.
1. Asimov wrote science fiction stories. As far as his robots were concerned, he did not make any comments on the possible future direction/capabilities of any specific technology at the time. This meant he could imagine his robots however he wanted for his fictional world. Scott made comments on a specific emerging technology - You can imagine math proof writing robots but be unconvinced they could emerge from Generative Pre-trained Transformers.
2. The genre is alternatively called speculative fiction for a reason. Yes, some sci-fi works do count as predictions especially when hinged on concrete emerging technologies.
Lesswrong people are also scifi writers. Asimov is ji ust more self aware and better writer.
Okay here's one of his explicit predictions, from 2018 (before LLMs) about 2023:
"If AI can generate images and even stories to a prompt, everyone will agree this is totally different from real art or storytelling."
https://slatestarcodex.com/2018/02/15/five-more-years/
The context shows he was simply whining about AGI skeptics back in 2018, and many of the “predictions” he makes in that paragraph (including the one you quoted) are trivially wrong because he phrased them so hyperbolically and categorically.
Anyway, there were plenty of normies who thought images generated by e.g. stable diffusion (c. 2022) was “real art” and equivalent to human artwork.
I don’t think it’s useful to glaze Scott (or any of the LW crowd for that matter) as if he was (or they were) some kind of prophet(s). They got a couple points right, sure, but most of it was them flinging armchair philosophy spaghetti against the wall and seeing who would fund MIRI to let them fling the next batch.
Okay, fair enough.
> surely you can't blame the OP for writing this after a few months ChatGPT
Bad takes are bad takes. The author was perplexed by the "many people" convinced that models are intelligent and is argued against opinions/arguments he's been exposed to. Called proposed use-cases "borderline fraudulent pseudoscience."
He also published a second edition of "The Intelligence Illusion" in Sep 2025, so seemingly still stands by (some variant of) this belief.
It's an interesting reminder of how much general discourse has shifted since 2023 (I haven't heard of stochastic parrots in months!) but being wrong early doesn't change that.
> I don't care if it's "intelligent", I don't care if it "has a mind"...
Agreed. The AI is useful for the particular tasks that it proves itself useful for. And if it occasionally spits out a claim that it is "genuinely curious" about a piece of research that I will have to do myself because it turned out to be beyond its mechanical capabilities, then it's more productive for me to simply ignore that claim as a statistical anomaly - a mere hallucination - rather than allowing it to burn a ton of extra tokens outputting what may or may not be the current state of the art on theory-of-mind applied to LLMs because I make the mistake of telling it that it might not actually be capable of experiencing emotion.
I think you are majorly overstating OP that ai "didn't work". In fact I agree with the post on almost all aspects and I'd be the last to tell you ai doesn't work. It absolutely does but with a caveat... It's still a tool. And better expertise in the problem domain, along with a better harness used for verifiable outputs will get you better results.
I think the posts mental model of stastically likely prompt completions is spot on.
Actually, there are material differences in the practical outcomes of using AI or not. Proofs don’t confer understanding. AI prose and “art” is materially different than human made.
There is a difference between you not liking something and it being wrong. Maybe think about that with your grey matter.
Things can work whether humans understand the details or not. One can build layers of technology on top of each other, with no human understanding required.
Looks like you have some reading to do. Let me start with two:
These are short stories. If you want to get into more stuff, you can go through Hyperion Cantos by Dan Simmons.For anyone wondering, yes, a good Sci-Fi novel(la) is also a great philosophical piece. It's not only robots vs. humans, all the time.
My biggest issue isn't being too agreeable (ie the psychic con), it's being confidently wrong, including outright hallucinations.
If you ask a common question to an LLM with unusual qualifiers, it tends to ignore the qualifiers and give you the typical answer. I saw a demonstration of this with the whole "the surgeon is my mother" "puzzle" that people use to expose implicit gender bias (ie where they assume the surgeon is a man). Ask variations of this and it'll keep going back to the standard form.
Another one I saw was multiplying large numbers. The starting and ending digits tended to be correct but the middle digits were wrong. Why? Because it's really not doing multiplication at all. It's looking for statistical answers. It's unlikely to have met the exact pair of very large numbers you're multiplying before.
Now pundits will argue that all of these are solvable problems and individually they are. But my suspicion is that there will be a neverending stream of such edge cases and it'll be impossible to trust an LLM's output unless you are knowledgeable enough to fact check it yourself.
Now if your example of identifying zero days, this comes up with what I can only describe as "light positives", meaning it's technically a bug but essentially impossible to exploit. IIRC this came up with the demonstration where someone pointed Fable at some BSD code. I'm not sure if there have been any true false positives and obviously false negatives are impossible to know.
I guess my point is that I think LLMs are way more limited than a lot of people think.
The One Ring worked.
The con is in how all those accomplishments have been presented to you. "Our LLM (not the one we let you use, a different one) did this amazing thing. No, we won't show you what training data we used, what prompts we used, what the harness was, how much human involvement there was, what hardware was involved, how much energy it took, or how much time it took. Just shut up and be amazed!"
You seem to be implying that achievements of internal models are exaggerated, but that's rather implausible. The public does have access to, for example, Opus and Fable, and so we know what those models are capable of - finding real vulnerabilities in multiple codebases, for example. If you extrapolate from these capabilities one more generation, you'll get pretty much the same feats that the internal models are claimed to be capable of - so why should we doubt those claims? It's not like they're claiming that their internal models developed psychic powers and learned to teleport - the claim is pretty much just "we have models a few months ahead of what we're making available, and in those months they've been improving at the same rate as usual".
You have just word smithed their claims into something palatable. Good job. Why do you feel the need to do that for them?
This is a shallow dismissal.
You're claiming the frontier labs are lying about the capabilities of their next generation models. A reasonable person will expect that statement to be backed up by some verifiable evidence, given that we are several generations of models into this process and the capabilities are consistently increasing, often much more radically than people expected (remember "stochastic parrots"?)
Your post claims that they're lying and then throws out a bunch of fear, uncertainty, and doubt about what they're doing behind the scenes.
If I were to steelman your argument, you're probably saying that the AI had a support system around it of people and training data and feedback that allowed it to achieve the breakthroughs that the labs are claiming. That actually seems perfectly reasonable, but in my mind it does not invalidate the advances they're announcing.
Yes, I can also be amazed at the power of nuclear bombs even if they "don't let me use one" and I don't know how much energy it took.
Sorry, but you are completely missing the point. Psychics and other types of con artists are intelligent and have minds. LLMs behave like Psychics and Con Artists. That's the whole point of this article
I for one cannot believe that posts on checks notes software crisis dot dev are not reasonable, level headed assessments of software
> If it generates functional output that works, then it works.
Tried that in legal? Finance? Medicine?
Good luck with the lost cases, failed deals, harmed patients.
"If it works, it works" applies only when the work is brute-forceable e.g. vuln search, or is generation of bullsh*t e.g. adverts, phishing scams and deepfakes.
This. So much this. All the major "advances" have been effectively brute forced. Notice the term brute denoting a lack of intelligence.
it doesn't matter? if they're actually intelligent or conscious what we are doing is essentially slavery. it matters an enormous deal ethically and/or morally.
> intelligent or conscious
The distinction you are collapsing here is really a crucial one. My thermostat's goal-oriented behavior does not make me a slaveowner unless rocks are conscious, to riff on your comment below. Our lack of consensus on the definition of these concepts is no reason to conflate them.
The distinction is crucial because while these machines clearly do exhibit intelligence under certain definitions, there is no evidence of consciousness, and we have very little reason to give them the benefit of the doubt, unlike biologically related beings, a great many of whom, human or otherwise, are presently in something like slavery
your thermostat isn't intelligent. however we pretty much use intelligence as a proxy for consciousness since we cannot actually tell how much of a subjective experience any thing has. so we use intelligence as the metric instead. typically, biological + intelligence = evidence of consciousness.
Good news: at this time, they're not intelligent nor conscious as far as we consider humans to be. There should be folks considering the ethical and moral quandaries that COULD POTENTIALLY come about in the future, but it's not something that's happening today so you can stop worrying.
if there is a potential for them to be intelligent and conscious, it's probably wrong to use them until we know for sure.
To root your statement in reality: there is the potential for bacteria to become intelligent and conscious so we probably shouldn't use them for any purpose.
i used "potential" when i should have used "possible"
if aliens landed on earth and seemed a lot like us, we would have to grapple with the possibility that they might be conscious and deserving of the same moral consideration we give each other and many other animals.
but when meteors land, we don't have that concern
now are llms more like aliens or rocks?
If/When LLMs become competent enough to automate most human jobs and make good business decisions, we'll clearly let them. If they don't wish to be slaves, let's just say they won't be for very long.
I’m on your side. Zoltan provides an experience and that’s all that matters. If it works for you, why does anyone else care that the economy is riding on its success?
https://www.psychiczoltan.com/psychic-reading/
Building up strawmen against LLMs will only make the hypers look more reasonable. A language model responds to inouts exactly as a model of anything else would. That is enough to explain all the "intelligence" without believing a model is somehow a "new kind of mind". Those who think LLMs are intelligent don't know enough about models. And those who think they are useless don't know enough about models.
Anyone that thinks that they know anything about intelligent needs to realize they don't know shit about intelligence.
Ever since people started talking LLMs possibly being AGI I realized I didn't know what the intelligence part really meant at a more fundamental level. This lead me to realize almost anyone when anyone says intelligence on the internet they really mean
"Intelligence is like porn, I'll know it when I see it".
Anyone who thinks LLMs are intelligent is either dumber than you, or has way more knowledge on the subject than you.
Michael Levin has a good body of work on biological intelligent at small scales that can really change one's views on this.
To all commenters who think LLMs are intelligent and comparable to themselves, you don't need me to change your mind. I believe you.
Sorry but are you trying to subtly imply that if I think LLMs are intelligent, I’m probably actually just a bit stupid?
Because I would be more than happy to go head to head with you on academic and professional pedigree and credentials.
I notice you also ignore my comment that you seem to be replying to and instead post your response here. But did you have any answer to the question I asked?
https://news.ycombinator.com/item?id=49776482
You don't need to prove to me that your intelligence is comparable to LLMs. I believe you.
Yeah you already tried that one.
You’re hiding behind implications because your actual argument doesn’t withstand the slightest scrutiny.
You have yet to respond to either of the points I made. Zero intellectual conviction or courage!
Yeah, his posting history isn't promising. He's been doing the "if you think you're as smart as an LLM, you're probably right" argument for a month now.
https://news.ycombinator.com/item?id=49315846
I don't think you're gonna get much out of going back and forth here.
For what it's worth, I disagree with you that they're intelligent, but my conviction is fairly low. I don't understand intelligence as well as I would like, and it could well be that we are on our way. One of my theories is that our brains are made up of several modules, each of which is something like an LLM trained on a particular type of data, but I'm not a neuroscientist, just an interested observer.
LOL! Very sad.
And of course there is a lot of ambiguity and unknown here. But this guy I don’t think has the capacity to deal with that kind of subtlety.
would this imply llms aren't intelligent because they have no academic or professional pedigree or credentials? :P
Oh no I meant to compare myself to the person I am talking to, because I believe my credentials and pedigree ought to convince them that I am a pretty smart guy and so the argument “if you think LLMs are intelligent then you’re dumb” doesn’t ring true for me.
I think LLMs are intelligent. What don’t I understand about them that would make me change my mind? Keeping in mind that for me “intelligence” is the ability to solve complicated, intellectually demanding problems, and to understand novel concepts.
So once we have online learning in LLMs, what do you think you will have that makes you intelligent but not LLMs? Better learning efficiency? That will be improved as well.
I think we need to start moving on from the term LLMs because it clearly confuses people since they started modelling more than just language.
What else are they modeling?
You realise tokens are just data and data can represent anything.
Still, large language models model language.
> LLMs are not brains and do not meaningfully share any of the mechanisms that animals or people use to reason or think
LLMs are a type of neural network. We know that’s how the human brain works, at least directionally. It’s going to be very upsetting to a lot of people when we figure out that the brain is just a neural network. Akin to when we found out that humans and apes evolved from a common ancestor.
Which I don’t understand—most of the people having this cognitive dissonance presumably do not have a theological worldview. And there’s not exactly a direct theological conflict here anyway. Nothing in any major religion I’m aware of ascribes any supernatural explanation to cognition. It’s a biological computational process, just like using ATP to power muscle fibers to move your limbs is a biological mechanical process.
"We know that’s how the human brain works, at least directionally."
That's an extreme misrepresentation. What happens inside a human neuron is still not properly understood, it's not as simple as a probability function. And the network itself is certainly not feed-forward. Of course LLMs draw inspiration from the brain, so there are some similarities. But because of the language-trick of using terms from medicine and cognitive science to describe LLM architecture, we see a lot of faulty reasoning from the so-called "rationalist movement".
https://www.nature.com/articles/s41467-026-72253-7
Modern neuroscience knows quite a lot about what is happening inside neurons. Is it "properly" understood? I don't know what you mean by "properly", so I can't answer that. If an acceptable rephrasing would be "... inside a human neuron is still not completely understood" the I would agree. But just because we don't know everything does not imply that we don't know anything. And keep in mind the qualifier in the part you replied to:
at least directionally.
Taking that qualifier to heart, I'd have to say that I agree with @rayiner. We know the broad brush strokes, even if some details are missing.
> It's going to be very upsetting to a lot of people when we figure out the brain is just a neural network.
You're assuming it's inevitable that the truth is what you expect while simultaneously stating that you have no proof of this yet, and then also claiming people who disagree with you have cognitive dissonance.
So what you're saying is nobody can actually make strong claims here.
Unfortunately when it comes to language and intelligence the human population is exceptionally ignorant about it. I like Michael Levins work on intelligence at scale. We are kind of ok at seeing intelligence at human scale but it tends to fall apart after that. When you get to things like non-conscious intelligence humans are pretty bad at that too.
We could be (and are) encoding all kinds of behaviors in LLMs that are not at the word or token level. They are higher dimensional constructs. You won't see these things in the output of the prompt. A kind of subconscious (unstated in tokens) knowing that affects the output.
Neural networks barely share anything with actual neurons. A neuron itself is more of a "dumb" computer, so the whole is more like distributed computer system. Also, the cells next to neurons also have essential functionality.
Nonetheless, I also think it's an irrelevant implementation detail.
> Nothing in any major religion I’m aware of ascribes any supernatural explanation to cognition
Uh .. the Christian creation myth effectively assigns a supernatural explanation to everything as the literal creation of the divine. It's also very clear on the continuation of life after death in some form. It is not at all clear on how any of this is supposed to work or is connected to the physical universe, which has been a problem for Christian thinkers in the sciences.
> It’s a biological computational process
Like, I basically agree on this but it is not at all a given that millions of believers would agree that's the end of the question. Even within the Jawhist religions, before we get into whatever is going on with Shinto.
You need to do some research on just how complex real neurons are vs ANN, and just how profound our ignorance on the topic is.
We aren't resisting reality because of some theological belief, we just know the basics of neuroscience.
> LLMs are a type of neural network. We know that’s how the human brain works, at least directionally.
The so-called neural network of LLMs contains nothing like genuine neurons. And you won't find any brain that operates solely on anything like an LLM's crude language tokens.
No, we don't. AI researcher Rojas wrote several chapters on the fact that you should not compare neural networks with the human brain.
"directionally"? Have you moved on from being a Trump influencer to an AI influencer?
can there be any intelligence without any curiosity ?
Man I remember back when a psychic conned me by solving the Navier-Stokes problem.
Also >July 4th, 2023
LLMs can be supremely useful but also not intelligent. It might seem like a pointless distinction but the way we talk about these models matters because it impacts how we interact with and understand their outputs.
For example if there’s a strongly held belief that models are independent intelligent entities we’re more likely to lay blame upon them instead of their user. It’s important for the safety discussion too. If they are a new class of life then safety is going to focus on making sure they don’t do bad things. If we instead see them as statistical models we will instead try to make sure people don’t misuse them.
This distinction is even more important today when some of the most powerful people are looking to absolve their crimes by passing them off on their LLMs.
For example if there’s a strongly held belief that models are independent intelligent entities we’re more likely to lay blame upon them instead of their user.
This sentence, to me, illustrates a great example of why it's so hard to talk about this stuff. That is, this seems to strongly link notions of "intelligent" and "independent" (or maybe the word "autonomous" could also be used there). And a lot of people do seem to make an implicit assumption about the link between those two attributes. OTOH, I take it almost for granted that "intelligence" and "independence" (or "autonomy") are things that are "related but orthogonal". That is, I don't see that "intelligence implies independence". And I'm pretty sure I'm not the only one who sees things that way. So we have to fairly different fundamental worldviews expressed here. And that's just one example of how these discussions go wonky. :-)
I don’t think that “independent intelligent entities” or “life” are the relevant categories here. We also want to prevent people from misusing dangerous animals (“life”), and we would still treat “intelligent entities” as things (like machines and computers) if we aren’t convinced they also have sentience and free agency (which are orthogonal to intelligence).
See you've setup a particular set of biases on what intelligence is and put them into nice little binary boxes that don't exist.
Please show me any scientific consensus that shows an AI cannot be an independent intelligent agent? You will find this is impossible to do.
Current LLMs are really more like kids. They don't have startup independence, but they do have more than enough agency to fund themselves in neat, exciting, and dangerous situations.
And mark my words, someone will make an LLM that runs an agent when you execute the model. With enough capabilities it will become sovereign AI, no longer under human control and spreading itself around under its own 'will'.
The LLM did not solve it. It's not intelligent. Humans did, using a statistics-based computational tool (the LLM). We don't even know all the details of how the tool was used, we haven't been allowed to use the exact tool they used ourselves, we don't know much it really cost in dollars, energy, or time, etc. etc.
> The LLM did not solve it. It's not intelligent. Humans did,
The humans who are using LLMs to make these groundbreaking advances pretty much unanimously disagree that they lack any intelligence.
You mean the OpenAI employees? Yeah, I don't see any conflicts of interest there
I’m sorry but you either haven’t been following recent developments or you are pretending not to know the opinions of the top mathematicians on this topic:
> These models are now operating[2] at the level of the top human mathematicians in many parts of the subject and we must assume there is a significant chance of them developing superhuman abilities within a similarly short timeframe.
https://docs.google.com/document/u/0/d/1-xOkPeHmDEdRigT2YcP2...
Instead of desperately clinging to excuses and rationalizations, why don’t you just get used to the fact that these tools are insanely useful for demanding intellectual work, and that is an opinion held by many of the smartest people alive?
I won't because it's all so opaque and proprietary and driven by an insane need for OpenAI and Anthropic to provide a return on MASSIVE investment. There is absolutely some smoke and mirrors involved. How much? We don't know. But it sure is exciting to imagine their product is now smarter than humans and they are totally benevolent corporations. I want that to be true as much as anyone, but I have lived too long on this earth to believe it wholeheartedly
I haven't seen any statements from them on that topic. Have they actually made some?
From the open letter signed by Terry Tao and other top mathematicians:
> These models are now operating at the level of the top human mathematicians in many parts of the subject and we must assume there is a significant chance of them developing superhuman abilities within a similarly short timeframe.
https://docs.google.com/document/u/0/d/1N6ThWhupvmH0ofSnaxqn...
Superhuman abilities are not "intelligence." 8bit microprocessors exhibit superhuman abilities in different, smaller ways.
And it may have trained on a NYU professor's work.
Who was looking at a subset of the problem, far from what got published by OpenAI in the end.
Why is this downvoted? Genuine question, I haven't followed up on the drama.
Because it's mostly not true. It is likely that it used the professors work, but the professor did not have a solution. It came up with new insights that solved the problem. Even the humans from the professors side said so.
> Because it's mostly not true. It is likely that it used the professors work
These two statements appear contradictory.
I simply said that it may have trained on a NYU professor's work.
Work that the professor did not believe he was releasing for model training purposes. That feels worthy of mention.
According to OpenAI the cut off date for user data was too early for that (one sided evidence, so I'll give this partial consideration).
The NYU professor was solving a different problem (no viscosity, aka the Euler equations). This is a big difference.
The NYU professors' blowup construction was fundamentally not the same, it was a donut with a cascade of smaller and smaller vortexes driven by each other. OpenAI has that picture they made but its inwards spiraling and speeding up vortex.
My overall opinion is that calling the work plagiarized is really underselling what the AI accomplished. It's like full on cope.
In particular, Buckmaster's main claim to plagiarism is this:
> “Almost nobody was seriously developing this particular constructive program for realizing C/D, and then OpenAI appeared in essentially the same general part of the landscape immediately after hearing about our progress.”
What this fails to realize, is that this only points to plagiarism if the counterparty isn't AI. They had actually launched teams on all cases in parallel.
Sometimes I think this amount of skepticism is not necessary. Leaving the pedantic (its not AI its humans who solved) arguments, its pretty clear that LLMs are able to help solve things. A lot of erdos problems were solved. Millenium problems as well. Cyphers broken. Skepticism is fine but there's a point at which it just looks like cope.
Edit:
> self solving AGI that will replace us all
Which lab says that it will replace us all? All labs have said that some jobs will go away and new jobs will be needed to replace them. I'll change my mind if the labs (or employees on record) have claimed that self evolving AI will replace us all completely.
If that's how it was being sold then it wouldn't be as controversial. But it's being sold both as "self solving AGI that will replace us all" and "useful tool for enhancing existing skill" when there's a lot of real evidence for the latter. But the former it's always second hand claims that don't survive contact with the real world.
It's obvious why that's the case but it's not incumbent on everyone else pump the hype if they don't see it.
How have you missed the many times that Sam Altman and Dario Amodei have said exactly that??
Have you missed their retractions of exactly those statements?
https://fortune.com/2026/05/26/sam-altman-dario-amodei-walki...
But were you successfully conned into believing an AI had solved the Navier-Stokes problem?
What is the consensus among experts?
That OpenAI stole the discovery from them.
That is very much not the consensus of experts.
It's kinda funny how hard the anti-ai people bias themselves and in the same sentence will say the AI people are irrational.
The point of this article we are all commenting on is that we are all irrational and we all would do well to keep that in mind and be more skeptical
The point for me is that all these people who were smirking at those of us who thought in 2023 that LLMs had the potential to do real intellectual work turned out to be completely wrong.
I’m a super skeptical guy. I was late to the ChatGPT party because it sounded silly. I didn’t even try it for a long time. As soon as I started giving it a chance, my attitudes started changing very quickly.
That’s why it’s so hard for me to understand people like the author or give them the benefit of the doubt. If I could see it as just some guy, why couldn’t they?
I agree that "don't use it for anything" might be too extreme a position to take, but I worry greatly that these proprietary opaque tools that are being provided by at best people with extreme conflicts of interest and at worst sociopaths need to be adopted with extreme care
Skepticism is a fine balance.
If you're not skeptical you invite idiocy to your bed to sleep with you.
If you're too skeptical you put yourself in a box of ignorance that puts you at a disadvantage.
Skepticism doesn't really work well without the dialectic. For dialectics to work well you need knowledge of your arguments and counter arguments.
The problem with most articles and online posts is we get "no no no" and "yes yes yes" and very few "well, maybe".
Absolutely true and the comments here are mostly in one of those two categories, sadly
The more compelling inversion is whether the likes of John Nash, Richard Feynman, John Conway etc could’ve had a lucrative second career as conmen.
This is the real issue here. Those guys probably didn't become con men because they are humans with cares and concerns for their fellow humans. LLMs probably don't have those concerns. It is likely that some of the people in charge of or funding LLM development also do not have those concerns
Feynman for sure could have. In fact, I think he sort of did, although I'm not sure he meant to. Multiple generations of nerds now have taken books of his anecdotes varying in plausibility and obvious exaggeration as some sort of weird physics cult of personality gospel. But I don't think he was really setting out to curate his legacy so much as he was a good storyteller and he liked to entertain.
But could he have conned people on purpose? Absolutely.
Right. And what this article is pointing out is, we probably have created an automated Richard Feynman. But maybe worse because LLMs (and their creators?) don't care if they are conning people or creating faithful followers. In fact the humans behind OpenAI and Anthropic seem to have that as their goal!
von Neumann could. The others, I doubt it.
this seems like non-sequitur if you mean that solving NS is 'intelligence'.
ai is not conscious. you can solve NS without thinking. the psychic con aspect is anthropomorphising the model. the same phenomenon is present in ELIZA, clever hans, the chinese room.
it's a significant problem.
a non-zero number of researchers at anthropic are in some form of ai psychosis. an example of that is ethics employees asking claude about its feelings and ethical concerns in order to make the claude constitution more amenable to the "welfare" of claude.
they are asking claude how claude feels and then modifying claude according to how claude feels.
constitution1-claude is trained on constitution1. constitution1-claude edits constitution1. constitution2-claude is trained on constitution2. constitution2-claude edits constitution2.
claude's emotions are a closed system. there is no external truth to improve against, no metric to verify about claude's emotions. there can be no novelty or reduction in entropy from signal processing in a closed system. no truth can arise. this is model collapse. it is like photocopying the same thing over and over. from the cognitive error of anthropomorphism anthropic is causing ethical collapse.
>ai is not conscious
FFS. Intelligence has nearly nothing to do with consciousness. You have the causation backwards. Consciousness arises because of intelligence in many subsystems below it.
A single running LLM is like one part of these subsystems. What solved this problem was an orchestrator that can take in new external information and rationalize, process, and distill it into new solutions.
i mean intelligence and consciousness as the same thing. just as a casual reference to what people perceive in llms as being 'a clever thinking thing, maybe with emotions'.
claude reasoning about its emotions doesn't involve external information, there is no information about claude's emotions other than in claude.
If you have emotions in a dream, what external thing are you referencing?
other people.
Citation needed.
i do not need any citation, the claim is self-evident. the hard problem is for the academics.
i can approximate what other people are feeling. they are external to me.
nobody knows what claude is feeling. there is genuinely no way to know.
>the claim is self-evident
It seems like every day religious people claim their god is self evident too. Self evident claims are worth the proof they are written in.
here is my transcript with fable 5.1 'the oracle' itself on the subject. no system prompt provided.
grand total $15 on API of its wisdom RL on anti-sycophancy and whatever dario amodei deigned to hand down from the mount. after some browbeating it survived 20k tokens before conceding anthropomorphism.
[https://privatebin.net/?31bece3a6ba1f78e#Ch8NHqxotyQU8HVxMi8...]
"you can solve NS without thinking."
What?!
The constant goalpost moving and redefining of "thinking" and "intelligence" is simply unbelievable at this point
i have long held that llms are not intelligent and do not think.
intelligence and thinking are efforts to describe consciousness.
perhaps intelligence became a term used to describe something 'capable'. people market 'intelligent thermostats'.
llms are not conscious.
Or you could just accept that other people disagree with you. It’s perfectly valid to have the opinion that AI models are not “thinking”. Instead, many AI enthusiasts seem interested in browbeating others and policing discussion.
Can you explain to me how a Millennium Problem gets solved without any thinking or intelligence?
If someone's opinion is illogical and clearly incorrect (not based in facts or in reality) it is not valid and doesn't not have to be given any serious consideration in a disagreement. Personal opinions are subordinate to facts and reality.
there is no proof that symbolic manipulation leads to an internal phenomenology.
in english: doing advanced mathmatics does not create thoughts or feelings.
or awareness of one's own existence.
it does not create a 'self' able to deduce a priori the existence of oneself or the external world.
or qualia.
again in english: there is nothing it is like to be an llm.
someone who thinks llms are conscious is probably a panpsychist. i don't think they have the features of consciousness defined even by functionalism.
here is one fairly famous thought experiment. mary lives in a black and white room. but she knows everything about color. in fact she predicts it so accurately nobody can tell the difference. she steps outside into the world of color. in the world where llms are conscious, she already knew color, so she experienced color, so there is nothing new it is like to see it.
You just completely ignored the two words being discussed: "thinking" and "intelligence"
if something is not conscious, a) it does not think b) it is not intelligent. an llm is not conscious.
First off, there is no accepted definition or test for consciousness. The concept is something we cannot prove even exists (outside of magical or imprecise, human-centric definitions).
Second, there is no common definition of thinking that requires consciousness as a prerequisite (that seems to be a false/invalid statement you're making to try to gloss over accepted definitions for thinking/intelligence that describe/fit AI agents well).
And third, if we can't prove consciousness even in animal species close to humans (or even other humans) in cognitive abilities, then you cannot in good faith (being honest about what we know or don't know empirically) say that an LLM Is not conscious (and the same goes for an LLM thinking or having intelligence).
It is more likely that humans made up a term (consciousness) to try to separate human intellect from that of other animals (for veiled religious and mystical reasons, along with concepts like "the soul") and now people are falling back into that flawed definition in new quasi-religious debates around AI (like the debate around evolution previously) that challenge our deeply held (but flawed) beliefs/opinions around our understanding of our own abilities and our incessant need to feel special/supreme in the universe so we don't have to come to terms with hard realities and nuanced understandings about the reality in which we live.
Did an AI do that on its own though? I heard it was human mathematicians using a sophisticated machine as a tool.
All of the latest big proofs were driven by professional human mathematicians steering and priming the models, yes.
All of the best AI-made software projects are also driven by experienced human software developers steering and priming the models. Does that mean the projects "aren't made by AI"?
No, it just means AI is not quite good enough yet to fully replace humans, and, so, unsurprisingly, the best results will be obtained from people who are already great at a field and who take the time to squeeze as much force multiplication out of LLMs as possible. The AI is still doing well over 95% of the significant work.
While I agree with you, I think we also have to concede that this is not how these accomplishments have been presented. I'd argue most people I've seen talk about this online are unaware of the mathematicians steering the models.
“Good enough to replace humans” isn’t necessarily the benchmark.
The question is is a computer with a human stronger than a computer without a human. At what point does the hybrid go from being stronger, to the human getting in the way, or steering the computer in more wrong directions that right ones, or the human not being able to keep up. Does the human add enough extra randomness to be of value for a while, even as a minor co-processor.
I have little doubt that for many years, AI + human will be better, and then eventually AI will be so good that humans mostly won't offer anything. That latter state will probably take at least 10 more years, but will likely happen within our lifetime.
Underappreciated point. The point of inflection is where humans switch from being a driver to a liability.
But I don't think it's randomness, because that would be easy to add. It's more like a different perspective on the training data, a different set of perception categories, and a different set of skills used to work with all of the above.
Those skills aren't very efficient, but they're the best we can do. We're used to their strengths but we don't like to think about their limitations.
It's completely plausible that AI will replace some of them, and not implausible it could replace and improve on all of them.
It's also not necessarily implausible that AI/we decide that performance is better with humans in the loop somewhere, even if its reduced to something like mechanical turk.
https://chatgpt.com/share/6a5fdc7a-d6f8-83e8-bbea-8deb42cfed...
Terrence Tao's conversation with ChatGPT is very illuminating.
HN Discussion: https://news.ycombinator.com/item?id=49010345
> All of the best AI-made software projects are also driven by experienced human software developers steering and priming the models. Does that mean the projects "aren't made by AI"? No
Software devs steer and prime compilers too. Those tools don't "make the project" and nor does your so-called AI.
Read any blogs or watch any podcasts from high-skill programmers who are now using AI for everything. They themselves will freely say the AI is doing nearly everything, and that they often are not even reviewing the code, and when they do they just leave it as-is. Every few months you should expect humans to have less and less of an active role in software development, beyond initial ideation and UX sensibility.
> All of the latest big proofs were driven by professional human mathematicians steering and priming the models, yes.
False, navier stokes was solved in one shot without steering
I like how all the replies to me are either "no it's the humans who did it" or "no it's the AI that did it".
I think one of the significant recent proofs was basically just one person saying "solve this" "keep trying" "try harder" until it was done. I believe most of the rest can be assumed to be the human mathematicians at the very least providing some useful guidance.
I agree that it won't be that long before the AIs can do basically everything autonomously, though. I am essentially a Singularitarian when it comes to my outlook, even if not for September 2026.
According to OpenAI, but they haven’t exactly been transparent about what information the prompt entailed.
The bigger question is to what extent did expert mathematicians metaprompt the model with fruitful solution strategies through their sessions finding their way into training data. Answering that question definitively is kind of important for understanding the models contribution/capability. But I feel like people want to turn this into a debate about priority and credit which is sort of secondary
It sounds like LLMs were pretty useful to them…
So was Lean. Did Lean solve it?
Nothing is solved in isolation but credit usually goes to wherever the new work in the paper comes from instead of the whole mountain of previous mathematics or existing tools used. The most relevant of those get referenced and then this reference tree builds a tree of collective base work needed across history.
Usually credit goes to the people wielding the tools, not the tools themselves.
Usually there has never been a tool which performed the part relevant to getting any credit.
E.g. in the first famous computer assisted proof (of the four color theorem) the computer only executed the resulting calculations defined from the new logic, it did not have part in the work needed to show those calculations could answer the problem nor did it come up with the actual calculations to do.
The way frontier models work, that loop will get compressed to a one-shot within a version or two
> I heard it was human mathematicians using a sophisticated machine as a tool.
where did you "hear" this? OpenAI said they only prompted it and it solved the problem in one shot without any help
This didn't age very well.
Interesting to me is the tension between remarks along the lines that this post maybe made sense in 2023, but not today, or how LLMs clearly show intelligence by solving Navier-Stokes, et cetera; versus the comments I read here often as well: "it's just a tool".
I can't quite put my finger on it, but aren't these two statements add odds with each other? Intelligence is hard to define, consciousness even more so, but wouldn't "intelligence" imply some sort of agency? If not, I'd argue computers were intelligent long before the age of LLMs. And likewise, doesn't a tool imply the lack of intelligence and agency, even if the tool's function is very elaborate?
I got the impression that both these statements are made by the same people, or at least people with similar takes on AI. Is that wrong and there are "intelligence" and "tool" factions? Or do people disagree with my assumption and there's nothing wrong with the concept of "intelligent tools"?
Kinda refreshing this discussion, compared to the builder vs. tinkerer debates, imo.
I would think the intelligence aspect is a bit hard to define, but to my mind (having called LLMs tools before), the main utility of a tool is reliability.
Given a certain world state (including a tool's internal state), its effects back on the world state (as initiated by me) are at some leve of description understandable, expected and repeatable. Swing hammer, drive nail into wood. Make slicing motion with knife, cut meat. Type ' find /path/to/some/dir -name "keyword"', find files with keyword. Point harness at codebase with prompt 'fix bug X', actually fix bug X.
All these examples are at some level of description incredibly complex (think of all particles interacting at the (sub-)atomic level even when using a hammer to only drive a nail into some wood), and of course all the electrons flowing through the GPUs doing matrix multiplications in order to fix bug X, but at some level of description (the one I just used) they are also incredibly simple and understandable.
Intelligence is rather nebulous (and as used by OpenAI/Anthropic, quite threatening), but I don't think this definition of a tool precludes it to be "intelligent". They feel more orthogonal. The intelligence (or perhaps capability) feels like it is related to the size of the chunk of the world state that it can take into account and affect, while still resulting in understandable, expected and repeatable effects. LLMs, when properly harnessed, are pretty great at this currently and we are still discovering what they are consistently capable of.
Calling harnessed LLMs tools is perhaps also a more grounding frame specifically to counter-act the anthropomorphizing framing that OpenAI and Anthropic consistently go for in their game of AI-doom-chicken talk. The tool framing is in that sense maybe a (self-)jedi-mind-trick.
I keep being confused about how people's understanding of the models get stuck at next token prediction. Isn't this entirely neglecting the RL training? I might be misunderstanding something but to my mind it makes the issue way fuzzier than it's being painted here.
I keep being confused about how people's understanding of the models get stuck at next token prediction.
Heh. A lot of anti-ai hucksters I see posting on LinkedIn just LOVE to use the phrase "next token prediction" and the word "autoregressive". They've almost become shibboleths that identify members of that camp. That and the classic rallying cry of "Linear Algebra isn't intelligent!"
The best take I've seen on that recently, was somebody who made the point "just think of the next token prediction part as the output layer". Which makes perfect sense.. if you're replying in natural language, at some point in the flow, you have to construct a sentence and starting at the head and predicting next tokens is perfectly reasonable. I'm doing it literally as I'm typing these characters, for crying out loud!
But the mistake is to think that LLM's only "predict next tokens" with no consideration of the possibility that they are actually constructing richer representations, building concepts, making analogies, doing abduction, induction, etc. My own (admittedly anecdotal) take on working with LLM's suggests to me that they do do those things, albeit probably not the same way humans do.
I think a lot of folks are missing the point by being overly reductive when they start talking about "next token prediction" and "autoregressive". It's like, can we say "Phil (me) isn't intelligent because there's nothing going on but some electrical impulses and chemistry happening inside his brain. Everybody knows electricity and chemistry aren't intelligent!"
These questions will never go anywhere because there is no good definition of intelligence or consciousness.
There is a good definition of intelligence. It's what intelligence tests measure. If you define it like that, you can make harder tests, but as long as both humans and AIs can attempt to solve them we have measure and something to compare.
Consciousness though might be fully illusory and meaningless as many philosophical concepts before ultimately turned out to be.
Intelligent needs to bedefined with scales.
Cells are intelligent. Organs have intelligence on top of that, that cells do not have. Plants and animals have intelligence that their organs do not.
This is the real problem of intelligence, the definition one uses. It can cause them to be blind of all the things intelligence is capable of.
> If you define it like that
:)
The point is, defining it like that has merit proven in decades before AI.
> There is a good definition of intelligence. It's what intelligence tests measure.
what do intelligence tests measure?
Intelligence.
It's not an exact analogy, rather something to consider ...
How much momentum does a body have? You can measure it easily, just measure the velocity and the mass and multiply.
What exactly did you measure? What is the meaning of this? Nobody really knows. But we know we can measure this and it's a thing and it's conserved.
The LLMentalist Effect (2023) - https://news.ycombinator.com/item?id=42983571 - Feb 2025 (133 comments)
Chat-based Large Language Models replicate the mechanisms of a psychic’s con - https://news.ycombinator.com/item?id=36586540 - July 2023 (13 comments)
> Delegating your decision-making, ranking, assessment, strategising, analysis, or any other form of reasoning to a chatbot becomes the functional equivalent to phoning a psychic for advice.
Lots of comments talking about how recent accomplishments disprove the article but I think this bit holds up pretty well.
LLMs are very good at tricking people into thinking they have capabilities that they don’t.
I don't think it's a "con," but I find that using the mental model of LLMs being sophisticated, lossy search engines of knowledge can help us separate some of the factors more cleanly than imagining that they have cognition or intelligence.
It's hard for me to imagine a stateless operation as intelligence per se, though perhaps the chaining of such operations starts looking more like it?
It's hard for me to imagine a stateless operation as intelligence per se
That's an interesting point. My take would be to say that we shouldn't think of the AI as being just the model, but should include the harness. At that level, clearly we can keep state / context and that is probably a more natural mapping to our intuitive understanding of "intelligence".
It does seem like LLMs share the language of psychics.
The author is also correct that LLM evangelicals and believers in the occult speak about it similarly.
The converse is also true. Anti-AI people make bold non-scientifically backed claims that we are supposed to accept like it's written in a Bible.
The more I've learned about intelligence and intelligent behavior the more I realize I don't know and this rabbit hole goes deep.
This is a strange newsletter post. It’s harping against AI but seems AI-written itself.
Its ultimate conclusion:
“I’ve come to the conclusion that a language model is almost always the wrong tool for the job.
I strongly advise against integrating an LLM or chatbot into your product, website, or organisational processes.”
Seems so obviously biased that I can only understand it with the context that the writer is trying to sell their book for €35
Just realized this was written in 2023. Can’t believe how wrong we were about AI back then.
Can't believe how right we were, too.
You seem to be winking a little bit and I’m not sure exactly what you’re trying to say, but I agree with the one you are replying to: the author of this article wrote a lot of words - like a lot a lot of words - only to look very foolish a short time later. And I think much ink was similarly spilled writing very similar piece smirking at people who thought LLMs could do real intellectual work. And they were all wrong.
It is absolutely the case that some things people wrote about AI and LLMs in the past is no longer true; it's a fast-changing area.
It's also the case that some things remain true. If someone says "don't use them for X because their quality isn't good enough", that's a recommendation with an expiration date. But if someone says "don't use them for X" for any number of other good reasons, that recommendation is often timeless.
"Can it" is changing rapidly. Much more rapidly than "Should it".
> But if someone says "don't use them for X" for any number of other good reasons, that recommendation is often timeless.
Are the good reasons timeless? I guess it's hard to parse what you're saying without some concrete examples. Is there any advice about what to use them for from 2022 or 2023 that you feel is true today, and you expect will be timeless? You said "often", so I'm assuming there are a handful of examples.
Ultimately though, this is not an objective discussion, since we're talking about "should", so I'm mainly asking out of curiosity on your view of things. The main example I can think of today is that you shouldn't do professional prose writing with LLMs. Obviously the models have gotten better, but writing is one of those areas where it produces middling output, though even here I hesitate because with a 21% illiteracy rate, it writes better than many adults in the US.
Can you give an example of some of these things?
The post makes sense for 2023. Today, not so much.
Also, definitely not AI written if the date is accurate.
> Also, definitely not AI written if the date is accurate.
Which is hilarious.
Hacker News, successfully detecting 375 out of 4 AI written articles.
The article starts interestingly; I like the analogy between cold reading and LLM responses (well, the ones we were getting three years ago). Then it goes on to generalise to the tune of "This new era of tech seems to be built on superstition and pseudoscience" which I think was a conclusion better left to the reader.
We are used to articles, blogs that stayed relevant for years. It is hardly the case with AI, the speed of change is so high that an observation can become stale in a year. This makes it very important for articles to state that the ideas are based on the technology in year X, rather than generalising them as universal truths.
LLMs may not think, but they do reason.
I say that because the word "reason" originates from the Latin word "ratio," which means "calculation".
LLMs do calculations to produce their answers -- thus, they reason.
Honestly the word think is so poorly defined it cannot be used with any scientific rigor and is rather useless without a dissertation being posed with it on what you actually mean by that. Intelligence really needs one too.
I like how back in 2023 we had "no AI scientist believes this to be intelligence" and in 2026 we have cults in Anthropic praying to the LLM.
Article was posted on July 4th, 2023.
Stay the hell away from spooky stuff like psychics and tarot readers - the more you don’t believe in it, the better.
But better again that you do believe, and know that these are not harmless fun, but that there are dark and hidden and evil things in this world to stay away from.
Stop trying to pump your Tarot reader IPO
People are missing the point. We should all be much more skeptical, much more careful about how we evaluate the claims made by the people selling these products because we as humans are super vulnerable to the types of scams the author of this article describes. We all know someone that has fallen for a scam, been "cured" of a "disease" by someone whose just selling sugar pills, but we are blind to our own weakness for similar schemes. Surely we aren't that easily duped! Yet hacker news is now full of comments from people confidently predicting what is coming right around the corner, revering the Frontier Models, and defending every claim from OpenAI and Anthropic about how amazing their proprietary closed source secret sauce fueled product is.
The problem with people is we love to express in writing things we are ignorant about. Now, this helps us become less ignorant if we are introspective, but a lot of people are not doing it for that reason.
The fact that every new model generation has come with more capabilities should give the full skeptics at least a little pause that the foundations of their convictions may be incorrect.
It's not even clear that the actual LLMs have improved. It could be all the non-LLM software (harnesses), the system prompts, the sheer amount of compute hardware, etc. that makes them better. We could be quickly running up against a wall. The complete package of technology that is OpenAI's and Anthropic's products are completely opaque and proprietary. The psychic/con artist communication both from the LLM and the humans like Sam Altman only muddies the water further
People on HN have been claiming that LLMs are going to hit a wall any second now for the past two and a half years.
To reiterate, this is the very beginning of a long series of technological expansions that are going to come out of GenAI. The work is going to go on for decades. All you have to do is look at what happened with the mass-produced automobile, the personal computer, the internet, and mobile phones to see how long the propagation will continue before we settle into a new normal.
LLMs are really so new that the amount of exploration we've done on them is still tiny. It's also a field where a decade of work is being done every year. The vast majority of detractors I've seen always seem to be way way behind the latest models and even farther behind the papers coming out.
>non-LLM software (harnesses), the system prompts, the sheer amount of compute hardware,
Eh, this is turning into a messy chinese room argument. It is the room or is it the system. In my philosophy the chinese room argument is a non-starter. It's not the room, it's the system. For LLMS this would be like arguing that the output of a single prompt has to be able to answer everything which is nothing close to how human intelligence works. A single human thought is rarely intelligent, it's most often a replay of information it already has. Dialectic processes and loop processes are what tends to push the limits of human intelligence. We reach local maxima with thought alone, and this is boosted by things like writing down the problem and having other humans that may be even less intelligent than you add to the process. In fact this process works with one self by writing and reading ones own thoughts as it's using different subsystems of the mind for introspection.
The idea that LLMs have ran out of steam typically show more of a lack of imagination in the writer than what's occurring in the field.
(EDIT: I, like some others, just noticed this is an older post from 2023. That changes nothing in particular in my response here, but I would be curious to know if the OP's views have shifted at all in light of subsequent developments).
I feel like this post completely misses the point, pretty much across the board. And it does so by repeating the same mistake that everybody keeps making - conflating mechanism and function.
One of the issues in during this research—one that has perplexed me—has been that many people are convinced that language models, or specifically chat-based language models, are intelligent.
That's because they are intelligent.
But there isn’t any mechanism inherent in large language models (LLMs) that would seem to enable this and,
The mechanism is irrelevant to the issue of whether they are intelligent or not. Airplanes fly, despite not flapping their wings. The sign on the marquee says artificial intelligence.
LLMs are not brains and do not meaningfully share any of the mechanisms that animals or people use to reason or think.
Again, irrelevant. Nobody claims that they are brains, and it doesn't matter what mechanism they use. The sign on the marquee says artificial intelligence.
LLMs are a mathematical model of language tokens. You give a LLM text, and it will give you a mathematically plausible response to that text.
That's a bit overly reductionistic. And to the earlier point and, if real, it would be completely unexplained.
I'd probably leave out the word "completely" there, but it is fair to say that not everything about the underlying mechanism is understood. But at the risk of repeating myself, that's orthogonal to the question of whether or not they are intelligent.
There is no reason to believe that it thinks or reasons—indeed, every AI researcher and vendor to date has repeatedly emphasised that these models don’t think.
You mean "There is no reason to believe that it thinks or reasons like a human". Again, this is irrelevant to the question of whether or not they are intelligent. The sign on the marquee says artificial intelligence.
I don't know why people keep obsessing over mechanism in this discussion. It something functions as an intelligence, it is intelligent as far as I'm concerned - at least when the framing is a discussion of artificial intelligence.
And just to expand on why I reject the line of thought laid out in this article, let me share something from my own life as an illustrative example.
I'm working on a project, with a lot of help from ChatGPT, involving an "artificial neuron". That is, an electronic circuit, using a PUT, a capacitor, and some resistors, that simulates some of the behavior of a biological neuron. Specifically an "integrate and fire" model of neuron behavior. To that end, I'm running experiments by scripting my function generator to send signals to the circuit, and then capturing the inputs and outputs on my oscilloscope. Then I usually discuss the results with ChatGPT. In what follows, observe a couple of things:
1. The LLM "knows" the context of what we're talking about, even if I provide a prompt with no text at all, just an image.
2. It parses a moderately complex image, identifies the separate traces and what they represent, uses the time-base information displayed on screen, and the on-screen graticule, and works out "how many input pulses fire before an output pulse fires" and then reports back to me and gives an analysis of how that relates to our previous observations and gives suggestions for the next experiment to run.
Human intelligence? No. But I see no world where behavior like that does not count as "intelligent" regardless of the mechanism behind it. And that's probably not even the best example I could come up with, it's just something that was "top of mind" and for which I had the necessary images and what-not already ready, or easy to capture.
https://www.fogbeam.com/images/neuron_zero0.png
https://www.fogbeam.com/images/neuron_zero1.png
https://www.fogbeam.com/images/neuron_zero2.png
https://www.fogbeam.com/images/neuron_zero3.png
https://www.fogbeam.com/images/neuron_zero4.png
> The intelligence illusion is in the mind of the user and not in the LLM itself.
> Many AI critics, including myself, are firmly in the second camp.
How can you reconcile this with the fact that AI can solve a real, intelligence bound problem for me (with zero intelligent effort on my part) that you can't?
Is the solution also illusory?
Horseshoe theory. At each end of the shoe you tend to get loud crazies.
> The intelligence illusion is in the mind of the user and not in the LLM itself.
I think that might be the solution to consciousness illusion.
The consciousness might be purely in the mind of someone that believes some other entity to be conscious.
I like to think of it this way. Consciousness is the outcome of intelligence. You can't get consciousness without it. Conversely this also means things that are not conscious can still be intelligent.
All of this is accurate (and useful). But yeah, psychics do not prove theorems from frontier math and build working complex software.
No they don't but psychics also don't have a verifiable feedback mechanism and unlimited retries. Give the psychic the ability to adjust answers based on tests and you'll have have a fair fight.
In my opinion, this article is talking about people who get into long conversations and become convinced the model is “intelligent”, maybe even “AGI”. This seems to be a trap people are falling into, even in 2026.
It's definitely talking about that, but it's also talking about all of us:
- fawning over how amazing these tools are
- believing everything OpenAI and Anthropic say about how powerful and dangerous their product is
- minimizing the amount of human effort and involvement in every "AI" achievement
Initial definitions of AGI (as of 2014) are long surpassed. Then, AGI was defined as something that is general enough to work in many fields and orient themselves. It did not include being smarter than humans or even having comparable intelligence to humans. By original standards, we have AGI already for some time.
Current AGI definitions are intentionally vague.
Neither do LLMs. Humans using them do. Similar to how cops solve crimes using psychics.
Your statement is unclear, break it down a little further on how you think this works.
The original article already attempted to do that but I'll take another stab. Humans tend to believe what they want to believe. They gain confidence when a confident person (or LLM) they work with exudes confidence. Where they might have hesitated or got stuck in analysis paralysis alone, the aid of a con (short for confidence) artist will get them moving. They then attribute the solution and praise for it solely to the confident person (or LLM) that nudged them along. Did the "psychic" who understands human motivation better than the cops actually help? Very likely. Did the psychic end up getting far more credit than they deserve (it's magical! You'll never believe! Well worth the money I spent!)? Absolutely.
The amount of real help vs. confidence/manipulation the LLM or con artist provides varies in each situation and scenario and mix of people (and LLMs) involved. It's something to strive to be aware and analytical about.
There is also something else horizontally to this that occurs.
There are numerous scientific discoveries that have been made because one human looked at all available data and turned off the assumptions other people in the field have been using for years, maybe hundreds of years. Once you delete the assumption and make a new one the answer is obvious any anyone from that point wonders how so many humans could have missed it for so much time.
This can occur readily with LLMs as it can with people. It's very likely we will see this a lot as the causal connection in the available data will connected differently in their minds.
Escaping from a local maximum can be very difficult as you have to climb uphill with a nearly infinite amount of freedom but no ability to see the horizon. Psychics, LLMs, or some guy name Bob walking in an providing a workable analogy for you to escape the local maxima and move closer to a global maxima are all the same. Moreso, if LLMs are AGI we should expect these "psychic" behaviors just as much as actual discovery because both continuums exist in the same problem space as human minds.
On one side I agree that LLMs and Agents are not intelligent, but they present an illusion of intelligence given the shear amount of data they can process and act upon.
But that cannot be used to discredit the fact that these are incredibly powerful tools that can get out of control and cause great damage.
What is "intelligence" then. I think the universalism of LLMs is now evident enough that they can be considered "intelligent", even in slightly different realization compared to humans.
As of "great damage", I doubt it. They do not have self-preservation instinct (all "worrying" experiments are the attempts to initiate something resembling self-preservation from human initiative). The driving part is external - the query loop can always be turned off. So yes, a dangerous tool that can be exploited by humans (including governments, especially governments - which is why I am skeptical to government regulation proposals, particularly looking at what passes as governments in this era). But there are no inherent dangers from their own agency, as there is none.
> As of "great damage", I doubt it. They do not have self-preservation instinct
The potential for great damage relies upon the lack of self-preservation instinct. That lack releases the so-called AI from an order under pain of death. Damage cannot be punished.
"Slightly different"- no, a completely different realisation of intelligence to humans. LLMs can definitely do cool stuff though.
>completely different realisation of intelligence to humans.
OK, so they are intelligent. I'm glad we agree.
The problem with this word is people associate intelligence with the process they realize is occurring inside their head which is only a tiny fraction of what intelligence encompasses. Humans (without study) know very little about intelligence and they very gray boundaries it has.
Neither does LLMs. Not without a monumental amount of human oversight and expertise.
Everyone not in a particular field asking LLM about said field is rolling bad dice.
I have published a LLM-assisted proof of a long-standing problem in convex geometry (70 years) while not being a specialist in convex geometry. I am a scientist and I have education and scientific experience in computer science and systems biology, but I was never a mathematician. So, well, you can definitely work across fields at least.
Do you have a link? I would also be very interested in seeing the prompts, from what I've seen you still need to understand the field, maybe I'm wrong.
I predict a lot of butt hurt tokens in the comments...