(1) they are claiming to produce apparently bijective closed-form symbolic representations/approximations of, among other things, LLMs. Is evaluating these closed-form representations more computationally efficient? The implications of that are potentially huge. It would be essentially analytic distillation. Fable on a chip and not a data center would be important — and disruptive - in many ways.
(2) Unsupervised, and even supervised, symbolic approaches to problem solving break down due to combinatorial explosion, among other things. This could potentially allow us to treat LLM training and inference as a search algorithm for novel symbolic approaches to solving new classes of complex problems hitherto unreachable through other approaches. If that works, I suspect it’s a feedback loop, too - the learnings from one representation push advances in the other. This would also increase the economic value of large training runs, since the model itself is now valuable, not just its inference.
(3) Per the above, can this push LLM design to greater capabilities?
The relationship between this and Anthropic’s J-space observation is also interesting. This is much, much deeper and more directly actionable, though.
EDIT: I ran my questions through Sonnet — yes, I appreciate the irony — and it was none too sanguine about questions (1) and (2), but thought (3) was reasonable. In any case, this is quite the paper. On reflection, I do think that the apparent reliance on very simple symbolic representations and tasks is underwhelming. But the approach is impressive. And obviously this is still early days, and the value of building a bridge between the very fuzzy LLM models and the rigorous, mechanically provable models would be enormous.
> "Second, there is no guarantee that a given neural network can be approximated by DISCOVER"
Page 7.
They train what appears as embeddings for outer product of roles and fillers. The role for language model can be a position in text, the filler can be an embedding of a word at that position. Then that matrix of a sum of these outer products is linearly mapped into NN encodings and then decoded by NN decoder.
The embeddings learned by this process are not necessarily smaller than original ones. Given that they participate in an outer product computation gives me impression that the resulting sum is much bigger than actual NN encoding, that is why it needs to be linearly mapped into NN encoding.
So, this paper will not necessarily lead to any computation savings.
Good find! But they stop short of saying it cannot be distilled to symbolic algebra. Regardless, your point stands. I scanned the paper in the middle of the night instead of sleeping. Clearly I didn’t do a close read! Thank you for pointing that out.
Their representation is a key-value store that they embed via a tensor product to reconstruct the vectors the model produces. The catch is that it works best if the key-value mapping is task-specific. So if you already know how to solve the task the model is performing and can transform the input into a data structure that makes it easy to just read off the answer, you can probably also recover the model output.
That in itself is of course not super useful for tasks that we don't already know to do symbolically, but it's nice to see that they can manipulate the keys and values of their representation somewhat independently. So at least some of what LLMs do internally seems to involve a bit more structure than just linear combinations of some overcomplete dictionary.
The interesting question is whether this can be used to interpret the operation of a single layer by lifting it to operate on key-value stores instead, replacing matrix-vector products with reading from some keys and then writing to others. That could potentially move the balance of power more towards CPU inference instead of GPUs.
> So at least some of what LLMs do internally seems to involve a bit more structure than just linear combinations of some overcomplete dictionary.
Well that's good to hear. Would have been rather exasperating to find out we could have somehow gotten Fable or Sol out of sufficiently advanced sparse-coding ages ago.
It's a fascinating emergent behaviour but also one that could have been predicted?
I'd imagine that our brains have become similarly biased towards generating such a generalised symbolic structure over hundreds of millions of years of evolution?
I'm thinking how certain regions of the brain reliably represent things like motor skills, speech, abstract thought.
I agree. In retrospect, this seems almost inevitable. And our own minds at least to do some form of symbolic reasoning — literally language, which you are apparently capable of, dear reader, as a lower bound. There might be more symbolic reasoning in the conscious, unconscious, and subliminal parts of the mental workspace. I always hesitate to infer similarities between the brain/mind and LLMs, but I certainly track the comparison here.
My naive middle-of-the-night Claude question said the same thing. At least for this approach. I have not read the paper closely enough to refute you. But the concept of a lower-dimensional closed-form solution — which the paper seeks to discuss, please correct me if you read differently — is tantalizing, if only because it opens the path to different math, which can lead to optimization.
Is it though? Brute force searching generally scales like 2^n. LLMs on the other hand can obviously be trained, and even though training is massively resource intensive, it's not 2^n.
Inference is just tokens transformed through a fixed crystalline structure, no? You already could put that on a chip. There's no particular reason it couldn't be represented as some set of symbolic equations instead of a layered process... it's just another kind of quantization.
When symbolic algorithms are that large, they're equally hard to reason with most of the time. The upshot would be a lot more storage required in exchange for more generalized computing, lessening the need for so much GPU in a lot of cases.
I don't see why a model couldn't be represented that way. After all, if you just polled the output of a model, you could evolve genetic algorithms to predict it with fairly high accuracy in a limited domain. Take that out to the Nth degree and you're basically just unspooling the model into a giant set of equations.
Yes, you certainly could put that on a chip. And people are doing it, for smaller models. The question is how big that chip would be for something like Fable, which is generally estimated in the trillions+ of parameters, and if, given the amount of memory, computation, and bandwidth required - at least with current methods, i.e., very high dimensional matrix math — a chip is the right way to go. Can you put a LM (just a smaller language model) on a chip? Empirically, yes, I believe it is done. Small enough ones probably even fit on a FPGA. Can you put a LLM on a chip? Depends on how L it is! My intuition says that some open smaller models might fit, like Haiku, but not Fable. But that’s just intuition talking.
The hope would be that this unlocks some substantially more efficient or parsimonious math that would fit better on a chip. And that’s clearly my words, not the authors’, per the comment above.
Maybe someone can clear this up for me. I always thought it was already obvious that neural networks, artificial or otherwise, are symbol systems. Some input pattern reliably associated with some output pattern is, nearly by definition, a symbol. No? Am I just using the word symbol differently than researchers in these fields?
A big problem with some of these supervised* interpretability approaches is that they can find spurious structure. (There are lots of ways to make the model do what you want; which is roughly what Hewitt and Liang 2019 showed). This paper draws a contrast to a previous method, DAS (distributed alignment search) on page 20. These and related methods rest on theories of causal abstraction, which are great in theory, but harder in practice. DAS, for example, has faced numerous recent criticisms (Makelov 2024, Meloux 2025, Sutter 2025, Grant 2026, Kumon 2026). My favorite is the quite approachable Meloux et al.; Sutter 2025 is also really good, but relies on a sort of real number argument that allows a lossless encoding of every input.
My forthcoming paper at EMNLP offers an alternative that instead grounds the notion of representation in a very simple notion of the effect it has on model learning/behavior when you adversarially perturb it. For example, if I tell a model that in the context "I saw a duck quacking" it should replace 'duck' with 'glam', how much does it desire to replace 'duck' with 'glam' in "I need to duck out of the meeting" vs. "At the park a duck protected her ducklings." This method turns out to work quite well, and as we use only a single example, avoids the need for supervision.
The linked paper argues that their method, DISCOVER, is not supervised in the same way as DAS, since it does not directly optimize for causal effect. I have only skimmed this, but I am not so sure it might not suffer from a similar issue. They're still supervising to align representations with their underlying hypothesis, even if they don't directly supervise for causal outcomes.
Refs
- Hewitt and Liang 2019. Designing and interpreting probes with control tasks
- Kumon and Yanaka, 2026. Fine-grained analysis of shared syntactic mechanisms
- Meloux et al., 2025. Everything everywhere all at once
- Rozner and Shain 2026. Perturbation: A simple and efficient adversarial tracer for representation learning in LMs. https://arxiv.org/abs/2603.23821
- Sutter et al. 2025. The nonlinear representation dilemma
So, basically you're saying: I have only skimmed the article, similar approaches had issues in the past, and my own method from my forthcoming paper is better.
I mean, no disrespect, but that's the core of your argument, yes?
His arguments apply generally to a large family of interpretability methods. Skimming suffices to figure out that the linked article is in that family.
I have issue with the comment because he says he didn't read it, then unfavorably compares it to a previous method, and finally uses that negative review to plug his own article instead. His criticism might be valid, I'm not in a position to judge, but the self-promotion leaves a sour tastes in my mouth and makes me question how much of the criticism is just drummed up to make his own contribution appear more relevant.
to his credit he did say he skimmed the paper, and it's honestly standard practice to do a first pass of skimming a paper before you'd go deeper into reading it anyways
The posted paper does use supervision to find alignments. I need to do a little more math to figure out how well the supervision critique applies, but I think it does. And although the current paper mentions DAS, it fails to discuss recent critiques of supervised abstraction methods (the various citations, including my own, that I provided).
I'm encouraged by this result. It's the primary hypothesis behind latentpedia.org.
Instead of distilling the geometry of a model into a huge knowledge graph, we start from the largest known open source graphs and build it up towards something that resembles this geometry.
Come and join us. Discuss on github.com/latentpedia. We have the basic tech covered. Need more compute, storage and enough business to cover the cost of serving.
The math and core experimentation here is beyond my abilities, but what I think I understand is that there are possible deeper patterns of representation that exist in LLMs that are distillations of core conceptual relations in grammar that we can get our heads around in a mathematical sense rather than apparent layer-smeared noise that somehow, un-interpretably (in a meaningful sense), resolve to correct grammar/inferences.
That's pretty cool. I hope I've got that kinda-right.
I haven't read this in depth yet, though I plan to. If this general line of research is interesting to you, I'd recommend checking out some of the lines of research it touches upon--they're really rich and fascinating, and some are pretty approachable mathematically even if ML research papers aren't usually your thing. The related works section here seems pretty well stocked, but mechanistic interpretability is a pretty interesting peephole into this general vein: https://transformer-circuits.pub/
As I am going through the article, I was wondering why is this more interesting than having the ability to recover java programs from byte code. So I asked copilot the same question. It told me that - "Honestly this is where the difference between an engineer and researcher shows up!" .
it is not. in this case these researchers have lost their way, with how symbolics entered the conversation. to remind, it is via wanting to prove that ai vs traditional program is understanding deeper. well guess what, ai is not understanding your input better, its your mind playing tricks with language. For analogy, digital world is not real in a physical sense.
if output is food, and input is ingredients, then symbolic programs care about macro slicing dicing stacking them, while ai is micro level spice & heat that doesn't especially conscious to the ingredients, just that chemistry appears magical. hey! we eat our information food tho.
> while ai is micro level spice & heat that doesn't especially conscious to the ingredients, just that chemistry appears magical.
Huh? Do you mean “isn’t especially conscious of the ingredients”?
Though, that interpretation seems confusing, because it seems to be meant to be contrasting with “macro slicing dicing stacking them” which doesn’t sound more attentive to what the ingredients are than “micro level spice and heat”.
So, I’m having trouble understanding. (This might be a me problem.)
yes, you are right there is no contrast, because both ai(vector nn) and traditional symbolic programs are not aware of inputs/understanding deeper/"excel in domain" outside our interpretation of it.
In the way digital is the symbolic representation of physical, ai programs may have internal symbolic representation. This paper shows that nn vectors can be closely approx with symbolic structures.
It's very caring and reassuring also. It gave me an elaborated response on how researchers may discuss ridiculously fun theories. And what should be my takeaways as engineer. I guess I should turn off the Work IQ.
Imagine a box of balls. They have size, weight, colour, density… etc. These properties, each a measure, are dimensions and they are orthogonal to each other. Taken together are multi-dimensional.
Now take a set of words. They have "sizeness", "weightness", "colorness" and "densityness"...and "pythonness" and "haskellness" and even "adjectiveness" and "verbness" and so on and so forth...You can readily see that this can encode arbitrary patters (like language grammer or program syntax)
Training identifies these dimensions in the training data and links it with each word/token. Then given a stream of such tokens, each with its own set of dimensions (which can be huge), and LLM predicts the dimensions that the next token is most likely to have...
my take from all these is that is perhaps possible to teach similarity search to 5th graders. not an elementary teacher myself, but many math concepts are much easier to explain with allegories (visual and other) than with formalism. a good example perhaps is what Mamikon does in his visual explanations of calculus (https://calteches.library.caltech.edu/4007/1/Calculus.pdf)
formalisms come only after there is understanding of the phenomena and a specific need to communicate concepts across members of community - in a concise and commonly comprehensive way. my understanding is that education often fails to train understanding and jumps directly to formalisms. and this is right also for STEM schools, where it is presumed everyone has some innate inclination to the matter. but this presumption, in my experience as university teacher, is not entirely correct, and students are not universally prepared to express lack of understanding, and particularly when it is expected from them to rather indicate understanding.
it seems very weird, the concept, to reward the correct answer, while suppressing and discouraging the expression of misunderstanding or confusion. why is weird? because learning comes trough trial and error, and the error part is half of it. so we need to encourage people to express their errors or lack of understanding, so that we can then fill in the gaps and eventually stand on a common ground. sadly, many forums, HN included, does not cherish misunderstanding. our posts here mostly get upvoted whenever they cross a sort of threshold about the amount of truth in our writing, and not because of daring expressions of confusion.
This is nonsense. The human mind cannot visualize more than 3 dimensions. It can perfectly comprehend any number of dimensions as long as they are represented in a vector space. In fact, that's what linear algebra does.
I’ve heard people claiming that they can but I’ve never heard compelling evidence that they’re directly visualising 4D objects rather than 3D projections of them, or some symbolic representation (arrays of numbers etc.)
All I’ve been able to do is mentally tag parts of a shape as being at different “depths” (possibly with two parts at the same 3D location but tagged as having different “depths”). Which, yeah, I agree isn’t really visualizing a 4D shape. I think someone better at it than me might be able to use this to leverage one’s visual spatial intuition in a way that is a bit closer to what 4D visualization might be like if it were possible.
But I don’t think it is possible for us. At least, for strict senses of visualization. I suspect that the architecture of our brains doesn’t support it. This is rather speculative because I don’t know basically any neuroscience, but I wonder if the whole “the brain is largely a very wrinkled surface (though with several layers to this surface), and the retinas are also surfaces” may have something to do with this. A retina in a eye-4-ball would have a 3D boundary (I.e. hypersurface), and I imagine that the amount of information that it would take in if it had an at all similar angular resolution, would be too great for our visual cortex to be able to represent all of it.
Of course, our vision only really has the high resolution it seems to everywhere in the narrow region we’re directly looking at, but still.
So it is like 3D because you do not visualize the complete 3D models but only 2D projections of them. Or do you really visualize the back and inside of a vase at the same time?
If I visualise a 3D object then it’s in at least pseudo-3D (as in front surface with depth) but I can switch it to a kinda volumetric view. This might be thanks to years worth of 3D modeling and CAD which helped me map between first person and third person cameras and between solid and transparent, though. 4D doesn’t even make sense though, like I can do an animated 3D volume?
Yes. It's worth pointing out that anything with n distinct parameters is just a point in n-dimensional space. We're so used to handling so many dimensions that nobody ever bats an eye until someone brings up the magic word "dimensions". It's quite intuitive actually.
The trivial example that comes to mind is the character customization sliders in many video games.
They're borderline useless and certainly potentially inadvertently malicious for writing, customer service, speech to text, writing large amounts of complex code, therapy, medical diagnostics... the list goes on
Sounds like you're part of the problem?
It's really a serious problem just eroding the fabric of society in real-time. Being complacent in it or believing in the promise is just wholly foolish and bad for everyone.
Too shallow of a dismissal, and you don't determine what everyone else takes seriously.
It's been several years now of LLMs only appeasing those with low expectations and inexperience. Unless the only goal was generating boilerplate or really sloppy proofs of concept, LLMs are a waste time for everyone else. This argument is so over already. We're all just hoping for a soft landing when the hangover really kicks in.
You are disconnected from reality. The whole industry is already completely dominated by LLMs generating code. Bury your head in the sand all you want. This is not about low expectations or inexperience at all. Your condescending tone doesn't make you look smarter, it makes you look like an Amish who expects the industrial revolution is temporary and soon people will come to their senses and stop using all this nonsense industrial technology.
I spent the last 6 months reviewing all of the AI generated code primarily with gpt 5.3 codex but after I bought the subscription I also used gpt 5.5 and although I was constantly rejecting the bad code and I was mostly happy with the end result.
Then they removed the approve everything manually mode and I thought, maybe I'm the one out of touch, I should try vibe coding. I gave it a big spec of things to implement and I was pretty vague on the details, but I was kind of hoping that the AI would get it right anyway.
I noticed that when you let the AI just do its thing, you spend a lot of time staring at the screen waiting for it to finish. No wonder all these people are obsessed with parallel agents.
Anyway, after a while it produced the end result and it was pretty bad and I don't even mean the code. This was with gpt 5.6 Terra. Definitively not the type of "did a week's worth of work in one hour" type of deal.
Meanwhile if you guide it with your own technical skills the end result is a tireless code generator that operates at 80% of the quality of your own code but is only roughly 1.5x to 2x faster than doing it yourself. The biggest multipliers come from making the AI do things humans don't want to do, like writing a bunch of thorough tests but those don't translate into increased productivity, the opposite in fact. They translate into higher software quality but if you vibe the tests and don't review them, even that disappears.
Doesn't that say more about the massive crumb tray nobody ever bothered to empty at the bottom of mathematics?
I'm sure someone will point out something like the 4-color theorem as a counterargument. Where is that kind of theorem proving in this generation of AI? We seem to have hit a dead end rather quickly.
Sounds reasonable... That the model is sometimes learning a lossy vector representation of something symbolic in nature... Sure, a NN can approximate a function?
They say this holds in... Some examples they found?
Reading stuff like this (as a layman), diminishing these things as 'Next token predictors' seems absurdly reductive. At some point we'll need to concede that 'selection' is a better term for this than prediction.
> which in no way challenge the established view that these bots are next-token predictors.
I mean, of course they are, that's literally what the inference loop does. You can look at the source of your favorite model runner and you'll see exactly that.
What I find misleading about this term is that it focuses attention on the "next token" part and glosses over the "prediction" part as some sort of unspecified "statistical algorithm" - even though this is where most of the work happens and where the interesting questions are.
Symbolism has tried to strike back repeatedly ever since statistical learning revived with AlexNet. With all the due respect one can have for the names Smolensky and Linzen from the perspective of linguistics, the question about the applicability, generalizability and robustness of the method proposed here should be raised. It seems from section 3.5 of the paper that one cannot be so optimistic about it at least as yet. I get it that the method is still in its infancy, but we've already got the kind of Mech Interp as pushed forward by Neel Nanda and co, among other lines of research. Not that we are forced to make a choice between all interpretability works, or this TPR method is inherently inferior to the other ones, but we can be moderately cautious when looking at such progress.
this is an obvious result. for example, this guy has been writing on substack about this for at least a year or two (with code snippets) explaining the phenomenon of grokking and the ghostbasin.com concept - https://richardaragon.substack.com/
their algorithm is even named "DISCOVER" so they set out to discover the connective tissue of why the universe has invariants like math, and lo it was discovered.
i guess good job for having credentials & publishing the math so people 2years behind the curve can learn from your tenure?
yes. large matrices can gradient descend to understand arbitrary symbolic logic.
ENGLISH IS INSUFFICIENT but it is at least a few decades of math proofs & progress :) welcome to the future Slackernews
Post the actual articles that you have in mind. What I've seen is vague slop.
A good example is https://richardaragon.substack.com/p/a-universal-prime-funct... describing a supposed "universal prime function" which is simply a finite approximation using a sum of 50 sines (each applied to a linear term plus a sine-log offset). The 53 parameters are fitted to the first 10^3 or so prime numbers. This is followed by the *absolutely ridiculous* claim that if the function approximates the first 10^3 primes well, it must also fit the remaining prime numbers (of which there are infinitely more than 10^3000000000) equally well.
Then they suggest "A formal proof connecting this function to the RH would involve the following steps" using this great discovery: "1. Correspondence with the Explicit Formula: Demonstrate that the oscillatory correction term in our function corresponds to the sum over zeta zeros in the explicit formula for ψ(x) or π(x). 2. Error Bound: Prove that the error in the prime counting function derived from our function is bounded by O(√x log x). 3. Contradiction: Show that if any non-trivial zero were to lie off the critical line ℜ(s) = 1/2, the error would exceed the bound, leading to a contradiction."
This isn't even midwit math.
It's the kind of naive ideas I had as a high schooler, who was good at high school math and who knew how to code functions and plots in Mathematica, but who had no understanding of higher math. This kind of naive approach to RH signals that one doesn't even understand the problem.
thanks for engaging but tbh just post your own articles. i actually talked to Richard after reading quite in depth and reproducing his code results myself, and do think he's brilliant. my point is he gets the punch line of these math academia papers, much faster.
that's all. easy to call things "vague slop", hard to be right first. Jensen's def'n of intelligence "can you see around corners and predict the future?" kinda all that matters at this point.
just an fyi. congrats on your nitpicking of what math counts as math lol
"Vectors seem inadequate for capturing the structure of language, logic, and other cognitive domains, yet neural networks achieve impressive performance in these areas". Missing the forest for the trees? Aren't neural networks modeled after biological systems? Our brains are obviously able to contain symbolic structure despite not having a "symbol processing unit".
People really overstate the relationship between ANNs and the brain, they have very different mechanisms and only have a similarity if you squint at 100000 feet. ANNs don't have neurotransmitters or even action potentials.
But neurotransmitters and action potentials don't help in modeling symbolic structure.
That is, yes, ANNs are not brains. There are countless differences. But are there differences at the computational level? ANNs are meant to model brain computation, not brain biology.
(There is still a lot to debate there, I'm not saying "ANNs are perfect computational models for the brain")
>But neurotransmitters and action potentials don't help in modeling symbolic structure.
This is exactly why Fodor argues that psychology should be explained on its on level with symbols rather than appealing to neurology. But if you're interested in modeling symbols, there's much better options than ANNs (see nearly any programming language ever).
>ANNs are meant to model brain computation
But we don't really know how that works! So if you know if you're not modelling the low-level behaviour right, you can't assume that there's a correspondence of the higher level computation when you don't really know what that higher level computations are.
We don't know all the details about how the brain computes, you are right.
But we do have a hypothesis: that it is done by a large number of simple units with very high connectivity and in deep layers. This is what neural networks model.
Personally I was skeptical of this model of the brain, but they have achieved remarkable success in practice, as well as Nobel prizes. The neural networks people may have been onto something all along (I say that grudgingly).
But the models learn in a very different way than humans, they're certainly vastly more sample inefficient. I'm not saying it isn't impressive or that it isn't necessarily a kind of intelligence, I'm just sceptical as to how much it really tells us about the brain.
While I agree with this statement, one could argue that what's important is not much the support but the emergent properties - in the same way that a wave is still a wave whether it is in the water or in the air.
Thus putting things similar to neurons in a network and making them able to learn could create behaviors similar to the brain. The fact that that biology used chemistry + electrical signal and computers use ReLU-like activation could be merely choosing the most efficient way to enable training.
Correct. Worst of all, even if you do build a spiking neural network, the update rule is kind of a mystery. To have a good update rule, a biological neuron needs to be kind of like a tiny computer in its own right. You might be able to model synapses as weights between neurons, but the neuron carries further internal states within in itself and how the "update rule" uses those internal states is not known at all.
I hate that whole intro - the first four sentences - so much. It’s nothing but unsupported assumptions. Basically, a strawman that they can do battle with in the paper. Not an auspicious start.
Paul Smolensky is a cognitive science titan from that era. He worked with Hinton, Rumelhart, and McClelland on parallel distributed processing, and literally wrote the book on tensor product representations in cognition, with Geraldine Legendre: https://mitpress.mit.edu/9780262516198/the-harmonic-mind-vol...
He's the axis of this particular group of researchers, being the most senior at the place where they all met, Johns Hopkins.
So this is less a straw man and more a quick reminder to his peers: "Right, so, remember this particular thread we've spent the last 40 years hashing out, here we've got another contribution to that particular conversation."
I’m aware, but it was a bad mainstream opinion which was contradicted by every bit of evidence ever since early LISP efforts in the late 1960s (still waiting for the release of SHRDLU 2.0!)
65 years is a long time to be beating your head against an obvious dead end.
The article intro presents the claims as self-evident, when they’re not at all.
But, someone rise pointed out this may have been a dig at those attitudes, which makes more sense.
It's like Neo says "You get used to it, though. Your brain does the translating. I don't even see the code." He was referring to something like a K, Q, V vector at the time I believe.
From the abstract "Further, our symbolic approximation allows us to modify an LLM's behavior in targeted ways via precise interventions on its internal representations [...]".
If this is true and easily computable, this might have big impact in AI safety, as it seems to be really lacking today.
The big questions I’m taking away are:
(1) they are claiming to produce apparently bijective closed-form symbolic representations/approximations of, among other things, LLMs. Is evaluating these closed-form representations more computationally efficient? The implications of that are potentially huge. It would be essentially analytic distillation. Fable on a chip and not a data center would be important — and disruptive - in many ways.
(2) Unsupervised, and even supervised, symbolic approaches to problem solving break down due to combinatorial explosion, among other things. This could potentially allow us to treat LLM training and inference as a search algorithm for novel symbolic approaches to solving new classes of complex problems hitherto unreachable through other approaches. If that works, I suspect it’s a feedback loop, too - the learnings from one representation push advances in the other. This would also increase the economic value of large training runs, since the model itself is now valuable, not just its inference.
(3) Per the above, can this push LLM design to greater capabilities?
The relationship between this and Anthropic’s J-space observation is also interesting. This is much, much deeper and more directly actionable, though.
EDIT: I ran my questions through Sonnet — yes, I appreciate the irony — and it was none too sanguine about questions (1) and (2), but thought (3) was reasonable. In any case, this is quite the paper. On reflection, I do think that the apparent reliance on very simple symbolic representations and tasks is underwhelming. But the approach is impressive. And obviously this is still early days, and the value of building a bridge between the very fuzzy LLM models and the rigorous, mechanically provable models would be enormous.
They train what appears as embeddings for outer product of roles and fillers. The role for language model can be a position in text, the filler can be an embedding of a word at that position. Then that matrix of a sum of these outer products is linearly mapped into NN encodings and then decoded by NN decoder.
The embeddings learned by this process are not necessarily smaller than original ones. Given that they participate in an outer product computation gives me impression that the resulting sum is much bigger than actual NN encoding, that is why it needs to be linearly mapped into NN encoding.
So, this paper will not necessarily lead to any computation savings.
But I am at page 6. ;)
Good find! But they stop short of saying it cannot be distilled to symbolic algebra. Regardless, your point stands. I scanned the paper in the middle of the night instead of sleeping. Clearly I didn’t do a close read! Thank you for pointing that out.
Their representation is a key-value store that they embed via a tensor product to reconstruct the vectors the model produces. The catch is that it works best if the key-value mapping is task-specific. So if you already know how to solve the task the model is performing and can transform the input into a data structure that makes it easy to just read off the answer, you can probably also recover the model output.
That in itself is of course not super useful for tasks that we don't already know to do symbolically, but it's nice to see that they can manipulate the keys and values of their representation somewhat independently. So at least some of what LLMs do internally seems to involve a bit more structure than just linear combinations of some overcomplete dictionary.
The interesting question is whether this can be used to interpret the operation of a single layer by lifting it to operate on key-value stores instead, replacing matrix-vector products with reading from some keys and then writing to others. That could potentially move the balance of power more towards CPU inference instead of GPUs.
> So at least some of what LLMs do internally seems to involve a bit more structure than just linear combinations of some overcomplete dictionary.
Well that's good to hear. Would have been rather exasperating to find out we could have somehow gotten Fable or Sol out of sufficiently advanced sparse-coding ages ago.
It's a fascinating emergent behaviour but also one that could have been predicted?
I'd imagine that our brains have become similarly biased towards generating such a generalised symbolic structure over hundreds of millions of years of evolution?
I'm thinking how certain regions of the brain reliably represent things like motor skills, speech, abstract thought.
I agree. In retrospect, this seems almost inevitable. And our own minds at least to do some form of symbolic reasoning — literally language, which you are apparently capable of, dear reader, as a lower bound. There might be more symbolic reasoning in the conscious, unconscious, and subliminal parts of the mental workspace. I always hesitate to infer similarities between the brain/mind and LLMs, but I certainly track the comparison here.
> Is evaluating these closed-form representations more computationally efficient?
No it is way worse since computationally they are same things with symbolic having overhead of attending to the symbols.
My naive middle-of-the-night Claude question said the same thing. At least for this approach. I have not read the paper closely enough to refute you. But the concept of a lower-dimensional closed-form solution — which the paper seeks to discuss, please correct me if you read differently — is tantalizing, if only because it opens the path to different math, which can lead to optimization.
Is it though? Brute force searching generally scales like 2^n. LLMs on the other hand can obviously be trained, and even though training is massively resource intensive, it's not 2^n.
Inference is just tokens transformed through a fixed crystalline structure, no? You already could put that on a chip. There's no particular reason it couldn't be represented as some set of symbolic equations instead of a layered process... it's just another kind of quantization.
When symbolic algorithms are that large, they're equally hard to reason with most of the time. The upshot would be a lot more storage required in exchange for more generalized computing, lessening the need for so much GPU in a lot of cases.
I don't see why a model couldn't be represented that way. After all, if you just polled the output of a model, you could evolve genetic algorithms to predict it with fairly high accuracy in a limited domain. Take that out to the Nth degree and you're basically just unspooling the model into a giant set of equations.
Yes, you certainly could put that on a chip. And people are doing it, for smaller models. The question is how big that chip would be for something like Fable, which is generally estimated in the trillions+ of parameters, and if, given the amount of memory, computation, and bandwidth required - at least with current methods, i.e., very high dimensional matrix math — a chip is the right way to go. Can you put a LM (just a smaller language model) on a chip? Empirically, yes, I believe it is done. Small enough ones probably even fit on a FPGA. Can you put a LLM on a chip? Depends on how L it is! My intuition says that some open smaller models might fit, like Haiku, but not Fable. But that’s just intuition talking.
The hope would be that this unlocks some substantially more efficient or parsimonious math that would fit better on a chip. And that’s clearly my words, not the authors’, per the comment above.
Maybe someone can clear this up for me. I always thought it was already obvious that neural networks, artificial or otherwise, are symbol systems. Some input pattern reliably associated with some output pattern is, nearly by definition, a symbol. No? Am I just using the word symbol differently than researchers in these fields?
A big problem with some of these supervised* interpretability approaches is that they can find spurious structure. (There are lots of ways to make the model do what you want; which is roughly what Hewitt and Liang 2019 showed). This paper draws a contrast to a previous method, DAS (distributed alignment search) on page 20. These and related methods rest on theories of causal abstraction, which are great in theory, but harder in practice. DAS, for example, has faced numerous recent criticisms (Makelov 2024, Meloux 2025, Sutter 2025, Grant 2026, Kumon 2026). My favorite is the quite approachable Meloux et al.; Sutter 2025 is also really good, but relies on a sort of real number argument that allows a lossless encoding of every input.
My forthcoming paper at EMNLP offers an alternative that instead grounds the notion of representation in a very simple notion of the effect it has on model learning/behavior when you adversarially perturb it. For example, if I tell a model that in the context "I saw a duck quacking" it should replace 'duck' with 'glam', how much does it desire to replace 'duck' with 'glam' in "I need to duck out of the meeting" vs. "At the park a duck protected her ducklings." This method turns out to work quite well, and as we use only a single example, avoids the need for supervision.
The linked paper argues that their method, DISCOVER, is not supervised in the same way as DAS, since it does not directly optimize for causal effect. I have only skimmed this, but I am not so sure it might not suffer from a similar issue. They're still supervising to align representations with their underlying hypothesis, even if they don't directly supervise for causal outcomes.
Refs
- Hewitt and Liang 2019. Designing and interpreting probes with control tasks
- Kumon and Yanaka, 2026. Fine-grained analysis of shared syntactic mechanisms
- Meloux et al., 2025. Everything everywhere all at once
- Rozner and Shain 2026. Perturbation: A simple and efficient adversarial tracer for representation learning in LMs. https://arxiv.org/abs/2603.23821
- Sutter et al. 2025. The nonlinear representation dilemma
So, basically you're saying: I have only skimmed the article, similar approaches had issues in the past, and my own method from my forthcoming paper is better.
I mean, no disrespect, but that's the core of your argument, yes?
His arguments apply generally to a large family of interpretability methods. Skimming suffices to figure out that the linked article is in that family.
It certainly comes across as disrespectful.
To my reading the nature of supervision creating the structure seems to be the core of the argument.
I have issue with the comment because he says he didn't read it, then unfavorably compares it to a previous method, and finally uses that negative review to plug his own article instead. His criticism might be valid, I'm not in a position to judge, but the self-promotion leaves a sour tastes in my mouth and makes me question how much of the criticism is just drummed up to make his own contribution appear more relevant.
to his credit he did say he skimmed the paper, and it's honestly standard practice to do a first pass of skimming a paper before you'd go deeper into reading it anyways
The posted paper does use supervision to find alignments. I need to do a little more math to figure out how well the supervision critique applies, but I think it does. And although the current paper mentions DAS, it fails to discuss recent critiques of supervised abstraction methods (the various citations, including my own, that I provided).
I'm encouraged by this result. It's the primary hypothesis behind latentpedia.org.
Instead of distilling the geometry of a model into a huge knowledge graph, we start from the largest known open source graphs and build it up towards something that resembles this geometry.
Come and join us. Discuss on github.com/latentpedia. We have the basic tech covered. Need more compute, storage and enough business to cover the cost of serving.
The math and core experimentation here is beyond my abilities, but what I think I understand is that there are possible deeper patterns of representation that exist in LLMs that are distillations of core conceptual relations in grammar that we can get our heads around in a mathematical sense rather than apparent layer-smeared noise that somehow, un-interpretably (in a meaningful sense), resolve to correct grammar/inferences.
That's pretty cool. I hope I've got that kinda-right.
I haven't read this in depth yet, though I plan to. If this general line of research is interesting to you, I'd recommend checking out some of the lines of research it touches upon--they're really rich and fascinating, and some are pretty approachable mathematically even if ML research papers aren't usually your thing. The related works section here seems pretty well stocked, but mechanistic interpretability is a pretty interesting peephole into this general vein: https://transformer-circuits.pub/
distillation is now illegal tho
As I am going through the article, I was wondering why is this more interesting than having the ability to recover java programs from byte code. So I asked copilot the same question. It told me that - "Honestly this is where the difference between an engineer and researcher shows up!" .
it is not. in this case these researchers have lost their way, with how symbolics entered the conversation. to remind, it is via wanting to prove that ai vs traditional program is understanding deeper. well guess what, ai is not understanding your input better, its your mind playing tricks with language. For analogy, digital world is not real in a physical sense. if output is food, and input is ingredients, then symbolic programs care about macro slicing dicing stacking them, while ai is micro level spice & heat that doesn't especially conscious to the ingredients, just that chemistry appears magical. hey! we eat our information food tho.
> while ai is micro level spice & heat that doesn't especially conscious to the ingredients, just that chemistry appears magical.
Huh? Do you mean “isn’t especially conscious of the ingredients”?
Though, that interpretation seems confusing, because it seems to be meant to be contrasting with “macro slicing dicing stacking them” which doesn’t sound more attentive to what the ingredients are than “micro level spice and heat”.
So, I’m having trouble understanding. (This might be a me problem.)
yes, you are right there is no contrast, because both ai(vector nn) and traditional symbolic programs are not aware of inputs/understanding deeper/"excel in domain" outside our interpretation of it.
In the way digital is the symbolic representation of physical, ai programs may have internal symbolic representation. This paper shows that nn vectors can be closely approx with symbolic structures.
as long as it is honest, everything is ok.
It's very caring and reassuring also. It gave me an elaborated response on how researchers may discuss ridiculously fun theories. And what should be my takeaways as engineer. I guess I should turn off the Work IQ.
The human mind cannot comprehend the capacity of massively multidimensional space.
Just going from 2D to 3D creates massive new positional potential (e.g. surface of the earth, vs. the atmosphere above earth...).
Now imagine 1,000 dimensions.
Imagine a box of balls. They have size, weight, colour, density… etc. These properties, each a measure, are dimensions and they are orthogonal to each other. Taken together are multi-dimensional.
Now take a set of words. They have "sizeness", "weightness", "colorness" and "densityness"...and "pythonness" and "haskellness" and even "adjectiveness" and "verbness" and so on and so forth...You can readily see that this can encode arbitrary patters (like language grammer or program syntax)
Training identifies these dimensions in the training data and links it with each word/token. Then given a stream of such tokens, each with its own set of dimensions (which can be huge), and LLM predicts the dimensions that the next token is most likely to have...
my take from all these is that is perhaps possible to teach similarity search to 5th graders. not an elementary teacher myself, but many math concepts are much easier to explain with allegories (visual and other) than with formalism. a good example perhaps is what Mamikon does in his visual explanations of calculus (https://calteches.library.caltech.edu/4007/1/Calculus.pdf)
formalisms come only after there is understanding of the phenomena and a specific need to communicate concepts across members of community - in a concise and commonly comprehensive way. my understanding is that education often fails to train understanding and jumps directly to formalisms. and this is right also for STEM schools, where it is presumed everyone has some innate inclination to the matter. but this presumption, in my experience as university teacher, is not entirely correct, and students are not universally prepared to express lack of understanding, and particularly when it is expected from them to rather indicate understanding.
it seems very weird, the concept, to reward the correct answer, while suppressing and discouraging the expression of misunderstanding or confusion. why is weird? because learning comes trough trial and error, and the error part is half of it. so we need to encourage people to express their errors or lack of understanding, so that we can then fill in the gaps and eventually stand on a common ground. sadly, many forums, HN included, does not cherish misunderstanding. our posts here mostly get upvoted whenever they cross a sort of threshold about the amount of truth in our writing, and not because of daring expressions of confusion.
This is nonsense. The human mind cannot visualize more than 3 dimensions. It can perfectly comprehend any number of dimensions as long as they are represented in a vector space. In fact, that's what linear algebra does.
I think you accidentally a word, there. GP is talking about comprehending the capacity of massively multi-dimensional space.
Correct. I implied nothing about visualizing. I was talking about (information) capacity.
I think the human mind can visualize anything, eg here are some visual descriptions of higher dimensional objects: https://gfody.substack.com/p/mini-hyper-golf
My aphantasia says your milage may vary on that.
On the other hand I deal with higher dimensions better than most, possibly because of my aphantasia.
I think it would be fair to say that at least a few people can do a small amount of 4d visualisation.
Agreed, I don't think anyone has a good grasp on, say, 64K D, but we can do a bit better than just 3.
I’ve heard people claiming that they can but I’ve never heard compelling evidence that they’re directly visualising 4D objects rather than 3D projections of them, or some symbolic representation (arrays of numbers etc.)
All I’ve been able to do is mentally tag parts of a shape as being at different “depths” (possibly with two parts at the same 3D location but tagged as having different “depths”). Which, yeah, I agree isn’t really visualizing a 4D shape. I think someone better at it than me might be able to use this to leverage one’s visual spatial intuition in a way that is a bit closer to what 4D visualization might be like if it were possible.
But I don’t think it is possible for us. At least, for strict senses of visualization. I suspect that the architecture of our brains doesn’t support it. This is rather speculative because I don’t know basically any neuroscience, but I wonder if the whole “the brain is largely a very wrinkled surface (though with several layers to this surface), and the retinas are also surfaces” may have something to do with this. A retina in a eye-4-ball would have a 3D boundary (I.e. hypersurface), and I imagine that the amount of information that it would take in if it had an at all similar angular resolution, would be too great for our visual cortex to be able to represent all of it.
Of course, our vision only really has the high resolution it seems to everywhere in the narrow region we’re directly looking at, but still.
So it is like 3D because you do not visualize the complete 3D models but only 2D projections of them. Or do you really visualize the back and inside of a vase at the same time?
If I visualise a 3D object then it’s in at least pseudo-3D (as in front surface with depth) but I can switch it to a kinda volumetric view. This might be thanks to years worth of 3D modeling and CAD which helped me map between first person and third person cameras and between solid and transparent, though. 4D doesn’t even make sense though, like I can do an animated 3D volume?
Yes. It's worth pointing out that anything with n distinct parameters is just a point in n-dimensional space. We're so used to handling so many dimensions that nobody ever bats an eye until someone brings up the magic word "dimensions". It's quite intuitive actually.
The trivial example that comes to mind is the character customization sliders in many video games.
>The human mind cannot comprehend the capacity of massively multidimensional space.
That is why the scam works, because investors are humans...
Which scam sorry?
The scam that is based on the implicit claim that LLM is a path to AGI.
Seeing LLMs for what they really are will also make it clear they are fundamentally unfit for a lot of tasks they are currently marketed for...
Too broad a statement, and without substantiation, to be taken serious, sorry.
Is it?
They're borderline useless and certainly potentially inadvertently malicious for writing, customer service, speech to text, writing large amounts of complex code, therapy, medical diagnostics... the list goes on
Sounds like you're part of the problem?
It's really a serious problem just eroding the fabric of society in real-time. Being complacent in it or believing in the promise is just wholly foolish and bad for everyone.
Too shallow of a dismissal, and you don't determine what everyone else takes seriously.
It's been several years now of LLMs only appeasing those with low expectations and inexperience. Unless the only goal was generating boilerplate or really sloppy proofs of concept, LLMs are a waste time for everyone else. This argument is so over already. We're all just hoping for a soft landing when the hangover really kicks in.
I have 40+ years experience and extremely high standards. What you say is entirely wrong.
>I have 40+ years experience
What domain?
Why is it always the ones with the highest karma giving the most hollow defense? How's retirement treating you?
You are disconnected from reality. The whole industry is already completely dominated by LLMs generating code. Bury your head in the sand all you want. This is not about low expectations or inexperience at all. Your condescending tone doesn't make you look smarter, it makes you look like an Amish who expects the industrial revolution is temporary and soon people will come to their senses and stop using all this nonsense industrial technology.
A sidenote, but that is not a typical Amish opinion of technology:
"Amish do not view technology as evil, and individuals may petition for acceptance of a particular technology in the local community" - https://en.wikipedia.org/wiki/Amish_way_of_life#Use_of_moder...
See also https://thereader.mitpress.mit.edu/how-the-amish-let-technol... about some Amish use of cell phones with adapters to be more payphone/landline-like and https://news.ycombinator.com/item?id=21349989 was a discussion about the use of pneumatic machinery by the Amish.
Yup and the industry is worse than ever.
Maybe you forgot that important tidbit?
People are using them because they're being shoved down their throats and they're complacent.
Software quality, maintainability, exploitability, morale, competency are all at all-time lows and just worsening.
It's really bad to defend this.
I spent the last 6 months reviewing all of the AI generated code primarily with gpt 5.3 codex but after I bought the subscription I also used gpt 5.5 and although I was constantly rejecting the bad code and I was mostly happy with the end result.
Then they removed the approve everything manually mode and I thought, maybe I'm the one out of touch, I should try vibe coding. I gave it a big spec of things to implement and I was pretty vague on the details, but I was kind of hoping that the AI would get it right anyway.
I noticed that when you let the AI just do its thing, you spend a lot of time staring at the screen waiting for it to finish. No wonder all these people are obsessed with parallel agents.
Anyway, after a while it produced the end result and it was pretty bad and I don't even mean the code. This was with gpt 5.6 Terra. Definitively not the type of "did a week's worth of work in one hour" type of deal.
Meanwhile if you guide it with your own technical skills the end result is a tireless code generator that operates at 80% of the quality of your own code but is only roughly 1.5x to 2x faster than doing it yourself. The biggest multipliers come from making the AI do things humans don't want to do, like writing a bunch of thorough tests but those don't translate into increased productivity, the opposite in fact. They translate into higher software quality but if you vibe the tests and don't review them, even that disappears.
to avoid lot os problems i use kaplira to help maintain de code in boundaries and not let the agent do everything they want.
yes i have written a jillion lines with AI as well
i'm a worse programmer now because of it
What about the mathematical advancements?
Doesn't that say more about the massive crumb tray nobody ever bothered to empty at the bottom of mathematics?
I'm sure someone will point out something like the 4-color theorem as a counterargument. Where is that kind of theorem proving in this generation of AI? We seem to have hit a dead end rather quickly.
Lol, are you saying the Erdos problems are a "crumb" that nobody bothered to empty? Are you doing a comedy routine?
Explain to the rest of us how far along those problems were before AI.
Are they crumbs? Why aren’t you answering?
Sounds reasonable... That the model is sometimes learning a lossy vector representation of something symbolic in nature... Sure, a NN can approximate a function?
They say this holds in... Some examples they found?
I don't enough about this area
yep
Reading stuff like this (as a layman), diminishing these things as 'Next token predictors' seems absurdly reductive. At some point we'll need to concede that 'selection' is a better term for this than prediction.
> diminishing these things as 'Next token predictors' seems absurdly reductive.
This shows a deep misunderstanding of the paper's claims, which in no way challenge the established view that these bots are next-token predictors.
Regardless, if all you want is a next-token selector, save your money and roll a die.
> This shows a deep misunderstanding of the paper's claims, which in no way challenge the established view that these bots are next-token predictors.
No, this shows an appreciation of the symbolic richness behind that token 'prediction' which the paper leads on.
> Regardless, if all you want is a next-token selector, save your money and roll a die.
Tell me, where is the emergent symbology guiding that dice?
> No, this shows an appreciation of the symbolic richness behind that token 'prediction' which the paper leads on.
The paper claims no symbolic richness beyond that evident from the undisputed next-token prediction.
> Tell me, where is the emergent symbology guiding that dice?
There's none. That's my point.
> which in no way challenge the established view that these bots are next-token predictors.
I mean, of course they are, that's literally what the inference loop does. You can look at the source of your favorite model runner and you'll see exactly that.
What I find misleading about this term is that it focuses attention on the "next token" part and glosses over the "prediction" part as some sort of unspecified "statistical algorithm" - even though this is where most of the work happens and where the interesting questions are.
For anyone like me who finds theory a little dense, you can prepend "quick" to an arxiv link to get a blogpost-style summary
e.g. quickarxiv.org/abs/2608.29530
N.B. quickarxiv.org just redirects to alphaxiv.org, so the following (i.e. change "ar" to "alpha") works just as well:
https://www.alphaxiv.org/abs/2608.29530
Excellent tip, thank you!
Symbolism has tried to strike back repeatedly ever since statistical learning revived with AlexNet. With all the due respect one can have for the names Smolensky and Linzen from the perspective of linguistics, the question about the applicability, generalizability and robustness of the method proposed here should be raised. It seems from section 3.5 of the paper that one cannot be so optimistic about it at least as yet. I get it that the method is still in its infancy, but we've already got the kind of Mech Interp as pushed forward by Neel Nanda and co, among other lines of research. Not that we are forced to make a choice between all interpretability works, or this TPR method is inherently inferior to the other ones, but we can be moderately cautious when looking at such progress.
I wonder if this paper has been peer-reviewed at a decent conference/journal.
this is an obvious result. for example, this guy has been writing on substack about this for at least a year or two (with code snippets) explaining the phenomenon of grokking and the ghostbasin.com concept - https://richardaragon.substack.com/
their algorithm is even named "DISCOVER" so they set out to discover the connective tissue of why the universe has invariants like math, and lo it was discovered.
i guess good job for having credentials & publishing the math so people 2years behind the curve can learn from your tenure?
yes. large matrices can gradient descend to understand arbitrary symbolic logic.
ENGLISH IS INSUFFICIENT but it is at least a few decades of math proofs & progress :) welcome to the future Slackernews
Post the actual articles that you have in mind. What I've seen is vague slop.
A good example is https://richardaragon.substack.com/p/a-universal-prime-funct... describing a supposed "universal prime function" which is simply a finite approximation using a sum of 50 sines (each applied to a linear term plus a sine-log offset). The 53 parameters are fitted to the first 10^3 or so prime numbers. This is followed by the *absolutely ridiculous* claim that if the function approximates the first 10^3 primes well, it must also fit the remaining prime numbers (of which there are infinitely more than 10^3000000000) equally well.
Then they suggest "A formal proof connecting this function to the RH would involve the following steps" using this great discovery: "1. Correspondence with the Explicit Formula: Demonstrate that the oscillatory correction term in our function corresponds to the sum over zeta zeros in the explicit formula for ψ(x) or π(x). 2. Error Bound: Prove that the error in the prime counting function derived from our function is bounded by O(√x log x). 3. Contradiction: Show that if any non-trivial zero were to lie off the critical line ℜ(s) = 1/2, the error would exceed the bound, leading to a contradiction."
This isn't even midwit math.
It's the kind of naive ideas I had as a high schooler, who was good at high school math and who knew how to code functions and plots in Mathematica, but who had no understanding of higher math. This kind of naive approach to RH signals that one doesn't even understand the problem.
thanks for engaging but tbh just post your own articles. i actually talked to Richard after reading quite in depth and reproducing his code results myself, and do think he's brilliant. my point is he gets the punch line of these math academia papers, much faster.
that's all. easy to call things "vague slop", hard to be right first. Jensen's def'n of intelligence "can you see around corners and predict the future?" kinda all that matters at this point.
just an fyi. congrats on your nitpicking of what math counts as math lol
here's my own paper: https://computerfuture.substack.com/p/demoting-laplaces-demo... (pdf - https://drive.google.com/file/d/1s4VER7SNjUFuCBodX_0myXRpNCW... ) and here's my MIT degree on the blockchain somewhere: https://trattner.github.io/img/18c-diploma.png
feedback welcome from nits... you made zero comments on ghostbasin.com which makes some extraordinary claims too.
lol you also have 107 karma on hackernews while i have 200+ so how's that for math? lol
"Vectors seem inadequate for capturing the structure of language, logic, and other cognitive domains, yet neural networks achieve impressive performance in these areas". Missing the forest for the trees? Aren't neural networks modeled after biological systems? Our brains are obviously able to contain symbolic structure despite not having a "symbol processing unit".
People really overstate the relationship between ANNs and the brain, they have very different mechanisms and only have a similarity if you squint at 100000 feet. ANNs don't have neurotransmitters or even action potentials.
But neurotransmitters and action potentials don't help in modeling symbolic structure.
That is, yes, ANNs are not brains. There are countless differences. But are there differences at the computational level? ANNs are meant to model brain computation, not brain biology.
(There is still a lot to debate there, I'm not saying "ANNs are perfect computational models for the brain")
>But neurotransmitters and action potentials don't help in modeling symbolic structure.
This is exactly why Fodor argues that psychology should be explained on its on level with symbols rather than appealing to neurology. But if you're interested in modeling symbols, there's much better options than ANNs (see nearly any programming language ever).
>ANNs are meant to model brain computation
But we don't really know how that works! So if you know if you're not modelling the low-level behaviour right, you can't assume that there's a correspondence of the higher level computation when you don't really know what that higher level computations are.
We don't know all the details about how the brain computes, you are right.
But we do have a hypothesis: that it is done by a large number of simple units with very high connectivity and in deep layers. This is what neural networks model.
Personally I was skeptical of this model of the brain, but they have achieved remarkable success in practice, as well as Nobel prizes. The neural networks people may have been onto something all along (I say that grudgingly).
But the models learn in a very different way than humans, they're certainly vastly more sample inefficient. I'm not saying it isn't impressive or that it isn't necessarily a kind of intelligence, I'm just sceptical as to how much it really tells us about the brain.
Those are fair points.
While I agree with this statement, one could argue that what's important is not much the support but the emergent properties - in the same way that a wave is still a wave whether it is in the water or in the air.
Thus putting things similar to neurons in a network and making them able to learn could create behaviors similar to the brain. The fact that that biology used chemistry + electrical signal and computers use ReLU-like activation could be merely choosing the most efficient way to enable training.
Correct. Worst of all, even if you do build a spiking neural network, the update rule is kind of a mystery. To have a good update rule, a biological neuron needs to be kind of like a tiny computer in its own right. You might be able to model synapses as weights between neurons, but the neuron carries further internal states within in itself and how the "update rule" uses those internal states is not known at all.
I hate that whole intro - the first four sentences - so much. It’s nothing but unsupported assumptions. Basically, a strawman that they can do battle with in the paper. Not an auspicious start.
These aren't really strawmen, they're more or less than mainstream opinion in the cognitive sciences from the 80s to maybe 2015-2020 or so.
Paul Smolensky is a cognitive science titan from that era. He worked with Hinton, Rumelhart, and McClelland on parallel distributed processing, and literally wrote the book on tensor product representations in cognition, with Geraldine Legendre: https://mitpress.mit.edu/9780262516198/the-harmonic-mind-vol...
He's the axis of this particular group of researchers, being the most senior at the place where they all met, Johns Hopkins.
So this is less a straw man and more a quick reminder to his peers: "Right, so, remember this particular thread we've spent the last 40 years hashing out, here we've got another contribution to that particular conversation."
If it was supposed to be sarcasm, that does help.
I’m aware, but it was a bad mainstream opinion which was contradicted by every bit of evidence ever since early LISP efforts in the late 1960s (still waiting for the release of SHRDLU 2.0!)
65 years is a long time to be beating your head against an obvious dead end.
The article intro presents the claims as self-evident, when they’re not at all.
But, someone rise pointed out this may have been a dig at those attitudes, which makes more sense.
got to top 2 HN tho lol
-1 karma jesus guess i should never make meta commentary lol
yep
It's like Neo says "You get used to it, though. Your brain does the translating. I don't even see the code." He was referring to something like a K, Q, V vector at the time I believe.
Cypher says that, and he's clearly referring to a blonde, a brunette, and a redhead.
AND then they walk by the frame, next shot.-
lol
From the abstract "Further, our symbolic approximation allows us to modify an LLM's behavior in targeted ways via precise interventions on its internal representations [...]".
If this is true and easily computable, this might have big impact in AI safety, as it seems to be really lacking today.