This is a really interesting article! While I can’t profess to understand all the math it is a great way to demonstrate it.
I recall awhile back hearing someone ask whether intelligence was actually the same or similar to “compression” or storing all of this knowledge in weights - IE the ability to create something from reason or weights or relationships / connections. This article makes it even more interesting!
This article does two things which make it unhelpful for me:
1. It uses math notation. This is a software development site, wouldn't this be much more clear to this audience written as psuedo code instead of a complex series of greek and latin letters?
2. It doesn't step through a concrete example. What is hollow R raised to the (M * d_v) ? Is it a number, a set, a list, a matrix?
To use an example I understand: RAG used to seem complicated to me when only math people were talking about it in math words. For example, let's take a line from wikipedia on embeddings:
> the representation is a real-valued vector that encodes the meaning of the word in such a way that the words that are closer in the vector space are expected to be similar in meaning
What made it clear was learning that generating an embedding is just a function(string) -> list[float], and RAG is just taking a list[float] for your prompt and finding other list[float] that are close to it in your dataset.
So what would make this article more clear, is step through a simplified example of putting a value into a memory store and querying that memory store, showing the actual data structures.
> This is a software development site, wouldn't this be much more clear to this audience written as psuedo code instead of a complex series of greek and latin letters
At the risk of stating the obvious, this is a forum where people can post arbitrary links to other sites, some of which might have audiences that might not be "the exact set of people who read Hacker News". I'm not sure you'd be able to make a compelling argument that alexwang.ai is aimed at software developers and no one else.
Yes, but you need to consider your audience when posting links to this site. For example, if I posted to HN a link to a really good cookie recipe, it wouldn't be a good fit for this site. The author did nothing wrong, but that doesn't make it a good submission to HN.
This article is a bit more of a grey area. It is on a subject that is deeply relevant to this site and I'm sure some HN readers will get value from it. My point is that an even better submission to HN would have been an article on the same topic but written for programmers rather than mathematicians.
I'm surprised to this take, because I feel like I see math stuff posted here fairly often, and often with a large number of people commenting on it in substantive ways. Often times the math savvy crowd will even help explain stuff to people like me who are not particularly fluent in advanced math.
> Often times the math savvy crowd will even help explain stuff to people
Perhaps, but HN apparently is pretty hostile to people explaining why something is not clear and how it could be more clear based on all the downvotes. I'll keep that in mind: "If you don't understand something, shut up!"
Expect to be challenged by content here. Everybody is from time to time. Asking for a mathy analysis to drop the math, fundamentally misunderstands the value of the math.
I think people likely hit an early "break" and "goto xkcd 386" when they saw "this is a software development site" in your comment, without reading the rest of your comment.
Clear explanations are a good thing, but different explanations help people in different ways. Your explanation seems helpful to me, and reminds me of Ben Affleck's explanation in the GQ interview. But Ben Affleck connected it directly to arrays of RGB pixel data, which clicks more with me personally, but probably wouldn't with everyone. It certainly didn't make things any clearer to the interviewer.
I didn't downvote, and I don't think there's a problem with explaining why something isn't clear. There's a difference between expressing criticism and saying that something doesn't belong here, and which I'd argue is evidenced from the fact that the reaction to my comment critiquing yours doesn't seem to have ruffled many feathers (at least not yet!)
Would be nice if articles defined the requirements to be a “you”.
See also:
NNCP: Lossless Data Compression with Neural Networks (2019)
https://news.ycombinator.com/item?id=27244004
https://bellard.org/nncp/
This is a really interesting article! While I can’t profess to understand all the math it is a great way to demonstrate it.
I recall awhile back hearing someone ask whether intelligence was actually the same or similar to “compression” or storing all of this knowledge in weights - IE the ability to create something from reason or weights or relationships / connections. This article makes it even more interesting!
Who? Me?
This article does two things which make it unhelpful for me:
1. It uses math notation. This is a software development site, wouldn't this be much more clear to this audience written as psuedo code instead of a complex series of greek and latin letters?
2. It doesn't step through a concrete example. What is hollow R raised to the (M * d_v) ? Is it a number, a set, a list, a matrix?
To use an example I understand: RAG used to seem complicated to me when only math people were talking about it in math words. For example, let's take a line from wikipedia on embeddings:
> the representation is a real-valued vector that encodes the meaning of the word in such a way that the words that are closer in the vector space are expected to be similar in meaning
What made it clear was learning that generating an embedding is just a function(string) -> list[float], and RAG is just taking a list[float] for your prompt and finding other list[float] that are close to it in your dataset.
So what would make this article more clear, is step through a simplified example of putting a value into a memory store and querying that memory store, showing the actual data structures.
> This is a software development site, wouldn't this be much more clear to this audience written as psuedo code instead of a complex series of greek and latin letters
At the risk of stating the obvious, this is a forum where people can post arbitrary links to other sites, some of which might have audiences that might not be "the exact set of people who read Hacker News". I'm not sure you'd be able to make a compelling argument that alexwang.ai is aimed at software developers and no one else.
Yes, but you need to consider your audience when posting links to this site. For example, if I posted to HN a link to a really good cookie recipe, it wouldn't be a good fit for this site. The author did nothing wrong, but that doesn't make it a good submission to HN.
This article is a bit more of a grey area. It is on a subject that is deeply relevant to this site and I'm sure some HN readers will get value from it. My point is that an even better submission to HN would have been an article on the same topic but written for programmers rather than mathematicians.
I'm surprised to this take, because I feel like I see math stuff posted here fairly often, and often with a large number of people commenting on it in substantive ways. Often times the math savvy crowd will even help explain stuff to people like me who are not particularly fluent in advanced math.
> Often times the math savvy crowd will even help explain stuff to people
Perhaps, but HN apparently is pretty hostile to people explaining why something is not clear and how it could be more clear based on all the downvotes. I'll keep that in mind: "If you don't understand something, shut up!"
Not understanding something isn’t the issue.
Also not an issue: the quality of the article.
Expect to be challenged by content here. Everybody is from time to time. Asking for a mathy analysis to drop the math, fundamentally misunderstands the value of the math.
I think people likely hit an early "break" and "goto xkcd 386" when they saw "this is a software development site" in your comment, without reading the rest of your comment.
Clear explanations are a good thing, but different explanations help people in different ways. Your explanation seems helpful to me, and reminds me of Ben Affleck's explanation in the GQ interview. But Ben Affleck connected it directly to arrays of RGB pixel data, which clicks more with me personally, but probably wouldn't with everyone. It certainly didn't make things any clearer to the interviewer.
There's a certain irony in praising clear explanations as a defense of an explanation that needlessly veered into value judgment
I didn't downvote, and I don't think there's a problem with explaining why something isn't clear. There's a difference between expressing criticism and saying that something doesn't belong here, and which I'd argue is evidenced from the fact that the reaction to my comment critiquing yours doesn't seem to have ruffled many feathers (at least not yet!)
Send the link to Claude or your favorite alternative, and cut-and-paste your comment into the box, and you'll get exactly what you want.