That's pretty innovative. But a brand new tokenizer and training method need a lot of effort to do the retraining. Whether it will be widely implemented depends on real frontier performance it can reach.
Exactly. I used to think new tokenizers for domain-specific models was the way (makes intuitive sense, better representations etc), but the added overhead of essentially training from scratch made it much more expensive than a FT.
If we cant' build the hardware fast enough. Can someone explain to me why there's apparently 50% of the gpu and memory that hyperscalers buy sitting in warehouses due to lack of data centers? The hyperscalers are hoarding tech and can't even use it. It's ridiculous.
As far as I can tell, because that is not happening.
The suggestion seems to have come from the claim that to maintain a priority queue, you have to be buying consistently. That might mean buying more than you need to ensure supply when you need it. The assumption that the excess goes into a warehouse seems to have come from someone's fevered imagination.
If there are GPUs built and not yet installed, and people who want to be using them now, money can be made getting the supply to find the demand. Most large scale companies know how to do that sort of thing. If someone has a data center and no GPUs arriving for six months, paying a few million to trade the GPUs that will exist in six months for the ones that exist now gets them an additional six months of operation, while ensuring the original buyer will have GPUs when they need them, plus a few million.
Yeah. Don't know anything about data center GPUs, but it's common knowledge you can get like 80% of the performance of a gaming GPU at half the power or even less. That's basically how Nvidia's mobile GPUs are. So if you had a warehouse full of GPUs, you could just go ahead and install them, run them at low power, and still get like 2/3rds of the performance.
There are reports and rumours that NVIDIA are forcing the data centre companies to buy the GPUs upfront. Though there isn't a lot of clear evidence that this is the case.
That depends on how long it’s sitting there, given they are two different things to manage it makes sense that you’d either have gpus sitting unused while a dc gets built or a dc sitting empty while you wait for gpus.
Because slowing down compute purchase is collapsing the hype train which is the same as death. Vibes are a wave that must be maintained or crash down onto reality.
Also, tinfoil alert, if they buy it means you can't (or are eventually priced out of the hw market). Their only "moat" (if you can call it that..) is the price of local inference capacity, which is still kinda prohibitive for the 99%.
I hope Apple Silicon will one day be competitive with GPUs. It dominates much of the CPU market, and it has a GPU, but it's still not even remotely comparable to Nvidia for inference (factor of >5 in some cases, especially in prefill) or even for gaming ("edge computing" in Nvidia terms).
I feel like their GPU architecture is going to need a significant redesign. There are limits to this tile-based mobile architecture.
> Fastest consumer Apple silicon versus fastest consumer Nvidia. So M5 Ultra vs RTX 5090. Not comparing price or efficiency or value or anything like that. Just the platforms as a whole. So M5 Ultra vs RTX 5090
The problem here is that they aren't equivalent platforms - no amount of money will buy you an RTX 5090 with 512 GB of RAM.
If you want to buy the 16x RTX 5090s you need to get to the same 512 GB of RAM, we're talking $100,000 just in GPUs, industrial levels of power/cooling/noise, and the interconnect bandwidth is still going to kill it for normal inferencing workloads.
Of course, that 16x cluster is an absolute beast in terms of raw compute - it's just not all that useful for the things people are trying to do with such machines today.
> The problem here is that they aren't equivalent platforms - no amount of money will buy you an RTX 5090 with 512 GB of RAM.
I agree that Apple silicon is far better in terms of memory capacity, which is part of why I'm rooting for it. But an SoC like Apple silicon is roughly analogous to a CPU+GPU when it comes to compute, and the GPU side is quite lacking compared to today's "traditional" GPUs, even though the CPU is somewhat industry leading.
Mainly, I'm disappointed with the performance of dense LLMs on Apple silicon compared to Nvidia, due to the large disparity in both memory bandwidth and GPU compute. Apple's latest chips are starting to push into just over a terabyte per second of memory bandwidth, about two-thirds of what the 5090 achieved nearly two years ago now. They are catching up quick and I do hope they pass, but they haven't quite yet.
I also hear that Apple silicon is pretty bad at prefill compared to Nvidia even when memory bandwidth is not an issue, but haven't verified. It would make sense though due to the much greater compute.
This looks very interesting. Their new language modelling objective shows some large improvements over masked diffusion modeling.
However, they hardly compare against the conventional next-token-prediction objective, which suggests they don't beat it, except for highly non-linear tasks like sudoku. Unfortunately, machine learning papers often tend to focus exclusively on the positives without mentioning bad aspects.
https://arxiv.org/abs/2602.06036 - they use a faster (single-pass) diffusion model to generate predictions for 7 to 15 tokens in a single step. Then if they're accepted that really speeds up the autoregressive model because it doesn't have to generate tokens one at a time.
I keep getting clickbait youtube videos about how "NCM" obsoletes "LLMs" and "Nvidia memory chips" (yes, it's that bad).
This makes me wonder what these people think an "LLM" even is. The name itself implies that it is large and about language. It does not imply that an "LLM" is inherently autoregressive or that it has any particular architecture even though most LLMs are transformers. Anything can be a large language model as long as it is large and models language.
NCM is basically a hierarchical diffusion transformer where the further layers take one big token and then generate two smaller tokens. So it is more like an evaluation strategy. You could apply this to any model architecture.
I personally think the name NCM itself is terrible because it contains the word "next" which inherently implies an ordering and somehow we are supposed to think that "LLM" which contains no mention of ordering, is constrained to ordered token production.
I'm not sure why but using X instead of the word "times" annoys me. Probably because one of the first things I got taught at university is that mathematical notation is not prose and shouldn't be used as such.
Different cultures have different norms and rules. I think you're saying that, in your native language, they taught you not to use mathematical notation in prose.
They didn't teach you that about English, though. It's quite common to write "21x" for "21 times" in English and is pretty much universally understood and accepted by native speakers.
It's a more general rule about when and where to use mathematical notation. Though style rules do differ in places.
For what it's worth I've not seen "12x" in formal contexts, but if that is proper then the times symbol still shouldn't be styled as part of the maths formula imho.
The Oxford comma is the comma before the conjunction in a list of three or more terms, e.g. "one, two, and three". I think your sentence actually didn't need a comma anway.
Technically yes, but quoting mathematical constants is definitely allowed. You could also say O(N) times for instance. The kind of language that is discouraged is writing "therefore efficiency increases by O(log(N)) times" as "∴ efficiency increases by O(log(N)) *".
For one it simply looks like you forgot to finish the formula.
That's pretty innovative. But a brand new tokenizer and training method need a lot of effort to do the retraining. Whether it will be widely implemented depends on real frontier performance it can reach.
Exactly. I used to think new tokenizers for domain-specific models was the way (makes intuitive sense, better representations etc), but the added overhead of essentially training from scratch made it much more expensive than a FT.
If we cant' build the hardware fast enough. Can someone explain to me why there's apparently 50% of the gpu and memory that hyperscalers buy sitting in warehouses due to lack of data centers? The hyperscalers are hoarding tech and can't even use it. It's ridiculous.
As far as I can tell, because that is not happening.
The suggestion seems to have come from the claim that to maintain a priority queue, you have to be buying consistently. That might mean buying more than you need to ensure supply when you need it. The assumption that the excess goes into a warehouse seems to have come from someone's fevered imagination.
If there are GPUs built and not yet installed, and people who want to be using them now, money can be made getting the supply to find the demand. Most large scale companies know how to do that sort of thing. If someone has a data center and no GPUs arriving for six months, paying a few million to trade the GPUs that will exist in six months for the ones that exist now gets them an additional six months of operation, while ensuring the original buyer will have GPUs when they need them, plus a few million.
Yeah. Don't know anything about data center GPUs, but it's common knowledge you can get like 80% of the performance of a gaming GPU at half the power or even less. That's basically how Nvidia's mobile GPUs are. So if you had a warehouse full of GPUs, you could just go ahead and install them, run them at low power, and still get like 2/3rds of the performance.
There are reports and rumours that NVIDIA are forcing the data centre companies to buy the GPUs upfront. Though there isn't a lot of clear evidence that this is the case.
[1] https://tech.yahoo.com/gaming/articles/did-nvidia-manufactur...
[2] https://www.reddit.com/r/Games/comments/1xdjbd1/multiple_rep...
[3] https://www.techradar.com/computing/gpu/is-the-gpu-stock-cri...
[4] https://www.businessinsider.com/amazon-nvidia-ai-gpu-data-ce...
I dont understand how this comment connects to the linked research paper. Anyone?
That depends on how long it’s sitting there, given they are two different things to manage it makes sense that you’d either have gpus sitting unused while a dc gets built or a dc sitting empty while you wait for gpus.
They could probably rush them to production, but it's added logistics and would lead to the parts degrading faster.
Lack of customers*. SpaceXaitwitter has some big beautiful idle datacenters with all GPUs ready to use.
Because slowing down compute purchase is collapsing the hype train which is the same as death. Vibes are a wave that must be maintained or crash down onto reality.
Also, tinfoil alert, if they buy it means you can't (or are eventually priced out of the hw market). Their only "moat" (if you can call it that..) is the price of local inference capacity, which is still kinda prohibitive for the 99%.
Who's going to train a nice big coding model based on this? I don't have the GPUs to do it...
I hope Apple Silicon will one day be competitive with GPUs. It dominates much of the CPU market, and it has a GPU, but it's still not even remotely comparable to Nvidia for inference (factor of >5 in some cases, especially in prefill) or even for gaming ("edge computing" in Nvidia terms).
I feel like their GPU architecture is going to need a significant redesign. There are limits to this tile-based mobile architecture.
Depends what you are comparing. Apple Macbook Pro M5Max out performs NVIDIA RTX N1X laptops for about the same cost.
Fastest consumer Apple silicon versus fastest consumer Nvidia. So M5 Ultra vs RTX 5090.
Not comparing price or efficiency or value or anything like that. Just the platforms as a whole.
> Fastest consumer Apple silicon versus fastest consumer Nvidia. So M5 Ultra vs RTX 5090. Not comparing price or efficiency or value or anything like that. Just the platforms as a whole. So M5 Ultra vs RTX 5090
The problem here is that they aren't equivalent platforms - no amount of money will buy you an RTX 5090 with 512 GB of RAM.
If you want to buy the 16x RTX 5090s you need to get to the same 512 GB of RAM, we're talking $100,000 just in GPUs, industrial levels of power/cooling/noise, and the interconnect bandwidth is still going to kill it for normal inferencing workloads.
Of course, that 16x cluster is an absolute beast in terms of raw compute - it's just not all that useful for the things people are trying to do with such machines today.
> The problem here is that they aren't equivalent platforms - no amount of money will buy you an RTX 5090 with 512 GB of RAM.
I agree that Apple silicon is far better in terms of memory capacity, which is part of why I'm rooting for it. But an SoC like Apple silicon is roughly analogous to a CPU+GPU when it comes to compute, and the GPU side is quite lacking compared to today's "traditional" GPUs, even though the CPU is somewhat industry leading.
Mainly, I'm disappointed with the performance of dense LLMs on Apple silicon compared to Nvidia, due to the large disparity in both memory bandwidth and GPU compute. Apple's latest chips are starting to push into just over a terabyte per second of memory bandwidth, about two-thirds of what the 5090 achieved nearly two years ago now. They are catching up quick and I do hope they pass, but they haven't quite yet.
I also hear that Apple silicon is pretty bad at prefill compared to Nvidia even when memory bandwidth is not an issue, but haven't verified. It would make sense though due to the much greater compute.
I'm trying to run it, so far no luck
This looks very interesting. Their new language modelling objective shows some large improvements over masked diffusion modeling.
However, they hardly compare against the conventional next-token-prediction objective, which suggests they don't beat it, except for highly non-linear tasks like sudoku. Unfortunately, machine learning papers often tend to focus exclusively on the positives without mentioning bad aspects.
Maybe if it's worse than next-token-prediction it could still be used as a drafting model, similar to how DFlash works.
What's that?
https://arxiv.org/abs/2602.06036 - they use a faster (single-pass) diffusion model to generate predictions for 7 to 15 tokens in a single step. Then if they're accepted that really speeds up the autoregressive model because it doesn't have to generate tokens one at a time.
https://www.youtube.com/watch?v=nv5_PH2uPVI
I keep getting clickbait youtube videos about how "NCM" obsoletes "LLMs" and "Nvidia memory chips" (yes, it's that bad).
This makes me wonder what these people think an "LLM" even is. The name itself implies that it is large and about language. It does not imply that an "LLM" is inherently autoregressive or that it has any particular architecture even though most LLMs are transformers. Anything can be a large language model as long as it is large and models language.
NCM is basically a hierarchical diffusion transformer where the further layers take one big token and then generate two smaller tokens. So it is more like an evaluation strategy. You could apply this to any model architecture.
I personally think the name NCM itself is terrible because it contains the word "next" which inherently implies an ordering and somehow we are supposed to think that "LLM" which contains no mention of ordering, is constrained to ordered token production.
I think they think an LLM is just whatever another LLM tells them it is when they ask it to generate a script.
Hey, me too.
Can we apply any insights from image and video compression and partitioning to this kind of dimensional masking ?
I'm not sure why but using X instead of the word "times" annoys me. Probably because one of the first things I got taught at university is that mathematical notation is not prose and shouldn't be used as such.
But what about “got taught” instead of “was taught” and the missing comma before “but”?
Well they didn't teach me English. Or in English for that matter.
And I don't think I've ever used the Oxford comma.
There was probably some rule about not beginning sentences with Or and And but you probably know better than I.
Different cultures have different norms and rules. I think you're saying that, in your native language, they taught you not to use mathematical notation in prose.
They didn't teach you that about English, though. It's quite common to write "21x" for "21 times" in English and is pretty much universally understood and accepted by native speakers.
It's a more general rule about when and where to use mathematical notation. Though style rules do differ in places.
For what it's worth I've not seen "12x" in formal contexts, but if that is proper then the times symbol still shouldn't be styled as part of the maths formula imho.
The first time I heard anyone pronounce 2X as "two ex" instead of "two times" was Steve Jobs in one of his presentation.
The Oxford comma is the comma before the conjunction in a list of three or more terms, e.g. "one, two, and three". I think your sentence actually didn't need a comma anway.
I wonder whether "10" in "10×" also counts as mathematical notation. Or if just "ten times" is allowed.
Technically yes, but quoting mathematical constants is definitely allowed. You could also say O(N) times for instance. The kind of language that is discouraged is writing "therefore efficiency increases by O(log(N)) times" as "∴ efficiency increases by O(log(N)) *".
For one it simply looks like you forgot to finish the formula.
I particularly dislike Greek letters, because I don't know how to pronounce them in my head. For "×" you at least know it's supposed to read "times".