Google's real moat isn't the TPU silicon itself—it's not about cooling, individual performance, or hyper-specialization—but rather the massive parallel scale enabled by their OCS interconnects.
To quote The Next Platform: "An Ironwood cluster linked with Google’s absolutely unique optical circuit switch interconnect can bring to bear 9,216 Ironwood TPUs with a combined 1.77 PB of HBM memory... This makes a rackscale Nvidia system based on 144 “Blackwell” GPU chiplets with an aggregate of 20.7 TB of HBM memory look like a joke."
Nvidia may have the superior architecture at the single-chip level, but for large-scale distributed training (and inference) they currently have nothing that rivals Google's optical switching scalability.
Also, Google owns the entire vertical stack, which is what most people need. It can provide an entire spectrum of AI services far cheaper, at scale (and still profitable) via its cloud. Not every company needs to buy the hardware and build models, etc., etc.; what most companies need is an app store of AI offerings they can leverage. Google can offer this with a healthy profit margin, while others will eventually run out of money.
If they get things like PyTorch to work well without carinng what hardware it is running on, it erodes Nvidia's CUDA moat. Nvidia's chips are excellent, without doubt, but their real moat is the ecosystem around CUDA.
The problem is that "hardware-agnostic PyTorch" is a myth, much like Java's "write once, run anywhere". At the high level (API), the code looks the same, but as soon as you start optimizing for performance, you inevitably drop down to CUDA. As long as researchers are writing their new algorithms in CUDA because it's the de facto language of science, Google will forever be playing catch-up, having to port these algorithms to XLA. An ecosystem is, after all, people and their habits, not just libraries.
I'd love for someone to give me an alternative to CUDA but I don't primarily use GPUs for inference, I do 64-bit unsigned integer workloads and the only people who seem to care even a little about this currently are NVidia, if imperfectly.
I _really_ want an alternative but the architecture churn imposed by targeting ROCm for say an MI350X is brutal. The way their wavefronts and everything work is significantly different enough that if you're trying to get last-mile perf (which for GPUs unfortunately yawns back into the 2-5x stretch) you're eating a lot of pain to get the same cost-efficiency out of AMD hardware.
FPGAs aren't really any more cost effective unless the $/kwh goes into the stratosphere which is a hypothetical I don't care to contemplate.
Pytorch, Jax, tensorflow are all examples to me of very capable products, that compete very well in ML space.
But more broadly work like XLA and IREE are very interesting toolkits for mapping a huge variety of computation onto many types of hardware. While Pytorch et al are fine example applications, are things you can do, XLA is the Big Tent idea, the toolkit to erode not just specific CUDA use cases, but to allow hardware in general to be more broadly useful.
They just need to actually make and market a good product though, and they seem to really struggle with this. Maybe on a long enough timeline their advantages will make this one inevitable.
That is comparing an all to all switched Nvlink fabric to a 3D torus for TPUs. Those are completely different network topologies with different tradeoffs.
For example the currently very popular Mixture of Experts architectures require a lot of all to all traffic (for expert parallelism) which works a lot better on the switched NVlink fabric as opposed where it doesn't need to traverse multiple links in the torus.
This is an underrated point. Comparing just the peak bandwidth is like saying Bulldozer was the far superior CPU of the era because it had a really high frequency ceiling.
Really? Fully-connected hardware is in buildable (at scale) which we already know from the HPC world. Fat trees and dragonfly networks are pretty scalable, but a 3d torus is a very good tradeofff, and respects the dimensionality of reality.
Bisection bandwidth is a useful metric, but is hop count? Per-hop cost tends to be pretty small.
Latency (of different types), jitter, and guaranteed bandwidth are the real underlying metrics. Hop count is just one potential driver of those, but different approaches may or may not tackle each of these parts differently.
NVFP4 is the thing no one saw coming. I wasn't watching the MX process really, so I cast no judgements, but it's exactly what it sounds like, a serious compromise in resource constrained settings. And it's in the silicon pipeline.
NVFP4 is to put it mildly a masterpiece, the UTF-8 of its domain and in strikingly similar ways it is 1. general 2. robust to gross misuse 3. not optional if success and cost both matter.
It's not a gap that can be closed by a process node or an architecture tweak: it's an order of magnitude where the polynomials that were killing you on the way up are now working for you.
sm_120 (what NVIDIA's quiet repos call CTA1) consumer gear does softmax attention and projection/MLP blockscaled GEMM at a bit over a petaflop at 300W and close to two (dense) at 600W.
This changes the whole game and it's not clear anyone outside the lab even knows the new equilibrium points, it's nothing like Flash3 on Hopper, lotta stuff looks FLOPs bound, GDDR7 looks like a better deal than HBMe3. The DGX Spark is in no way deficient, it has ample memory bandwidth.
This has been in the pipe for something like five years and even if everyone else started at the beginning of the year when this was knowable, it would still be 12-18 months until tape out. And they haven't started.
Years Until Anyone Can Compete With NVIDIA is back up to the 2-5 it was 2-5 years ago.
This was supposed to be the year ROCm and the new Intel stuff became viable.
This reads like a badly done, sponsored hype video on YouTube.
So if we look at what NVIDIA has to say about NVFP4 it sure sounds impressive [1]. But look closely that initial graph never compares fp8 and fp4 on the same hardware. They jump from H100 to B200 while implying a 5x jump of going with fp4 which it isn't. Accompanied with scary words like if you use MXFP4 "Risk of noticeable accuracy drop compared to FP8" .
Contrast that with what AMD has to say on the open MXFP4 approach which is quite similar to NVFP4 [2]. Ohh the horrors of getting 79.6 instead of 79.9 on GPQA Diamond when using MXFP4 instead of FP8.
Looking into NVFP4/Nvidia vs MXFP4/AMD the summation was that seem to be pretty close when including the MI355X which leads in VRAM and throughput but trails in accuracy (slightly)--and for that mixing in MXFP6 makes up for it.
Not vibes. TPUs have fallen behind or had to be redesigned from scratch many times as neural architectures and workloads evolved, whereas the more general purpose GPUs kept on trucking and building on their prior investments. There's a good reason so much research is done on Nvidia clusters and not TPU clusters. TPU has often turned out to be over-specialized and Nvidia are pointing that out.
You say that like I d a bad thing. Nvidia architectures keep changing and getting more advanced as well, with specialized tensor operations, different accumulators and caches, etc. I see no issue with progress.
That’s missing the point. Things like tensor cores were added in parallel with improvements to existing computer and CUDA kernels from 10 years ago generally run without modification. Hardware architecture may change, but Nvidia has largely avoided changing how you interact with it.
They don't have to, CUDA is a high-level API in this respect. The hardware will conform to the demands of the market and the software will support whatever the compute capability defines, Nvidia is clearer than most about this.
Sure, but you do have to do it pretty quick. Let’s pick a H100. You’ve probably heard that just writing scalar code is leaving 90+% of the flops idle. But even past that, if you’re using the tensor core but using the wrong instructions you’re basically capped at 300-400 TFLOPS of the 1000 the hardware supports. If using the new instructions but poorly you’re probably not going to hit even 500 TFLOPS. That’s just barely better than the previous generation you paid a bunch of money to replace.
Portable doesn't normally mean that it runs on arbitrarily old hardware. CUDA was never portable, it only runs on Nvidia hardware. The question is whether old versions of Whisper GPU run on newer hardware, that'd be backwards compatibility.
The tweet gives their justification; CUDA isn't ASIC. Nvidia GPUs were popular for crypto mining, protein folding, and now AI inference too. TPUs are tensor ASICs.
FWIW I'm inclined to agree with Nvidia here. Scaling up a systolic array is impressive but nothing new.
Sure, but their company's 4.3 trillion valuation isn't based on how good their GPUs are for general purpose computing, it's based on how good they are at AI.
No, not at all. If this were true Google would be killing it in MLPerf benchmarks, but they are not.
It’s better to have a faster, smaller network for model parallelism and a larger, slower one for data parallelism than a very large, but slower, network for everything. This is why NVIDIA wins.
For all the excitement surrounding this, I fail to comprehend how Google can't even meet the current demand for Gemini 3^. Moreover, they are unwilling to invest in expansion directly (apparently have a mandate to double their compute every 6 months without spending more than their current budget). So, pardon me if I can't see how they will scale operations as demand grows while simultaneously selling their chips to competitors?! This situation doesn't make any sense.
^Even now I get capacity related error messages, so many days after the Gemini 3 launch. Also, Jules is basically unusable. Maybe Gemini 3 is a bigger resource hog than anyone outside of Google realizes.
I also suspect Google is launching models it can’t really sustain in volume or that are operating at a loss. Nothing preventing them from like doubling model size compared to the rest or allocating an insane amount of compute just to make the headlines on model performance (clearly it’s good for the stock). These things are opaque anyway, buried deep into the P&L.
OCS is indeed an engineering marvel, but look at NVIDIA's NVL72. They took a different path: instead of flexible optics, they used the brute force of copper, turning an entire rack into one giant GPU with unified memory. Google is solving the scale-out problem, while NVIDIA is solving the scale-up problem. For LLM training tasks, where communication is the bottleneck, NVIDIA's approach with NVLink might actually prove even more efficient than Google's optical routing.
Check the specs again. Per chip, TPU 7x has 192GB of HBM3e, whereas the NVIDIA B200 has 186GB.
While the B200 wins on raw FP8 throughput (~9000 vs 4614 TFLOPs), that makes sense given NVIDIA has optimized for the single-chip game for over 20 years. But the bottleneck here isn't the chip—it's the domain size.
NVIDIA's top-tier NVL72 tops out at an NVLink domain of 72 Blackwell GPUs. Meanwhile, Google is connecting 9216 chips at 9.6Tbps to deliver nearly 43 ExaFlops. NVIDIA has the ecosystem (CUDA, community, etc.), but until they can match that interconnect scale, they simply don't compete in this weight class.
Correct --- found a remark on Twitter calling this "Jenson Math".
Same logic when NVidia quote the "bidirectional bandwidth" of high speed interconnects to make the numbers look big, instead of the more common BW per direction, forcing everyone else to adopt the same metric in marketing materials.
Wow, no, not at all. It’s better to have a set of smaller, faster cliques connected by a slow network than a slower-than-clique flat network that connects everything. The cliques connected by a slow DCN can scale to arbitrary size. Even Google has had to resort to that for its biggest clusters.
I guess “this weight class” is some theoretical class divorced from any application? Almost all players are running Nvidia other than Google. The other players are certainly more than just competing with Google.
> Almost all players are running Nvidia other than Google.
No surprises there, Google is not the greatest company at productizing their tech for external consumption.
> The other players are certainly more than just competing with Google.
TBF, its easy to stay in the game when you're flush with cash, and for the past N-quarters, investors have been throwing money at AI companies, Nvidia's margins have greatly benefited from this largesse. There will be blood on the floor once investors start demanding returns to their investments.
Ok? The person I was replying to was saying that Google’s compute offering is substantially superior to Nvidia’s. What do your comments about market positioning have to do with that?
If Google’s TPUs were really substantially superior, don’t you think that would result in at least short term market advantages for Gemini? Where are they?
They are suggesting it is easier for others to buy buy more NVidia chips and feed them more power. Whilst operating costs are covered by investors. If they move on to competing on having to do inference the cheepest then the TPUs will shine.
The original post made no comments about inference or training or even cost in any way. It said you could hook up more TPUs together with more memory and higher average bandwidth than you could with a datacenter of Nvidia GPUs. From an architectural point of view, it isn’t clear (and is not explained) what that enables. It clearly hasn’t led to a business outcome for Google where they are the clear market leader.
Seemingly fast interconnects benefit training more than inference since training can have more parallel communication between nodes. Inference for users is more embarrassingly parallel (requires less communication) than updating and merging network weights.
My point: cool benchmark, what does it matter? The original post says Nvidia doesn’t have anything to compete with massively interconnected TPUs. It didn’t merely say Google’s TPUs were better. It said that Nvidia can’t compete. That’s clearly bullshit and wishful thinking, right? There is no evidence in the market to support that, and no actual technical points have been presented in this thread either. OpenAI, Anthropic, etc are certainly competing with Google, right?
And then people explained why the effects are smoothed over right now but will matter eventually and you rejected them as if they didn't understand your question. They answered it, take the answer.
> It didn’t merely say Google’s TPUs were better. It said that Nvidia can’t compete.
Can't compete at clusters of a certain size. The argument is that anyone on nVidia simply isn't building clusters that big.
Catch-up in what exactly? Google isn't building hardware to sell, they aren't in the same market.
Also I feel you completely misunderstand that the problem isn't how fast is ONE gpu vs ONE tpu, what matters is the costs for the same output. If I can fill a datacenter at half the cost for the same output, does it matters I've used twice the TPUs and that a single Nvidia Blackwell was faster? No...
And hardware cost isn't even the biggest problem, operational costs, mostly power and cooling are another huge one.
So if you design a solution that fits your stack (designed for it) and optimize for your operational costs you're light years ahead of your competition using the more powerful solution, that costs 5 times more in hardware and twice in operational costs.
All I say is more or less true for inference economics, have no clue about training.
You're doing operations on the memory once it's been transferred to gpu memory. Either shuffling it around various caches or processors or feeding it into tensor cores or other matrix operations. You don't want to be sitting idle.
I always enjoy being wrong and I was very wrong in my predictions about Google : I thought they should theoretically win, but I was also very confident they couldn't possibly turn their execution ship around to actually pull together a coherent competitor to OpenAI. But they do seem to have done that and it's very impressive. If they do continue to execute, I can't see anybody stopping them dominating and I would be bearish on nearly every other player catching them.
The biggest problem though is trust, and I'm still holding back from letting anyone under my authority in my org use Gemini because of the lack of any clear or reasonable statement or guidelines on how they use your data. I think it won't matter in the end if they execute their way to domination - but it's going to give everyone else a chance at least for a while.
The LLM provider I trust the most right now is AWS. Anybody else seems to have very conflicted purposes when it comes to sending them my data and interactions.
You're not wrong... but any space where Amazon, of all companies, has a shot at being the "most trustworthy player" is one I'm going to avoid where I can.
They make Amazon Q, but they don't make a general purpose LLM, which is precisely why I fid them more trustworthy. They run others' LLMs [1] , but don't have an interest in using customer data to improve the next generation of their LLM, or move into adjacent businesses to commercialize LLMs. At least not yet.
I don’t know if Google will win but the message they sent is that Nvidia will not: if Google can release the best model without using their own chips instead of GPUs, and save money in the process, then others (Microsoft, Amazon, etc.) will do it too sooner or later.
That is the opposite of good. If only big companies can afford bespoke hardware, they literally become a monopoly, At home LLMs and other usages become either prohibitively expensive in purchase and running cost or you have to deal with a low quality model. Just admit that it's environmentally fucked and not open in any sense. This is not something that the community at large should strive for.
It doesn’t help when their thankgiving doodle that sends me to Gemini on how to plan making thanksgiving dinner on time completely fails in ridiculous ways
> It is also important to note that, until recently, the GenAI industry’s focus has largely been on training workloads. In training workloads, CUDA is very important, but when it comes to inference, even reasoning inference, CUDA is not that important, so the chances of expanding the TPU footprint in inference are much higher than those in training (although TPUs do really well in training as well – Gemini 3 the prime example).
Does anyone have a sense of why CUDA is more important for training than inference?
NVIDIA chips are more versatile. During training, you might need to schedule things to the SFU(Special Function unit that does sin, cos, 1/sqrt(x), etc), you might need to run epilogues, save intermediary computations, save gradients, etc. When you train, you might need to collect data from various GPUs, so you need to support interconnects, remote SMEM writing, etc.
Once you have trained, you have frozen weights/feed-forward networks that consist out of frozen weights that you can just program in and run data over. These weights can be duplicated across any amount of devices and just sit there and run inference with new data.
If this turns out to be the future use-case for NNs(it is today), then Google are better set.
Won't the need to train increase as the need for specialized, smaller models increases and we need to train their many variations? Also what about models that continuously learn/(re)train? Seems to me the need for training will only go up in the future.
This is a very important point - the market for training chips might be a bubble, but the market for inference is much, much larger. At some point we might have good enough models and the need for new frontier models will cool down. The big power-hungry datacenters we are seeing are mostly geared towards training, while inference-only systems are much simpler and power efficient.
A real shame, BTW, all that silicon doesn't do FP32 (very well). After training ceases to be that needed, we could use all that number crunching for climate models and weather prediction.
it's already the case that people are eeking out most further gains through layering "reasoning" on top of what existing models can do - in other words, using massive amounts of inference to substitute for increases model performance. Whereever things plateau I expect this will still be the case - so inference ultimately will always be the end game market.
It's just more common as a legacy artifact from when nvidia was basically the only option available. Many shops are designing models and functions, and then training and iterating on nvidia hardware, but once you have a trained model it's largely fungible. See how Anthropic moved their models from nvidia hardware to Inferentia to XLA on Google TPUs.
Further it's worth noting that the Ironwood, Google's v7 TPU, supports only up to BF16 (a 16-bit floating point that has the range of FP32 minus the precision. Many training processes rely upon larger types, quantizing later, so this breaks a lot of assumptions. Yet Google surprised and actually training Gemini 3 with just that type, so I think a lot of people are reconsidering assumptions.
This is not the case for LLMs. FP16/BF16 training precision is standard, with FP8 inference very common. But labs are moving to FP8 training and even FP4.
When training a neural network, you usually play around with the architecture and need as much flexibility as possible. You need to support a large set of operations.
Another factor is that training is always done with batches. Inference batching depends on the number of concurrent users. This means training tends to be compute bound where supporting the latest data types is critical, whereas inference speeds are often bottlenecked by memory which does not lend itself to product differentiation. If you put the same memory into your chip as your competitor, the difference is going to be way smaller.
Training is taking an enormous problem and trying to break it into lots of pieces and managing the data dependency between those pieces. It's solving 1 really hard problem. Inference is the opposite, it's lots of small independent problems. All of this "we have X many widgets connected to Y many high bandwidth optical telescopes" is all a training problem that they need to solve. Inference is "I have 20 tokens and I want to throw them at these 5,000,000 matrix multiplies, oh and I don't care about latency".
I think it’s the same reason windows is inportant to desktop computers. Software was written to depend on it. Same with most of the software out there today to train being built around CUDA. Even a version difference of CUDA can break things.
CUDA is just a better dev experience. Lots of training is experiments where developer/researcher productivity matters. Googlers get to use what they're given, others get to choose.
Once you settle on a design then doing ASICs to accelerate it might make sense. But I'm not sure the gap is so big, the article says some things that aren't really true of datacenter GPUs (Nvidia dc gpus haven't wasted hardware on graphics related stuff for years).
That quote left me with the same question. Something about decent amount of ram on one board perhaps? That’s advantageous for training but less so for inference?
inference is often a static, bounded problem solvable by generic compilers. training requires the mature ecosystem and numerical stability of cuda to handle mixed-precision operations. unless you rewrite the software from the ground up like Google but for most companies it's cheaper and faster to buy NVIDIA hardware
This feels a lot like the RISC/CISC debate. More academic than it seems. Nvidia is designing their GPUs primarily to do exactly the same tasks TPUs are doing right now. Even within Google it's probably hard to tell whether or not it matters on a 5-year timeframe. It certainly gives Google an edge on some things, but in the fullness of time "GPUs" like the H100 are primarily used for running tensor models and they're going to have hardware that is ruthlessly optimized for that purpose.
And outside of Google this is a very academic debate. Any efficiency gains over GPUs will primarily turn into profit for Google rather than benefit for me as a developer or user of AI systems. Since Google doesn't sell TPUs, they are extremely well-positioned to ensure no one else can profit from any advantages created by TPUs.
> Since Google doesn't sell TPUs, they are extremely well-positioned to ensure no one else can profit from any advantages created by TPUs.
First part is true at the moment, not sure the second follows. Microsoft is developing their own “Maia” chips for running AI on Azure with custom hardware, and everyone else is also getting in the game of hardware accelerators. Google is certainly ahead of the curve in making full-stack hardware that’s very very specialized for machine learning. But everyone else is moving in the same direction: lots of action is in buying up other companies that make interconnects and fancy networking equipment, and AMD/NVIDIA continue to hyper specialize their data center chips for neural networks.
Google is in a great position, for sure. But I don’t see how they can stop other players from converging on similar solutions.
I feel like this is more like the console/PC debate in the 90s. Consoles like the SNES had dedicated fixed function graphics hardware with weaker general specs, but with the special HW they could perform as well as a much more expensive PC - but as devs made more and more varied and clever games, that fixed function hardware couldn't support it and the PC became the superior choice.
Weird they'd do this after developing several generations of their own inference chip. Google is basically a competitor. This may just be a ploy to get better pricing from Nvidia.
I don't think what the article writes about matters all that much. Gemini 3 Pro is arguably not even the best model anymore, and it's _weeks_ old, and Google has far more resources than Anthropic does. If the hardware actually was the secret sauce, Google would be wiping the floor with little everyone else.
But they're not.
There's a few confounding problems:
1. Actually using that hardware effectively isn't easy. It's not as simple as jacking up some constant values and reaping the benefits. Actually using the hardware is hard, and by the time you've optimized for it, you're already working on the next model.
2. This is a problem that, if you're not Google, you can just spend your way out of. A model doesn't take a petabyte of memory to train or run. Regular old H100s still mostly work fine. Faster models are nice, but Gemini 3 Pro being 50% of the latency as Opus 4.5 or GPT 5.1 doesn't add enough value to matter to really anyone.
3. There's still a lot of clever tricks that work as low hanging fruit to improve almost everything about ML models. You can make stuff remarkably good with novel research without building your own chips.
4. A surprising amount of ML model development is boots on the ground work. Doing evals. Curating datasets. Tweaking system prompts. Having your own Dyson sphere doesn't obviate a lot of the typing and staring at a screen that necessarily has to be done to make a model half decent.
5. Fancy bespoke hardware means fancy bespoke failure modes. You can search stack overflow for CUDA problems, you can't just Bing your way to victory when your fancy TPU cluster isn't doing the thing you want it to do.
I think you are addressing the issue from a developer's perspective. I don't think TPUs are going to be sold to individual users anytime soon. What the article is pointing out is that Google is now able to squeeze significantly more performance per dollar than their peer competitors in the LLM space.
For example, OpenAI has announced trillion-dollar investments in data centers to continue scaling. They need to go through a middle-man (Nvidia), while Google does not, and will be able to use their investment much more efficiently to train and serve their own future models.
> Google is now able to squeeze significantly more performance per dollar than their peer competitors in the LLM space
Performance per dollar doesn't "win" anything though. Performance (as in speed) hardly cracks the top five concerns that most folks have when choosing a model provider, because fast, good models already exist at price points that are acceptable. That might mean slightly better margins for Google, but ultimately isn't going to make them "win"
It's not slightly better margins, we are talking about huge cost reductions on the main expense which is compute. In a context where companies are making trillion dollar investments, it matters a lot.
Also, performance and user choice are definitely impacted by compute. If they ever find a way to replace a job with LLMs, those who can throw more compute at it for a lower price point will win.
Google owns 14% of Anthropic and Anthropic is using Google TPUs, as well as AWS Trainium and of course GPUs. It isn't necessary for one company to create both the winning hardware and the winning software to be part of the solution. In fact with the close race in software hardware seems like the better bet.
But price per token isn't even a directly important concern anymore. Anyone with a brain would pay 5x more per token for a model that uses 10x fewer tokens with the same accuracy. I've gone all in on Opus 4.5 because even though it's more expensive, it solves the problems I care about with far fewer tokens.
Slightly more seriously: what you say makes sense if and only if you're projecting Sam Altman and assuming that a) real legit superhuman AGI is just around the corner, and b) all the spoils will accrue to the first company that finds it, which means you need to be 100% in on building the next model that will finally unlock AGI.
But if this is not the case -- and it's increasingly looking like it's not -- it's going to continue to be a race of competing AIs, and that race will be won by the company that can deliver AI at scale the most cheaply. And the article is arguing that company will be Google.
I think you are missing the point. They are saying "weeks old" isn't very old.
> it's going to continue to be a race of competing AIs, and that race will be won by the company that can deliver AI at scale the most cheaply.
I don't see how that follows at all. Quality and distribution both matter a lot here.
Google has some advantages but some disadvantages here too.
If you are on AWS GovCloud, Anthropic is right there. Same on Azure, and on Oracle.
I believe Gemini will be available on the Oracle Cloud at some point (it has been announced) but they are still behind in the enterprise distribution race.
OpenAI is only available on Azure, although I believe their new contract lets them strike deals elsewhere.
On the consumer side, OpenAI and Google are well ahead of course.
Last week it looked like Google had won (hence the blog post) but now almost nobody is talking about antigravity and Gemini 3 anymore so yeah what op says is relevant
It definitely depends on how you're measuring. But the benchmarks don't put it at the top for many ways of measuring, and my own experience doesn't put it at the top. I'm glad if it works for you, but it's not even a month old and there are lots of folks like me who see it as definitely worse for classes of problems that 3 Pro could be the best at.
Which is to say, if Google was set up to win, it shouldn't even be a question that 3 Pro is the best. It should be obvious. But it's definitely not obvious that it's the best, and many benchmarks don't support it as being the best.
On point 5, I think this is the real moat for CUDA. Does Google have tools to optimize kernels on their TPUs? Do they have tools to optimize successive kernel launches on their TPUs? How easy is it to debug on a TPU(arguably CUDA could use work here but still...)? Does Google help me fully utilize their TPUs? Can I warm up a model on a TPU, checkpoint it, and launch the checkpoints to save time?
I am fairly pro-google(they invented the LLM, FFS...) and recognize the advantages(price/token, efficiency, vertical integration, established DCs w/ power allocations) but also know they have a habit of slightly sucking at everything but search.
A question I don't see addressed in all these articles: what prevents Nvidia from doing the same thing and iterating on their more general-purpose GPU towards a more focused TPU-like chip as well, if that turns out to be what the market really wants.
The big difference is that Google is both the chip designer *and* the AI company. So they get both sets of profits.
Both Google and Nvidia contract TSMC for chips. Then Nvidia sells them at a huge profit. Then OpenAI (for example) buys them at that inflated rate and them puts them into production.
So while Nvidia is "selling shovels", Google is making their own shovels and has their own mines.
on top of that Google is also cloud infrastructure provider - contrary to OpenAI that need to have someone like Azure plug those GPUs and host servers.
The own shovels for own mines strategy has a hidden downside: isolation. NVIDIA sells shovels to everyone - OpenAI, Meta, xAI, Microsoft - and gets feedback from the entire market. They see where the industry is heading faster than Google, which is stewing in its own juices. While Google optimizes TPUs for current Google tasks (Gemini, Search), NVIDIA optimizes GPUs for all possible future tasks. In an era of rapid change, the market's hive mind usually beats closed vertical integration.
Selling shovels may still turn out to be the right move: Nvidia got rich off the cryptocurrency bubble, now they're getting even richer off the AI bubble.
Having your own mines only pays off if you actually do strike gold. So far AI undercuts Google's profitable search ads, and loses money for OpenAI.
So when the bubble pops the companies making the shovels (TSMC, NVIDIA) might still have the money they got for their products and some of the ex-AI companies might least be able to sell standard compliant GPUs on the wider market.
And Google will end up with lots of useless super specialized custom hardware.
It seems unlikely that large matrix multipliers will become useless. If nothing else, Google uses AI extensively internally. It already did in ways that weren’t user-visible long before the current AI boom. Also, they can still put AI overviews on search pages regardless of what the stock market does. They’re not as bad as they used to be, and I expect they’ll improve.
Even if TPU’s weren’t all that useful, they still own the data centers and can upgrade equipment, or not. They paid for the hardware out of their large pile of cash, so it’s not debt overhang.
Another issue is loss of revenue. Google cloud revenue is currently 15% of their total, so still not that much. The stock market is counting on it continuing to increase, though.
If the stock market crashes, Google’s stock price will go down too, and that could be a very good time to buy, much like it was in 2008. There’s been a spectacular increase since then, the best investment I ever made. (Repeating that is unlikely, though.)
How could Google's custom hardware become useless? They've used it for their business for years now and will do so for years into the future. It's not like their hardware is LLM specific. Google cannot lose with their vast infrastructure.
Meanwhile OpenAI et al dumping GPUs while everyone else is doing the same will get pennies on the dollar. It's exactly the opposite to what you describe.
I hope that comes to pass, because I'll be ready to scoop up cheap GPUs and servers.
Same way cloud hardware always risks becoming useless. The newer hardware is so much better you can't afford to not upgrade, e.g. an algorithmic improvement that can be run on CUDA devices but not on existing TPUs, which changes the economics of AI.
> And Google will end up with lots of useless super specialized custom hardware.
If it gets to the point where this hardware is useless (I doubt it), yes Google will have it sitting there. But it will have cost Google less to build that hardware than any of the companies who built on Nvidia.
Right, and the inevitable bubble pop will just slow things down for a few years - it's not like those TPUs will suddenly be useless, Google will still have them deployed, it's just that instead of upgrading to a newer TPU they'll stay with the older ones longer. It seems like Google will experience much less repercussions when the bubble pops compared to Nvidia, OpenAI, Anthropic, Oracle etc. as they're largely staying out of the money circles between those companies.
I think people are confusing the bubble popping with AI being over. When the dot-com bubble popped, it's not like internet infrastructure immediately became useless and worthless.
that's actually not all that true... a lot of fiber that had been laid went dark, or was never lit, and was hoarded by telecoms in an intentional supply constrained market in order to drive up the usage cost of what was lit.
If it was hoarded by anyone, then by definition not useless OR worthless. Also, you are currently on the internet if you're reading this, so the point kinda stands.
Google uses TPUs for its internal AI work (training Gemini for example), which surely isn't decreasing in demand or usage as their portfolio and product footprint increases. So I have a feeling they'd be able to put those TPUs to good use?
Nvidia doesn't have the software stack to do a TPU.
They could make a systolic array TPU and software, perhaps. But it would mean abandoning 18 years of CUDA.
The top post right now is talking about TPU's colossal advantage in scaling & throughput. Ironwood is massively bigger & faster than what Nvidia is shooting for, already. And that's a huge advantage. But imo that is a replicateable win. Throw gobs more at networking and scaling and nvidia could do similar with their architecture.
The architectural win of what TPU is more interesting. Google sort of has a working super powerful Connection Machine CM-1. The systolic array is a lot of (semi-)independent machines that communicate with nearby chips. There's incredible work going on to figure out how to map problems onto these arrays.
Where-as on a GPU, main memory is used to transfer intermediary results. It doesn't really matter who picks up work, there's lots of worklets with equal access time to that bit of main memory. The actual situation is a little more nuanced (even in consumer gpu's there's really multiple different main memories, which creates some locality), but there's much less need for data locality in the GPU, and much much much much tighter needs, the whole premise of the TPU is to exploit data locality. Because sending data to a neighbor is cheap, sending storing and retrieving data from memory is slower and much more energy intense.
CUDA takes advantage of, relies strongly on the GPU's reliance in main memory being (somewhat) globally accessible. There's plenty of workloads folks do in CUDA that would never work on TPU, on these much more specialized data-passing systolic arrays. That's why TPUs are so amazing, because they are much more constrained devices, that require so much more careful workload planning, to get the work to flow across the 2D array of the chip.
Google's work on projects like XLA and IREE is a wonderful & glorious general pursuit of how to map these big crazy machine learning pipelines down onto specific hardware. Nvidia could make their own or join forces here. And perhaps they will. But the CUDA moat would have to be left behind.
But it's still something grafted onto the existing architecture, of many grids with many blocks with many warps, and lots and lots of coordination and passing intermediary results around. It's only a 4x4x4 unit, afaik. There's still a lot of main memory being used to combine data, a lot of orchestration among the different warps and blocks and grids, to get big matrices crunched.
The systolic array is designed to allow much more fire and forget operations. It's inputs are 128 x 128 and each cell is its own compute node basically, shuffling data through and across (but not transitting a far off memory).
TPU architecture has plenty of limitations. It's not great at everything. But if you can design work to flow from cell to neighboring cell, you can crunch very sizable chunks of data with amazing data locality. The efficiency there is unparalleled.
Nvidia would need a radical change of their architecture to get anything like the massive data locality wins a systolic array can do. It would come with massively more constraints too.
Deepmind gets to work directly with the TPU team to make custom modifications and designs specifically for deepmind projects. They get to make pickaxes that are made exactly for the mine they are working.
Everyone using Nvidia hardware has a lot of overlap in requirements, but they also all have enough architectural differences that they won't be able to match Google.
OpenAI announced they will be designing their own chips, exactly for this reason, but that also becomes another extremely capital intensive investment for them.
This also doesn't get into that Google also already has S-tier dataceters and datacenter construction/management capabilities.
Isn’t there a suspicion that OpenAI buying custom chips from another Sam Altman venture is just graft? Wasn’t that one of the things that came up when the board tried to out him?
> Deepmind gets to work directly with the TPU team to make custom modifications
You don't think Nvidia has field-service engineers and applications engineers with their big customers? Come on man. There is quite a bit of dialogue between the big players and the chipmaker.
They do, but they need to appease a dozen different teams from a dozen different labs, forcing nvidia to take general approaches and/or dictating approaches and pigeonholing labs into using those methods.
Deepmind can do whatever they want, and get the exact hardware to match it. It's a massive advantage when you can discover a bespoke way of running a filter, and you can get a hardware implementation of it without having to share that with any third parties. If OpenAI takes a new find to Nvidia, everyone else using Nvidia chips gets it too.
This ignores the way it often works: Customer comes to NVDA with a problem and NVDA comes up with a solution. This solution now adds value for every customer.
In your example, if OpenAI makes a massive new find they aren't taking it to NVDA.
Nvidia has the advantage of a broad base of customers that gives it a lot of information on what needs work and it tries to quickly respond to those deficiencies.
It's not that the TPU is better than an NVidia GPU, it's just that it's cheaper since it doesn't have a fat NVidia markup applied, and is also better vertically integrated since it was designed/specified by Google for Google.
TPUs are also cheaper because GPUs need to be more general purpose whereas TPUs are designed with a focus on LLM workloads meaning there's not wasted silicon. Nothing's there that doesn't need to be there. The potential downside would be if a significantly different architecture arises that would be difficult for TPUs to handle and easier for GPUs (given their more general purpose). But even then Google could probably pivot fairly quickly to a different TPU design.
The T in TPU stands for tensor, which in this context is just a fancy matrix. These days both are optimised for matrix algebra, i.e. general ML workloads, not just LLMs.
If LLMs become unfashionable they’ll still be good for other ML tasks like image recognition.
Nothing in principle.
But Huang probably doesn't believe in hyper specializing their chips at this stage because it's unlikely that the compute demands of 2035 are something we can predict today.
For a counterpoint, Jim Keller took Tenstorrent in the opposite direction. Their chips are also very efficient, but even more general purpose than NVIDIA chips.
How is Tenstorrent h/w more general purpose than NVIDIA chips? TT hardware is only good for matmuls and some elementwise operations, and plain sucks for anything else. Their software is abysmal.
Of course there's the general purpose RISC V CPU controller component but also, each NPU is designed in troikas that have one core reading data in, one core performing the actual kernel work, and the third core forwarding data out.
For users buying H200s for AI workloads, the "ASIC" tensor cores deliver the overwhelming bulk of performance. So they already do this, and have been since Volta in 2017.
To put it into perspective, the tensor cores deliver about 2,000 TFLOPs of FP8, and half that for FP16, and this is all tensor FMA/MAC (comprising the bulk of compute for AI workloads). The CUDA cores -- the rest of the GPU -- deliver more in the 70 TFLOP range.
So if data centres are buying nvidia hardware for AI, they already are buying focused TPU chips that almost incidentally have some other hardware that can do some other stuff.
I mean, GPUs still have a lot of non-tensor general uses in the sciences, finance, etc, and TPUs don't touch that, but yes a lot of nvidia GPUs are being sold as a focused TPU-like chip.
Is it the Cuda cores that run the vertex/fragment/etc shaders in normal GPUs? Where does the ray tracing units fit in? How much of a modern Nvidia GPU is general purpose vs specialized to graphics pipelines?
Except the native width of Tensor Cores are about 8-32 (depending on scalar type), whereas the width of TPUs is up to 256. The difference in scale is massive.
That's pretty much what they've been doing incrementally with the data center line of GPUs versus GeForce since 2017. Currently, the data center GPUs now have up to 6 times the performance at matrix math of the GeForce chips and much more memory. Nvidia has managed to stay one tape out away from addressing any competitors so far.
The real challenge is getting the TPU to do more general purpose computation. But that doesn't make for as good a story. And the point about Google arbitrarily raising the prices as soon as they think they have the upper hand is good old fashioned capitalism in action.
For sure, I did not mean to imply they could do it quickly or easily, but I have to assume that internally at Nvidia there's already work happening to figure out "can we make chips that are better for AI and cheaper/easier to make than GPUs?"
It’s not binary. It’s not existential. What’s at stake for Nvidia is its HUGE profit margins. 5 years from now, Nvidia could be selling 100x as many chips. But its market cap could be a fraction of what it is now if competition is so intense that its making 5% profit margin instead of 90%.
> what prevents Nvidia from doing the same thing and iterating on their more general-purpose GPU towards a more focused TPU-like chip as well, if that turns out to be what the market really wants.
Nothing prevents them per se, but it would risk cannibalising their highly profitable (IIRC 50% margin) higher end cards.
My personal guess would be what drives the cost and size of these chips is the memory bandwidth and the transcievers required to support it. Since transcievers/memory controllers are on the edge of the chip, you get a certain minimum circumference for a given bandwidth, which determines your min surface area.
It might be even 'free' to fill it with more complicated logic (especially one that allows you write clever algorithms that let you save on bandwidth).
I have never understood why, in these discussions, nobody brings up other specialized silicon providers like Groq, SambaNova, or my personal favorite, Cerebras.
Cerebras CS-3 specs:
• 4 trillion transistors
• 900,000 AI cores
• 125 petaflops of peak AI performance
• 44GB on-chip SRAM
• 5nm TSMC process
• External memory: 1.5TB, 12TB, or 1.2PB
• Trains AI models up to 24 trillion parameters
• Cluster size of up to 2048 CS-3 systems
• Memory B/W of 21 PB/s
• Fabric B/W of 214 Pb/s (~26.75 PB/s)
Comparing GPU to TPU is helpful for showcasing the advantages of the TPU in the same way that comparing CPU to Radeon GPU is helpful for showcasing the advantages of GPU, but everyone knows Radeon GPU's competition isn't CPU, it's Nvidia GPU!
TPU vs GPU is new paradigm vs old paradigm. GPUs aren't going away even after they "lose" the AI inference wars, but the winner isn't necessarily guaranteed to be the new paradigm chip from the most famous company.
Cerebras inference remains the fastest on the market to this day to my knowledge due to the use of massive on-chip SRAM rather than DRAM, and to my knowledge, they remain the only company focused on specialized inference hardware that has enough positive operating revenue to justify the costs from a financial perspective.
I get how valuable and important Google's OCS interconnects are, not just for TPUs or inference, but really as a demonstrated PoC for computing in general. Skipping the E-O-E translation in general is huge and the entire computing hardware industry would stand to benefit from taking notes here, but that alone doesn't automatically crown Google the victor here, does it?
I have read in the past that ASICs for LLMs are not as simple a solution compared to cryptocurrency. In order to design and build the ASIC you need to commit to a specific architecture: a hashing algorithm for a cryptocurrency is fixed but the LLMs are always changing.
Am I misunderstanding "TPU" in the context of the article?
LLMs require memory and interconnect bandwidth so needs a whole package that is capable of feeding data to the compute. Crypto is 100% compute bound. Crypto is a trivially parallelized application that runs the same calculation over N inputs.
Regardless of architecture (which is anyways basically the same for all LLMs), the computational needs of modern neural networks are pretty generic, centered around things like matrix multiply, which is what the TPU provides. There is even TPU support for some operations built into PyTorch - it is not just a proprietary interface that Google use themselves.
It’s true that architectures change, but they are built from common components. The most important of those is matrix multiplication, using a relatively small set of floating point data types. A device that accelerates those operations is, effectively, an ASIC for LLMs.
DSP is simply a compute architecture that focuses on mutliply and accumulate operations on particular numerical formats, often either fixed point q15/q31 type values or floats f16/f32.
The basic operation that a NN needs accelerating is... go figure multiply and accumulate with the added activation function.
A DSP contains analog to digital and digital to analog converters plus DMA for fast transfers to main memory and fixed function blocks for finite impulse response and infinite pulse response filters.
The fact that they also support vector operations or matrix multiplication is kind of irrelevant and not a defining characteristic of DSPs. If you want to go that far, then everything is a DSP, because all signals are analog.
Maybe also note that Qualcomm has renamed their Hexagon DSP to Hexagon NN. Likely the change was adding activation functions but otherwise its a VLIW architecture with accelerated MAC operations, aka a DSP architecture.
ASICs bake one algorithm into the chip. DSPs are programmable, like GPUs or CPUs. The thing that historically set them apart were MAC/FMA and zero overhead loops. Then there are all the nice to haves, like built in tables of FFT twiddle factors, helpers for 1D convolution, vector instructions, fixed point arithmetic, etc.
What makes a DSP different from a GPU is the algorithms typically do not scale nicely to large matrices and vectors. For example, recursive filters. They are also usually much cheaper and lower power, and the reason they lost popularity was because Arm MCUs got good enough and economy of scale kicked in.
I've written code for DSPs both in college and professionally. It's much like writing code for CPUs or MCUs (it's all C or C++ at the end of the day). But it's very different from writing compute shaders or designing an ASIC.
"Application-specific" doesn't necessarily mean unprogrammable. Bitcoin miners aren't programmable because they don't need to be. TPUs are ASICs for ML and need to be programmable so they can run different models. In theory, you could make an ASIC hardcoded for a specific model, but given how fast models evolve, it probably wouldn't make much economic sense.
Cryptocurrency architectures also change - Bitcoin is just about the lone holdout that never evolves. The hashing algorithm for Monero is designed so that a Monero hashing ASIC is literally just a CPU, and it doesn't even matter what the instruction set is.
If Google won, it would cannibalize its current ad-driven business and replace it with something that is extremely expensive to run and difficult to make profit from. A Pyrrhic win essentially.
But that would happen regardless of who won, better to at least dominate the new paradigm and figure out how to extract value from it. I also suspect that once the value generation is figured out they will cease offering these APIs to anyone, if you had a golden goose would you rent it?
I mean, focus is a thing that Google has always struggled with. But I kind of doubt that customers who need online marketing (ads) are going to convert en masse to users who rent cloud TPUs instead.
- ASIC won the crypto mining battle in the past, it's orders of magnitude faster
- Google is not owning the technology but builds a cohesive cloud around it, Tesla, Meta work on their own asic ai chips and I guess others
- A signal is already given: Softbank sold it's entire Nvidia stock and berkshire added google on their portfolio.
Microsoft "has" a lot of companies data, and google is probably building the most advanced ai cloud.
However, I can't think they had a cloud which was light-years ahead of aws 15 years ago and now GCP is no 3, they also released opensource gpt models more than 5 years ago that constituted the foundation for openai closed sourced models.
Given the importance of scale for this particular product, any company placing itself on "just" one layer of the whole story is at a heavy disadvantage, I guess. I'd rather have a winning google than openai or meta anyway.
I agree, it would be the best of bad cases, in a sense. I have low trust in OpenAI due to its leadership, and in Meta, because, well, Meta has history, let's say.
Sparse models have same quality of results but have less coefficients to process, in case described in the link above sixteen (16) times as less.
This means that these models need 8 times less data to store, can be 16 and more times faster and use 16+ times less energy.
TPUs are not all that good in the case of sparse matrices. They can be used to train dense versions, but inference efficiency with sparse matrices may be not all that great.
The part that surprised me is how much TPUs gain from the systolic array design. It basically cuts down the constant memory shuffling that GPUs have to do, so more of the chip’s time is spent actually computing.
The downside is the same thing that makes them fast: they’re very specialized. If your code already fits the TPU stack (JAX/TensorFlow), you get great performance per dollar. If not, the ecosystem gap and fear of lock-in make GPUs the safer default.
Will Google sell TPUs that can be plugged into stock hardware, or custom hardware with lots of TPUs? Our customers want all their video processing to happen on site, and don't want their video or other data to touch the cloud, so they're not happy about renting cloud TPUs or GPUs. Also it would be nice to have smart cameras with built-in TPUs.
It's not Google Cloud per se, it's any cloud. There are a million reasons not to trust (or spend money on) any cloud. They want all their video and data on premises and completely under their control.
With its AI offerings, can Google suck the oxygen out of AWS? AWS grew big because of compute. The AI spend will be far larger than compute. Can Google launch AI/Cloud offerings with free compute bundled? Use our AI, and we'll throw in compute for free.
The funniest thing about this story is that NVIDIA has essentially become a TPU company. Look at the Hopper and Blackwell architectures: Tensor Cores are taking up more space, the Transformer Engine has appeared, and NVLink has started to look like a supercomputer interconnect. Jensen Huang isn't stupid. He saw the threat of specialized ASICs and just built the ASIC inside the GPU. Now we have a GPU that is 80% matrix multiplier but still keeps CUDA compatibility. Google tried to kill the GPU, but instead forced the GPU to mutate into a TPU
It's a cool subject and article and things I only have a general understanding of (considering the place of posting).
What I'm sure about is having a programming unit more purposed to a task is more optimal than a general programming unit designed to accommodate all programming tasks.
More and more of the economics of programming boils down to energy usage and invariably towards physical rules, the efficiency of the process has the benefit of less energy consumed.
As a Layman is makes general sense. Maybe a future where productivity is based closer on energy efficiency rather than monetary gain pushes the economy in better directions.
Cryptocurrency and LLMs seem like they'll play out that story over the next 10 years.
You can't really buy a TPU, you have to buy the entire data center that includes the TPU plus the services and support. In Google Colab, I often don't prefer the TPU either because the documentation for the AI isn't made for it. While this could all change in the long term, I also don't see these changes in Google's long term strategy. There's also the problem with Google's graveyard which isn't mentioned in the long term of the original article. Combined with these factors, I'm still skeptical about Google's lead on AI.
How much of current GPU and TPU design is based around attn's bandwith hungry design? The article makes it seem like TPUs aren't very flexible so big model architecture changes, like new architectures that don't use attn, may lead to useless chips. That being said, I think it is great that we have some major competing architectures out there. GPUs, TPUs and UMA CPUs are all attacking the ecosystem in different ways which is what we need right now. Diversity in all things is always the right answer.
This is a bizarre argument to make for AI, since Google started working on TPUs in 2013 (12 years ago) and Sundar started publicly banging on about being an AI-first company in 2016. They missed the first boat on LLMs, but Google has been invested in AI for way longer than any of the competition.
Their incentive structure doesn't lead to longevity. Nobody gets promoted for keeping a product alive, they get promoted for shipping something new. That's why we're on version 37 of whatever their chat client is called now.
I think we can be reasonably sure that search, Gmail, and some flavor of AI will live on, but other than that, Google apps are basically end-of-life at launch.
It's also paradoxically the talent in tech that isolates them. The internal tech stack is so incredibly specialized, most Google products have to either be built for internal users or external users.
Agree there are lots of other contributing causes like culture, incentives, security, etc.
[1] Ref: he mistakenly posted what was meant to be an internal memo, publicly on G+. He quickly took it down but of course The Internet Never Forgets https://gist.github.com/chitchcock/1281611
It's telling that basically all of Google's successful projects were either acquisitions or were sponsored directly by the founders (or sometimes, were acquisitions that were directly sponsored by the founders). Those are the only situations where you are immune from the performance review & promotion process.
They've actually had many very successful projects that make the few products and acquisitions you are thinking of work. It's true most of their end products don't work or get abandoned but it stretches their infrastructure in ways that works out well in the long run
I should probably have said "products" rather than "projects". There's a fair bit of extremely good engineering that goes on in the infrastructure side, but when it comes to consumer products, if one of the founders isn't explicitly sponsoring it it gets killed.
They get killed since ROI of putting more people on search or ads is almost always higher than ROI on the new projects, not because the founders doesn't like them.
> The GPUs were designed for graphics [...] However, because they are designed to handle everything from video game textures to scientific simulations, they carry “architectural baggage.” [...] A TPU, on the other hand, strips away all that baggage. It has no hardware for rasterization or texture mapping.
With simulations becoming key to training models doesn't this seem like a huge problem for Google?
deepseek kind of innovated on this using off-the-shelf components right ?
to quote from their paper
"In order to ensure sufficient computational performance for DualPipe, we customize efficient
cross-node all-to-all communication kernels (including dispatching and combining) to conserve
the number of SMs dedicated to communication. The implementation of the kernels is codesigned with the MoE gating algorithm and the network topology of our cluster."
The most fun fact about all the developments post-ChatGPT is that people apparently forgot that Google was doing actual AI before AI meant (only) ML and GenAI/LLMs, and they were top players at it.
Arguably main OpenAI raison d'être was to be a counterweight to that pre-2023 Google AI dominance. But I'd also argue that OpenAI lost its way.
That and they were harvesting data way before it was cool, and now that it is cool, they're in a privileged position since almost no-one can afford to block GoogleBot.
They do voluntarily offer a way to signal that the data GoogleBot sees is not to be used for training, for now, and assuming you take them at their word, but AFAIK there is no way to stop them doing RAG on your content without destroying your SEO in the process.
All this assumes that LLMs are the sole mechanism for AI and will remain so forever: no novel architectures (neither hardware nor software), no progress in AI theory, nothing better than LLMs, simply brute force LLM computation ad infinitum.
Perhaps the assumptions are true. The mere presence of LLMs seems to have lowered the IQ of the Internet drastically, sopping up financial investors and resources that might otherwise be put to better use.
It's all about the right mind set at the very top level. At the beginning of the PC era, nobody would bet IBM to lose. Same in the dawn of internet, all money was on MS. so it happened to Nokia and Ericsson.
Google is a giant without a direction. The ads money is so good that it just doesn't have the gut to leave it on the table.
That's mentioned in the article, but is the lock-in really that big? In some cases, it's as easy as changing the backend of your high-level ML library.
That's what it is on paper. But in practice you trade one set of hardware idiosyncrasies for another and unless you have the right people to deal with that, it's a hassle.
On top, when you get locked into Google Cloud, you’re effectively at the mercy of their engineers to optimize and troubleshoot. Do you think Google will help their potential competitors before they help themselves? Highly unlikely considering their actions in the past decade plus.
That's actually one of the reasons why Google might win.
Nvidia is tied down to support previous and existing customers while Google can still easily shift things around without needing to worry too much about external dependencies.
At the time Google+ was started and shortly after, leadership (larry page at that time) focused the attention of the company on it. There was a social bonus (that you'd get if you integrated your product), there were large changes to existing systems to support Google+, and the company made it quite clear it thought that social was the direction to go and that Google+ was going to be an enormous product.
I and a lot of other googlers were really confused by all of this because at the time we were advocating that Google put more effort into its nascent cloud business (often to get the reply "but we already have appengine" or "cloud isn't as profitable as ads") and that social, while getting a lot of attention, wasn't really a good business for google to be in (with a few exceptions like Orkut and Youtube, Google's attempts at social have been pretty uninspired).
There were even books written at the time that said Google looked lazy and slow and that Meta was going to eat their lunch. But shortly after Google+ tanked, Google really began to focus on Cloud (in a way that pissed off a lot of Googlers in the same way Google+ did- by taking resources and attention from other projects). Now, Meta looks like its going to have a challenging future while Google is on to achieving what Larry Page originally intended: a reliable revenue stream that is reinvested into development of true AI.
In my recollection, what killed g+ was forcing your YouTube account to become your g+ account, with your public name attached to the trashpit YouTube comments used to be. Everybody protested using g+, but the "Google account for everything" stuck around anyways.
Chromecast is "gone" because it bridged the gap of dumb tvs needing streaming capabilities. Now almost every tv sold has some kind of smart feature or can stream natively so Chromecast aren't needed.
chromecast is alive, podcast, music were migrated to youtube app, url shortener is not core business and just side hustle for google. Not familiar with g suite legacy.
Wait until Apple's ChromeBook competitor shows up to eat their lunch just like switching to another proprietary stack with no dev ecosystem will die out. Sure they'll go after big ticket accounts, also take a guess at what else gets sanctioned next.
This is the “Microsoft will dominate the Internet” stage.
The truth is the LLM boom has opened the first major crack in Google as the front page of the web (the biggest since Facebook), in the same way the web in the long run made Windows so irrelevant Microsoft seemingly don’t care about it at all.
Exactly, ChatGPT pretty much ate away ad volume & retention if th already garbage search results weren't enough. Don't even get me started on Android & Android TV as an ecosystem.
How high are the chances that as soon as China produces their own competitive TPU/GPU, they'll invade Taiwan in order to starve the West in regards to processing power, while at the same time getting an exclusive grip on the Taiwanese Fabs?
The US would destroy TSMC before letting China have it. China also views military conquest of Taiwan as less than ideal for a number of reasons, so I think right now it's seen as a potential defensive move in the face of American aggression.
Not very. Those fabs are vulnerable things, shame if something happens to them. If China attacks, it would be for various other reasons and processors are only one of many considerations, no matter how improbable it might sound to an HN-er.
What if China becomes self-sufficient enough to no longer rely on Taiwanese Fabs, and hence having no issues with those Fabs getting destroyed. That would put China as the leader once and for all.
First, the US has advanced fab capabilities and in case of a need can develop them further. On the other side, China will suffer a Russia style blockback while caught up in a nasty war with Taiwan.
Totally possible, but the second order effects are much more complex than "leader once for all". The path for victory for China is not war despite the west, but a war when the west would not care.
Even the US suffers (their veterans do, anyway) but that's the country that in general least suffers from their constant involvement in warfare. They have this industry down to a "T". However, you cannot generalize from a nation that can project military force almost anywhere in the globe, with little fear of repercussion back home; most countries cannot afford this. China certainly cannot.
So what about the rest? Internecine conflicts are outrageously wasteful, and sadly common in the modern age. Russia's war with Ukraine has turned incredibly wasteful and costly, and Russians are suffering (and dying) regardless of whatever Putin says.
I think China is not generally oriented towards waging war. They do have their military, military projects, and their nationalistic things (what I learn from Wikipedia is called "irredentism"), but generally they seem to be trying to become an economic world power. War would mess and interfere with that. War is too fucking risky.
If they have the fabs but ASML doesn't send them their new machines, they will just end up in the same situation as now, just one generation later. If China wants to compete, they need to learn how to make the EUV light and mirrors.
Seems low at the moment with the concept of G2 being floated as generic understanding of China's ascension to where Russia used to be effectively recreating bipolar semi cold war world order. Mind, I am not saying impossible, but there are reasons China would want to avoid this scenario ( probably one of the few things US would not tolerate and would likely retaliate ).
Highly unlikely. Despite the rampant anti-Chinese FUD that's so prevalent in the media (and, sadly, here on HN), China isn't really in the habit of invading other lands.
But China also doesn't see war as the best path forward in Taiwan (they want to return it to the mainland, not lay waste to it). The grandparent comment is unfairly downvoted in my opinion, the fact remains modern China is far less likely to be involved in military campaigns than, say, the US.
In my 20+ years of following NVIDIA, I have learned to never bet against them long-term. I actually do not know exactly why they continually win, but they do. The main issue they have a 3-4 year gap between wanting a new design pivot and realizing it (silicon has a long "pipeline"), it can seem that they may be missing a new trend or swerve in the demands of the market, it is often simply because there is this delay.
Fair, but the 75% margins can be reduced to 25% with healthy competition. The lack of competition in the frontier chips space was always the bottleneck to commoditization of computation, if such a thing is even possible
Google's real moat isn't the TPU silicon itself—it's not about cooling, individual performance, or hyper-specialization—but rather the massive parallel scale enabled by their OCS interconnects.
To quote The Next Platform: "An Ironwood cluster linked with Google’s absolutely unique optical circuit switch interconnect can bring to bear 9,216 Ironwood TPUs with a combined 1.77 PB of HBM memory... This makes a rackscale Nvidia system based on 144 “Blackwell” GPU chiplets with an aggregate of 20.7 TB of HBM memory look like a joke."
Nvidia may have the superior architecture at the single-chip level, but for large-scale distributed training (and inference) they currently have nothing that rivals Google's optical switching scalability.
Also, Google owns the entire vertical stack, which is what most people need. It can provide an entire spectrum of AI services far cheaper, at scale (and still profitable) via its cloud. Not every company needs to buy the hardware and build models, etc., etc.; what most companies need is an app store of AI offerings they can leverage. Google can offer this with a healthy profit margin, while others will eventually run out of money.
Google's work on Jax, pytorch, tensorflow, and the more general XLA underneath are exactly the kind of anti-moat everyone has been clamoring for.
Anti-moat like commoditizing the compliment?
If they get things like PyTorch to work well without carinng what hardware it is running on, it erodes Nvidia's CUDA moat. Nvidia's chips are excellent, without doubt, but their real moat is the ecosystem around CUDA.
The problem is that "hardware-agnostic PyTorch" is a myth, much like Java's "write once, run anywhere". At the high level (API), the code looks the same, but as soon as you start optimizing for performance, you inevitably drop down to CUDA. As long as researchers are writing their new algorithms in CUDA because it's the de facto language of science, Google will forever be playing catch-up, having to port these algorithms to XLA. An ecosystem is, after all, people and their habits, not just libraries.
I'd love for someone to give me an alternative to CUDA but I don't primarily use GPUs for inference, I do 64-bit unsigned integer workloads and the only people who seem to care even a little about this currently are NVidia, if imperfectly.
I _really_ want an alternative but the architecture churn imposed by targeting ROCm for say an MI350X is brutal. The way their wavefronts and everything work is significantly different enough that if you're trying to get last-mile perf (which for GPUs unfortunately yawns back into the 2-5x stretch) you're eating a lot of pain to get the same cost-efficiency out of AMD hardware.
FPGAs aren't really any more cost effective unless the $/kwh goes into the stratosphere which is a hypothetical I don't care to contemplate.
That's new to me -- what sorts of workloads are centered on 64-bit uints?
PyTorch is only part of it. There is still a huge amount of CUDA that isn’t just wrapped by PyTorch and isn’t easily portable.
... but not in deep learning or am I missing something important here?
Yes, absolutely in deep learning. Custom fused CUDA kernels everywhere.
Yep. MoE, FlashAttention, or sparse retrieval architectures for example.
Yes!
Pytorch, Jax, tensorflow are all examples to me of very capable products, that compete very well in ML space.
But more broadly work like XLA and IREE are very interesting toolkits for mapping a huge variety of computation onto many types of hardware. While Pytorch et al are fine example applications, are things you can do, XLA is the Big Tent idea, the toolkit to erode not just specific CUDA use cases, but to allow hardware in general to be more broadly useful.
*complement
when chatgpt came, I thought google didn't have the leadership and team spirit to recover, seems like i was very wrong
They just need to actually make and market a good product though, and they seem to really struggle with this. Maybe on a long enough timeline their advantages will make this one inevitable.
all this vertical integration no wonder Apple and Google have such a tight relationship.
That is comparing an all to all switched Nvlink fabric to a 3D torus for TPUs. Those are completely different network topologies with different tradeoffs.
For example the currently very popular Mixture of Experts architectures require a lot of all to all traffic (for expert parallelism) which works a lot better on the switched NVlink fabric as opposed where it doesn't need to traverse multiple links in the torus.
This is an underrated point. Comparing just the peak bandwidth is like saying Bulldozer was the far superior CPU of the era because it had a really high frequency ceiling.
Really? Fully-connected hardware is in buildable (at scale) which we already know from the HPC world. Fat trees and dragonfly networks are pretty scalable, but a 3d torus is a very good tradeofff, and respects the dimensionality of reality.
Bisection bandwidth is a useful metric, but is hop count? Per-hop cost tends to be pretty small.
Latency (of different types), jitter, and guaranteed bandwidth are the real underlying metrics. Hop count is just one potential driver of those, but different approaches may or may not tackle each of these parts differently.
NVFP4 is the thing no one saw coming. I wasn't watching the MX process really, so I cast no judgements, but it's exactly what it sounds like, a serious compromise in resource constrained settings. And it's in the silicon pipeline.
NVFP4 is to put it mildly a masterpiece, the UTF-8 of its domain and in strikingly similar ways it is 1. general 2. robust to gross misuse 3. not optional if success and cost both matter.
It's not a gap that can be closed by a process node or an architecture tweak: it's an order of magnitude where the polynomials that were killing you on the way up are now working for you.
sm_120 (what NVIDIA's quiet repos call CTA1) consumer gear does softmax attention and projection/MLP blockscaled GEMM at a bit over a petaflop at 300W and close to two (dense) at 600W.
This changes the whole game and it's not clear anyone outside the lab even knows the new equilibrium points, it's nothing like Flash3 on Hopper, lotta stuff looks FLOPs bound, GDDR7 looks like a better deal than HBMe3. The DGX Spark is in no way deficient, it has ample memory bandwidth.
This has been in the pipe for something like five years and even if everyone else started at the beginning of the year when this was knowable, it would still be 12-18 months until tape out. And they haven't started.
Years Until Anyone Can Compete With NVIDIA is back up to the 2-5 it was 2-5 years ago.
This was supposed to be the year ROCm and the new Intel stuff became viable.
They had a plan.
This reads like a badly done, sponsored hype video on YouTube.
So if we look at what NVIDIA has to say about NVFP4 it sure sounds impressive [1]. But look closely that initial graph never compares fp8 and fp4 on the same hardware. They jump from H100 to B200 while implying a 5x jump of going with fp4 which it isn't. Accompanied with scary words like if you use MXFP4 "Risk of noticeable accuracy drop compared to FP8" .
Contrast that with what AMD has to say on the open MXFP4 approach which is quite similar to NVFP4 [2]. Ohh the horrors of getting 79.6 instead of 79.9 on GPQA Diamond when using MXFP4 instead of FP8.
[1] https://developer.nvidia.com/blog/introducing-nvfp4-for-effi...
[2] https://rocm.blogs.amd.com/software-tools-optimization/mxfp4...
Looking into NVFP4/Nvidia vs MXFP4/AMD the summation was that seem to be pretty close when including the MI355X which leads in VRAM and throughput but trails in accuracy (slightly)--and for that mixing in MXFP6 makes up for it.
This comment reads as if it were LLM-generated.
Agree.
It's fun when then you read last Nvidia tweet [1] suggesting that still their tech is better, based on pure vibes as anything in the (Gen)AI-era.
[1] https://x.com/nvidianewsroom/status/1993364210948936055
Not vibes. TPUs have fallen behind or had to be redesigned from scratch many times as neural architectures and workloads evolved, whereas the more general purpose GPUs kept on trucking and building on their prior investments. There's a good reason so much research is done on Nvidia clusters and not TPU clusters. TPU has often turned out to be over-specialized and Nvidia are pointing that out.
You say that like I d a bad thing. Nvidia architectures keep changing and getting more advanced as well, with specialized tensor operations, different accumulators and caches, etc. I see no issue with progress.
That’s missing the point. Things like tensor cores were added in parallel with improvements to existing computer and CUDA kernels from 10 years ago generally run without modification. Hardware architecture may change, but Nvidia has largely avoided changing how you interact with it.
Modern CUDA programs that hit roofline look absolutely nothing like those from 10 or even 5 years ago. Or even 2 if you’re on Blackwell.
They don't have to, CUDA is a high-level API in this respect. The hardware will conform to the demands of the market and the software will support whatever the compute capability defines, Nvidia is clearer than most about this.
But for research you often don't have to max out the hardware right away.
And the question is what do programs that max out Ironwood look like vs TPU programs written 5 years ago?
Sure, but you do have to do it pretty quick. Let’s pick a H100. You’ve probably heard that just writing scalar code is leaving 90+% of the flops idle. But even past that, if you’re using the tensor core but using the wrong instructions you’re basically capped at 300-400 TFLOPS of the 1000 the hardware supports. If using the new instructions but poorly you’re probably not going to hit even 500 TFLOPS. That’s just barely better than the previous generation you paid a bunch of money to replace.
And yet current versions of Whisper GPU will not run on my not-quite-10-year old Pascal GPU anymore because the hardware CUDA version is too old.
Just because it's still called CUDA doesn't mean it's portable over a not-that-long of a timeframe.
Portable doesn't normally mean that it runs on arbitrarily old hardware. CUDA was never portable, it only runs on Nvidia hardware. The question is whether old versions of Whisper GPU run on newer hardware, that'd be backwards compatibility.
> There's a good reason so much research is done on Nvidia clusters and not TPU clusters.
You are aware that Gemini was trained on TPU, and that most research at Deepmind is done on TPU?
> based on pure vibes
The tweet gives their justification; CUDA isn't ASIC. Nvidia GPUs were popular for crypto mining, protein folding, and now AI inference too. TPUs are tensor ASICs.
FWIW I'm inclined to agree with Nvidia here. Scaling up a systolic array is impressive but nothing new.
Sure, but their company's 4.3 trillion valuation isn't based on how good their GPUs are for general purpose computing, it's based on how good they are at AI.
> NVIDIA is a generation ahead of the industry
a generation is 6 months
For GPUs a generation is 1-2 years.
no https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_proces...
What in that article makes you think a generation is shorter?
* Turing: September 2018
* Ampere: May 2020
* Hopper: March 2022
* Lovelace (designed to work with Hopper): October 2022
* Blackwell: November 2024
* Next: December 2025 or later
With a single exception for Lovelace (arguably not a generation), there are multiple years between generations.
No, not at all. If this were true Google would be killing it in MLPerf benchmarks, but they are not.
It’s better to have a faster, smaller network for model parallelism and a larger, slower one for data parallelism than a very large, but slower, network for everything. This is why NVIDIA wins.
I mean, Google just isn't participating it seems?
For all the excitement surrounding this, I fail to comprehend how Google can't even meet the current demand for Gemini 3^. Moreover, they are unwilling to invest in expansion directly (apparently have a mandate to double their compute every 6 months without spending more than their current budget). So, pardon me if I can't see how they will scale operations as demand grows while simultaneously selling their chips to competitors?! This situation doesn't make any sense.
^Even now I get capacity related error messages, so many days after the Gemini 3 launch. Also, Jules is basically unusable. Maybe Gemini 3 is a bigger resource hog than anyone outside of Google realizes.
I also suspect Google is launching models it can’t really sustain in volume or that are operating at a loss. Nothing preventing them from like doubling model size compared to the rest or allocating an insane amount of compute just to make the headlines on model performance (clearly it’s good for the stock). These things are opaque anyway, buried deep into the P&L.
OCS is indeed an engineering marvel, but look at NVIDIA's NVL72. They took a different path: instead of flexible optics, they used the brute force of copper, turning an entire rack into one giant GPU with unified memory. Google is solving the scale-out problem, while NVIDIA is solving the scale-up problem. For LLM training tasks, where communication is the bottleneck, NVIDIA's approach with NVLink might actually prove even more efficient than Google's optical routing.
100 times more chips for equivalent memory, sure.
Check the specs again. Per chip, TPU 7x has 192GB of HBM3e, whereas the NVIDIA B200 has 186GB.
While the B200 wins on raw FP8 throughput (~9000 vs 4614 TFLOPs), that makes sense given NVIDIA has optimized for the single-chip game for over 20 years. But the bottleneck here isn't the chip—it's the domain size.
NVIDIA's top-tier NVL72 tops out at an NVLink domain of 72 Blackwell GPUs. Meanwhile, Google is connecting 9216 chips at 9.6Tbps to deliver nearly 43 ExaFlops. NVIDIA has the ecosystem (CUDA, community, etc.), but until they can match that interconnect scale, they simply don't compete in this weight class.
Isn’t the 9000 TFLOP/s number Nvidia’s relatively useless sparse FLOP count that is 2x the actual dense FLOP count?
Correct --- found a remark on Twitter calling this "Jenson Math".
Same logic when NVidia quote the "bidirectional bandwidth" of high speed interconnects to make the numbers look big, instead of the more common BW per direction, forcing everyone else to adopt the same metric in marketing materials.
Wow, no, not at all. It’s better to have a set of smaller, faster cliques connected by a slow network than a slower-than-clique flat network that connects everything. The cliques connected by a slow DCN can scale to arbitrary size. Even Google has had to resort to that for its biggest clusters.
Is this claim based on observed comm patterns in some particular AI architecture?
I guess “this weight class” is some theoretical class divorced from any application? Almost all players are running Nvidia other than Google. The other players are certainly more than just competing with Google.
> Almost all players are running Nvidia other than Google.
No surprises there, Google is not the greatest company at productizing their tech for external consumption.
> The other players are certainly more than just competing with Google.
TBF, its easy to stay in the game when you're flush with cash, and for the past N-quarters, investors have been throwing money at AI companies, Nvidia's margins have greatly benefited from this largesse. There will be blood on the floor once investors start demanding returns to their investments.
Ok? The person I was replying to was saying that Google’s compute offering is substantially superior to Nvidia’s. What do your comments about market positioning have to do with that?
If Google’s TPUs were really substantially superior, don’t you think that would result in at least short term market advantages for Gemini? Where are they?
They are suggesting it is easier for others to buy buy more NVidia chips and feed them more power. Whilst operating costs are covered by investors. If they move on to competing on having to do inference the cheepest then the TPUs will shine.
The original post made no comments about inference or training or even cost in any way. It said you could hook up more TPUs together with more memory and higher average bandwidth than you could with a datacenter of Nvidia GPUs. From an architectural point of view, it isn’t clear (and is not explained) what that enables. It clearly hasn’t led to a business outcome for Google where they are the clear market leader.
Seemingly fast interconnects benefit training more than inference since training can have more parallel communication between nodes. Inference for users is more embarrassingly parallel (requires less communication) than updating and merging network weights.
My point: cool benchmark, what does it matter? The original post says Nvidia doesn’t have anything to compete with massively interconnected TPUs. It didn’t merely say Google’s TPUs were better. It said that Nvidia can’t compete. That’s clearly bullshit and wishful thinking, right? There is no evidence in the market to support that, and no actual technical points have been presented in this thread either. OpenAI, Anthropic, etc are certainly competing with Google, right?
> My point: cool benchmark, what does it matter?
And then people explained why the effects are smoothed over right now but will matter eventually and you rejected them as if they didn't understand your question. They answered it, take the answer.
> It didn’t merely say Google’s TPUs were better. It said that Nvidia can’t compete.
Can't compete at clusters of a certain size. The argument is that anyone on nVidia simply isn't building clusters that big.
The fact that NVidia are currently winning is undisputed.
Yet everyone uses NVIDIA and Google is at catchup position.
Ecosystem is MASSIVE factor and will be a massive factor for all but the biggest models
Catch-up in what exactly? Google isn't building hardware to sell, they aren't in the same market.
Also I feel you completely misunderstand that the problem isn't how fast is ONE gpu vs ONE tpu, what matters is the costs for the same output. If I can fill a datacenter at half the cost for the same output, does it matters I've used twice the TPUs and that a single Nvidia Blackwell was faster? No...
And hardware cost isn't even the biggest problem, operational costs, mostly power and cooling are another huge one.
So if you design a solution that fits your stack (designed for it) and optimize for your operational costs you're light years ahead of your competition using the more powerful solution, that costs 5 times more in hardware and twice in operational costs.
All I say is more or less true for inference economics, have no clue about training.
Also, isn't memory a bit moot? At scale I thought that the ASICs frequently sat idle waiting for memory.
You're doing operations on the memory once it's been transferred to gpu memory. Either shuffling it around various caches or processors or feeding it into tensor cores or other matrix operations. You don't want to be sitting idle.
I think it's not about the cost but the limits of quickly accessible RAM
Ironwood is 192GB, Blackwell is 96GB, right? Or am i missing something?
182GB and B300 is 288GB. IIRC
I always enjoy being wrong and I was very wrong in my predictions about Google : I thought they should theoretically win, but I was also very confident they couldn't possibly turn their execution ship around to actually pull together a coherent competitor to OpenAI. But they do seem to have done that and it's very impressive. If they do continue to execute, I can't see anybody stopping them dominating and I would be bearish on nearly every other player catching them.
The biggest problem though is trust, and I'm still holding back from letting anyone under my authority in my org use Gemini because of the lack of any clear or reasonable statement or guidelines on how they use your data. I think it won't matter in the end if they execute their way to domination - but it's going to give everyone else a chance at least for a while.
> because of the lack of any clear or reasonable statement or guidelines on how they use your data.
They’ve been very clear, in my opinion: https://cloud.google.com/gemini/docs/discover/data-governanc...
I suppose there will always be the people who refuse to trust them or choose to believe they’re secretly doing something different.
However I’m not sure what you’re referring to by saying they haven’t said anything about how data is used.
This echos my experience with GCP and Google in general.
If you’re a business/enterprise, you get a different ToS that very clearly states that your data is yours.
If you use the free/consumer options, that’s where they are vague or direct about vacuuming up data.
The LLM provider I trust the most right now is AWS. Anybody else seems to have very conflicted purposes when it comes to sending them my data and interactions.
You're not wrong... but any space where Amazon, of all companies, has a shot at being the "most trustworthy player" is one I'm going to avoid where I can.
Amazon makes an LLM?
They make Amazon Q, but they don't make a general purpose LLM, which is precisely why I fid them more trustworthy. They run others' LLMs [1] , but don't have an interest in using customer data to improve the next generation of their LLM, or move into adjacent businesses to commercialize LLMs. At least not yet.
[1] https://aws.amazon.com/bedrock/model-choice/
I don’t know if Google will win but the message they sent is that Nvidia will not: if Google can release the best model without using their own chips instead of GPUs, and save money in the process, then others (Microsoft, Amazon, etc.) will do it too sooner or later.
That is the opposite of good. If only big companies can afford bespoke hardware, they literally become a monopoly, At home LLMs and other usages become either prohibitively expensive in purchase and running cost or you have to deal with a low quality model. Just admit that it's environmentally fucked and not open in any sense. This is not something that the community at large should strive for.
It doesn’t help when their thankgiving doodle that sends me to Gemini on how to plan making thanksgiving dinner on time completely fails in ridiculous ways
Unless they nerf Gemini 3.0 after a few weeks like they did with 2.5. Remember?
> If they do continue to execute
Yes, but Google will never be able to compete with their greatest challenge... Google's attention span.
> It is also important to note that, until recently, the GenAI industry’s focus has largely been on training workloads. In training workloads, CUDA is very important, but when it comes to inference, even reasoning inference, CUDA is not that important, so the chances of expanding the TPU footprint in inference are much higher than those in training (although TPUs do really well in training as well – Gemini 3 the prime example).
Does anyone have a sense of why CUDA is more important for training than inference?
NVIDIA chips are more versatile. During training, you might need to schedule things to the SFU(Special Function unit that does sin, cos, 1/sqrt(x), etc), you might need to run epilogues, save intermediary computations, save gradients, etc. When you train, you might need to collect data from various GPUs, so you need to support interconnects, remote SMEM writing, etc.
Once you have trained, you have frozen weights/feed-forward networks that consist out of frozen weights that you can just program in and run data over. These weights can be duplicated across any amount of devices and just sit there and run inference with new data.
If this turns out to be the future use-case for NNs(it is today), then Google are better set.
All of those are things you can do with TPUs
Won't the need to train increase as the need for specialized, smaller models increases and we need to train their many variations? Also what about models that continuously learn/(re)train? Seems to me the need for training will only go up in the future.
That's the thing - nobody knows. LLM architecture is constantly evolving and people are trying all kinds of things.
This is a very important point - the market for training chips might be a bubble, but the market for inference is much, much larger. At some point we might have good enough models and the need for new frontier models will cool down. The big power-hungry datacenters we are seeing are mostly geared towards training, while inference-only systems are much simpler and power efficient.
A real shame, BTW, all that silicon doesn't do FP32 (very well). After training ceases to be that needed, we could use all that number crunching for climate models and weather prediction.
it's already the case that people are eeking out most further gains through layering "reasoning" on top of what existing models can do - in other words, using massive amounts of inference to substitute for increases model performance. Whereever things plateau I expect this will still be the case - so inference ultimately will always be the end game market.
Some more traditional number crunching has long looked at lower- and mixed-precision hardware.
It's just more common as a legacy artifact from when nvidia was basically the only option available. Many shops are designing models and functions, and then training and iterating on nvidia hardware, but once you have a trained model it's largely fungible. See how Anthropic moved their models from nvidia hardware to Inferentia to XLA on Google TPUs.
Further it's worth noting that the Ironwood, Google's v7 TPU, supports only up to BF16 (a 16-bit floating point that has the range of FP32 minus the precision. Many training processes rely upon larger types, quantizing later, so this breaks a lot of assumptions. Yet Google surprised and actually training Gemini 3 with just that type, so I think a lot of people are reconsidering assumptions.
This is not the case for LLMs. FP16/BF16 training precision is standard, with FP8 inference very common. But labs are moving to FP8 training and even FP4.
When training a neural network, you usually play around with the architecture and need as much flexibility as possible. You need to support a large set of operations.
Another factor is that training is always done with batches. Inference batching depends on the number of concurrent users. This means training tends to be compute bound where supporting the latest data types is critical, whereas inference speeds are often bottlenecked by memory which does not lend itself to product differentiation. If you put the same memory into your chip as your competitor, the difference is going to be way smaller.
Training is taking an enormous problem and trying to break it into lots of pieces and managing the data dependency between those pieces. It's solving 1 really hard problem. Inference is the opposite, it's lots of small independent problems. All of this "we have X many widgets connected to Y many high bandwidth optical telescopes" is all a training problem that they need to solve. Inference is "I have 20 tokens and I want to throw them at these 5,000,000 matrix multiplies, oh and I don't care about latency".
I can't think of any case where inference doesn't care about latency.
I cant thinl of any reason training isnt going to become real time with a significant cpu budget.
I think it’s the same reason windows is inportant to desktop computers. Software was written to depend on it. Same with most of the software out there today to train being built around CUDA. Even a version difference of CUDA can break things.
CUDA is just a better dev experience. Lots of training is experiments where developer/researcher productivity matters. Googlers get to use what they're given, others get to choose.
Once you settle on a design then doing ASICs to accelerate it might make sense. But I'm not sure the gap is so big, the article says some things that aren't really true of datacenter GPUs (Nvidia dc gpus haven't wasted hardware on graphics related stuff for years).
That quote left me with the same question. Something about decent amount of ram on one board perhaps? That’s advantageous for training but less so for inference?
inference is often a static, bounded problem solvable by generic compilers. training requires the mature ecosystem and numerical stability of cuda to handle mixed-precision operations. unless you rewrite the software from the ground up like Google but for most companies it's cheaper and faster to buy NVIDIA hardware
> static, bounded problem
What does it even mean in neural net context?
> numerical stability
also nice to expand a bit.
This feels a lot like the RISC/CISC debate. More academic than it seems. Nvidia is designing their GPUs primarily to do exactly the same tasks TPUs are doing right now. Even within Google it's probably hard to tell whether or not it matters on a 5-year timeframe. It certainly gives Google an edge on some things, but in the fullness of time "GPUs" like the H100 are primarily used for running tensor models and they're going to have hardware that is ruthlessly optimized for that purpose.
And outside of Google this is a very academic debate. Any efficiency gains over GPUs will primarily turn into profit for Google rather than benefit for me as a developer or user of AI systems. Since Google doesn't sell TPUs, they are extremely well-positioned to ensure no one else can profit from any advantages created by TPUs.
Google does not sell them, but you can rent them:
https://cloud.google.com/tpu
As you note, they'll set the margins to benefit themselves, but you can still eke out some benefit.
Also, you can buy Edge TPUs, but as the name says these are for edge AI inference and useless for any heavy lifting workloads like training or LLMs.
https://www.amazon.com/Google-Coral-Accelerator-coprocessor-...
> Since Google doesn't sell TPUs, they are extremely well-positioned to ensure no one else can profit from any advantages created by TPUs.
First part is true at the moment, not sure the second follows. Microsoft is developing their own “Maia” chips for running AI on Azure with custom hardware, and everyone else is also getting in the game of hardware accelerators. Google is certainly ahead of the curve in making full-stack hardware that’s very very specialized for machine learning. But everyone else is moving in the same direction: lots of action is in buying up other companies that make interconnects and fancy networking equipment, and AMD/NVIDIA continue to hyper specialize their data center chips for neural networks.
Google is in a great position, for sure. But I don’t see how they can stop other players from converging on similar solutions.
I feel like this is more like the console/PC debate in the 90s. Consoles like the SNES had dedicated fixed function graphics hardware with weaker general specs, but with the special HW they could perform as well as a much more expensive PC - but as devs made more and more varied and clever games, that fixed function hardware couldn't support it and the PC became the superior choice.
I guess that's why Nintendo stopped making game consoles and Sony's PlayStation never went anywhere.
Consoles still exist, but they have evolved to use PC hardware.
This is highly relevant:
"Meta in talks to spend billions on Google's chips, The Information reports"
https://www.reuters.com/business/meta-talks-spend-billions-g...
Weird they'd do this after developing several generations of their own inference chip. Google is basically a competitor. This may just be a ploy to get better pricing from Nvidia.
keyword: "...talks..."
I don't think what the article writes about matters all that much. Gemini 3 Pro is arguably not even the best model anymore, and it's _weeks_ old, and Google has far more resources than Anthropic does. If the hardware actually was the secret sauce, Google would be wiping the floor with little everyone else.
But they're not.
There's a few confounding problems:
1. Actually using that hardware effectively isn't easy. It's not as simple as jacking up some constant values and reaping the benefits. Actually using the hardware is hard, and by the time you've optimized for it, you're already working on the next model.
2. This is a problem that, if you're not Google, you can just spend your way out of. A model doesn't take a petabyte of memory to train or run. Regular old H100s still mostly work fine. Faster models are nice, but Gemini 3 Pro being 50% of the latency as Opus 4.5 or GPT 5.1 doesn't add enough value to matter to really anyone.
3. There's still a lot of clever tricks that work as low hanging fruit to improve almost everything about ML models. You can make stuff remarkably good with novel research without building your own chips.
4. A surprising amount of ML model development is boots on the ground work. Doing evals. Curating datasets. Tweaking system prompts. Having your own Dyson sphere doesn't obviate a lot of the typing and staring at a screen that necessarily has to be done to make a model half decent.
5. Fancy bespoke hardware means fancy bespoke failure modes. You can search stack overflow for CUDA problems, you can't just Bing your way to victory when your fancy TPU cluster isn't doing the thing you want it to do.
I think you are addressing the issue from a developer's perspective. I don't think TPUs are going to be sold to individual users anytime soon. What the article is pointing out is that Google is now able to squeeze significantly more performance per dollar than their peer competitors in the LLM space.
For example, OpenAI has announced trillion-dollar investments in data centers to continue scaling. They need to go through a middle-man (Nvidia), while Google does not, and will be able to use their investment much more efficiently to train and serve their own future models.
> Google is now able to squeeze significantly more performance per dollar than their peer competitors in the LLM space
Performance per dollar doesn't "win" anything though. Performance (as in speed) hardly cracks the top five concerns that most folks have when choosing a model provider, because fast, good models already exist at price points that are acceptable. That might mean slightly better margins for Google, but ultimately isn't going to make them "win"
It's not slightly better margins, we are talking about huge cost reductions on the main expense which is compute. In a context where companies are making trillion dollar investments, it matters a lot.
Also, performance and user choice are definitely impacted by compute. If they ever find a way to replace a job with LLMs, those who can throw more compute at it for a lower price point will win.
Google owns 14% of Anthropic and Anthropic is using Google TPUs, as well as AWS Trainium and of course GPUs. It isn't necessary for one company to create both the winning hardware and the winning software to be part of the solution. In fact with the close race in software hardware seems like the better bet.
https://www.anthropic.com/news/expanding-our-use-of-google-c...
They are using that hardware to wipe the floor with everyone if you look at the price per million tokens.
Gemini3 is slightly more expensive than GPT5.1 for both input and output tokens though?
But price per token isn't even a directly important concern anymore. Anyone with a brain would pay 5x more per token for a model that uses 10x fewer tokens with the same accuracy. I've gone all in on Opus 4.5 because even though it's more expensive, it solves the problems I care about with far fewer tokens.
Which model is doing so?
_Weeks_ old! What a fossil!
Slightly more seriously: what you say makes sense if and only if you're projecting Sam Altman and assuming that a) real legit superhuman AGI is just around the corner, and b) all the spoils will accrue to the first company that finds it, which means you need to be 100% in on building the next model that will finally unlock AGI.
But if this is not the case -- and it's increasingly looking like it's not -- it's going to continue to be a race of competing AIs, and that race will be won by the company that can deliver AI at scale the most cheaply. And the article is arguing that company will be Google.
> _Weeks_ old! What a fossil!
I think you are missing the point. They are saying "weeks old" isn't very old.
> it's going to continue to be a race of competing AIs, and that race will be won by the company that can deliver AI at scale the most cheaply.
I don't see how that follows at all. Quality and distribution both matter a lot here.
Google has some advantages but some disadvantages here too.
If you are on AWS GovCloud, Anthropic is right there. Same on Azure, and on Oracle.
I believe Gemini will be available on the Oracle Cloud at some point (it has been announced) but they are still behind in the enterprise distribution race.
OpenAI is only available on Azure, although I believe their new contract lets them strike deals elsewhere.
On the consumer side, OpenAI and Google are well ahead of course.
> _Weeks_ old! What a fossil!
Last week it looked like Google had won (hence the blog post) but now almost nobody is talking about antigravity and Gemini 3 anymore so yeah what op says is relevant
Fairly certain google is aiming for "realtime" model training which would definitely require a new arcjitscture
I didn't doubt it, but I also don't think realtime model training makes them "win" anything.
"Gemini 3 Pro is arguably not even the best model anymore"
Arguably indeed, because I think it still is.
It definitely depends on how you're measuring. But the benchmarks don't put it at the top for many ways of measuring, and my own experience doesn't put it at the top. I'm glad if it works for you, but it's not even a month old and there are lots of folks like me who see it as definitely worse for classes of problems that 3 Pro could be the best at.
Which is to say, if Google was set up to win, it shouldn't even be a question that 3 Pro is the best. It should be obvious. But it's definitely not obvious that it's the best, and many benchmarks don't support it as being the best.
On point 5, I think this is the real moat for CUDA. Does Google have tools to optimize kernels on their TPUs? Do they have tools to optimize successive kernel launches on their TPUs? How easy is it to debug on a TPU(arguably CUDA could use work here but still...)? Does Google help me fully utilize their TPUs? Can I warm up a model on a TPU, checkpoint it, and launch the checkpoints to save time?
I am fairly pro-google(they invented the LLM, FFS...) and recognize the advantages(price/token, efficiency, vertical integration, established DCs w/ power allocations) but also know they have a habit of slightly sucking at everything but search.
A question I don't see addressed in all these articles: what prevents Nvidia from doing the same thing and iterating on their more general-purpose GPU towards a more focused TPU-like chip as well, if that turns out to be what the market really wants.
They will, I'm sure.
The big difference is that Google is both the chip designer *and* the AI company. So they get both sets of profits.
Both Google and Nvidia contract TSMC for chips. Then Nvidia sells them at a huge profit. Then OpenAI (for example) buys them at that inflated rate and them puts them into production.
So while Nvidia is "selling shovels", Google is making their own shovels and has their own mines.
on top of that Google is also cloud infrastructure provider - contrary to OpenAI that need to have someone like Azure plug those GPUs and host servers.
I am pretty sure OpenAI has data centers of its own.
The own shovels for own mines strategy has a hidden downside: isolation. NVIDIA sells shovels to everyone - OpenAI, Meta, xAI, Microsoft - and gets feedback from the entire market. They see where the industry is heading faster than Google, which is stewing in its own juices. While Google optimizes TPUs for current Google tasks (Gemini, Search), NVIDIA optimizes GPUs for all possible future tasks. In an era of rapid change, the market's hive mind usually beats closed vertical integration.
Aka vertical integration.
Selling shovels may still turn out to be the right move: Nvidia got rich off the cryptocurrency bubble, now they're getting even richer off the AI bubble.
Having your own mines only pays off if you actually do strike gold. So far AI undercuts Google's profitable search ads, and loses money for OpenAI.
> AI ... profits
Citation needed. But the vertical integration is likely valuable right now, especially with NVidia being supply constrained.
So when the bubble pops the companies making the shovels (TSMC, NVIDIA) might still have the money they got for their products and some of the ex-AI companies might least be able to sell standard compliant GPUs on the wider market.
And Google will end up with lots of useless super specialized custom hardware.
It seems unlikely that large matrix multipliers will become useless. If nothing else, Google uses AI extensively internally. It already did in ways that weren’t user-visible long before the current AI boom. Also, they can still put AI overviews on search pages regardless of what the stock market does. They’re not as bad as they used to be, and I expect they’ll improve.
Even if TPU’s weren’t all that useful, they still own the data centers and can upgrade equipment, or not. They paid for the hardware out of their large pile of cash, so it’s not debt overhang.
Another issue is loss of revenue. Google cloud revenue is currently 15% of their total, so still not that much. The stock market is counting on it continuing to increase, though.
If the stock market crashes, Google’s stock price will go down too, and that could be a very good time to buy, much like it was in 2008. There’s been a spectacular increase since then, the best investment I ever made. (Repeating that is unlikely, though.)
How could Google's custom hardware become useless? They've used it for their business for years now and will do so for years into the future. It's not like their hardware is LLM specific. Google cannot lose with their vast infrastructure.
Meanwhile OpenAI et al dumping GPUs while everyone else is doing the same will get pennies on the dollar. It's exactly the opposite to what you describe.
I hope that comes to pass, because I'll be ready to scoop up cheap GPUs and servers.
Same way cloud hardware always risks becoming useless. The newer hardware is so much better you can't afford to not upgrade, e.g. an algorithmic improvement that can be run on CUDA devices but not on existing TPUs, which changes the economics of AI.
> And Google will end up with lots of useless super specialized custom hardware.
If it gets to the point where this hardware is useless (I doubt it), yes Google will have it sitting there. But it will have cost Google less to build that hardware than any of the companies who built on Nvidia.
Right, and the inevitable bubble pop will just slow things down for a few years - it's not like those TPUs will suddenly be useless, Google will still have them deployed, it's just that instead of upgrading to a newer TPU they'll stay with the older ones longer. It seems like Google will experience much less repercussions when the bubble pops compared to Nvidia, OpenAI, Anthropic, Oracle etc. as they're largely staying out of the money circles between those companies.
And running loads long term profitable may require both lower power use as well as longer chip lifetimes - something associated with lower power use.
aka Google will have less of a pile of money than Nvidia will
Alphabet is the most profitable company in the world. For all the criticisms you can throw at Google, lacking a pile of money isn't one of them.
I think people are confusing the bubble popping with AI being over. When the dot-com bubble popped, it's not like internet infrastructure immediately became useless and worthless.
that's actually not all that true... a lot of fiber that had been laid went dark, or was never lit, and was hoarded by telecoms in an intentional supply constrained market in order to drive up the usage cost of what was lit.
If it was hoarded by anyone, then by definition not useless OR worthless. Also, you are currently on the internet if you're reading this, so the point kinda stands.
Are you saying that the internet business didn't grow a lot after the bubble popped?
And then they sold it to Google who lit it up.
Google uses TPUs for its internal AI work (training Gemini for example), which surely isn't decreasing in demand or usage as their portfolio and product footprint increases. So I have a feeling they'd be able to put those TPUs to good use?
Nvidia doesn't have the software stack to do a TPU.
They could make a systolic array TPU and software, perhaps. But it would mean abandoning 18 years of CUDA.
The top post right now is talking about TPU's colossal advantage in scaling & throughput. Ironwood is massively bigger & faster than what Nvidia is shooting for, already. And that's a huge advantage. But imo that is a replicateable win. Throw gobs more at networking and scaling and nvidia could do similar with their architecture.
The architectural win of what TPU is more interesting. Google sort of has a working super powerful Connection Machine CM-1. The systolic array is a lot of (semi-)independent machines that communicate with nearby chips. There's incredible work going on to figure out how to map problems onto these arrays.
Where-as on a GPU, main memory is used to transfer intermediary results. It doesn't really matter who picks up work, there's lots of worklets with equal access time to that bit of main memory. The actual situation is a little more nuanced (even in consumer gpu's there's really multiple different main memories, which creates some locality), but there's much less need for data locality in the GPU, and much much much much tighter needs, the whole premise of the TPU is to exploit data locality. Because sending data to a neighbor is cheap, sending storing and retrieving data from memory is slower and much more energy intense.
CUDA takes advantage of, relies strongly on the GPU's reliance in main memory being (somewhat) globally accessible. There's plenty of workloads folks do in CUDA that would never work on TPU, on these much more specialized data-passing systolic arrays. That's why TPUs are so amazing, because they are much more constrained devices, that require so much more careful workload planning, to get the work to flow across the 2D array of the chip.
Google's work on projects like XLA and IREE is a wonderful & glorious general pursuit of how to map these big crazy machine learning pipelines down onto specific hardware. Nvidia could make their own or join forces here. And perhaps they will. But the CUDA moat would have to be left behind.
> They could make a systolic array TPU and software, perhaps. But it would mean abandoning 18 years of CUDA.
Tensor cores are specialized and have CUDA support.
Tensor cores can help a lot for matrix maths, sure, definitely. They made a big splash in 2017 & have been essential. https://developer.nvidia.com/blog/programming-tensor-cores-c...
But it's still something grafted onto the existing architecture, of many grids with many blocks with many warps, and lots and lots of coordination and passing intermediary results around. It's only a 4x4x4 unit, afaik. There's still a lot of main memory being used to combine data, a lot of orchestration among the different warps and blocks and grids, to get big matrices crunched.
The systolic array is designed to allow much more fire and forget operations. It's inputs are 128 x 128 and each cell is its own compute node basically, shuffling data through and across (but not transitting a far off memory).
TPU architecture has plenty of limitations. It's not great at everything. But if you can design work to flow from cell to neighboring cell, you can crunch very sizable chunks of data with amazing data locality. The efficiency there is unparalleled.
Nvidia would need a radical change of their architecture to get anything like the massive data locality wins a systolic array can do. It would come with massively more constraints too.
Would love if anyone else has recommended reading. I have this piece earmarked. https://henryhmko.github.io/posts/tpu/tpu.html https://news.ycombinator.com/item?id=44342977
Deepmind gets to work directly with the TPU team to make custom modifications and designs specifically for deepmind projects. They get to make pickaxes that are made exactly for the mine they are working.
Everyone using Nvidia hardware has a lot of overlap in requirements, but they also all have enough architectural differences that they won't be able to match Google.
OpenAI announced they will be designing their own chips, exactly for this reason, but that also becomes another extremely capital intensive investment for them.
This also doesn't get into that Google also already has S-tier dataceters and datacenter construction/management capabilities.
Isn’t there a suspicion that OpenAI buying custom chips from another Sam Altman venture is just graft? Wasn’t that one of the things that came up when the board tried to out him?
The chips are being done in-house.
It was only brought in-house after the $5,000,000,000,000 self-dealing AI chip venture failed to launch.
Nvidia?
Sam Altman attempted to raise $5Tn for an AI-chip startup
Link? I only know of Rain and they raised <<$1 billion IIRC
Note that I said "attempted". https://www.wsj.com/tech/ai/sam-altman-seeks-trillions-of-do...
I think this was to build datacenters
> Deepmind gets to work directly with the TPU team to make custom modifications
You don't think Nvidia has field-service engineers and applications engineers with their big customers? Come on man. There is quite a bit of dialogue between the big players and the chipmaker.
They do, but they need to appease a dozen different teams from a dozen different labs, forcing nvidia to take general approaches and/or dictating approaches and pigeonholing labs into using those methods.
Deepmind can do whatever they want, and get the exact hardware to match it. It's a massive advantage when you can discover a bespoke way of running a filter, and you can get a hardware implementation of it without having to share that with any third parties. If OpenAI takes a new find to Nvidia, everyone else using Nvidia chips gets it too.
This ignores the way it often works: Customer comes to NVDA with a problem and NVDA comes up with a solution. This solution now adds value for every customer.
In your example, if OpenAI makes a massive new find they aren't taking it to NVDA.
Nvidia has the advantage of a broad base of customers that gives it a lot of information on what needs work and it tries to quickly respond to those deficiencies.
>In your example, if OpenAI makes a massive new find they aren't taking it to NVDA.
Right, and therefore they are stuck doing it in software, while google can do it in hardware.
It's not that the TPU is better than an NVidia GPU, it's just that it's cheaper since it doesn't have a fat NVidia markup applied, and is also better vertically integrated since it was designed/specified by Google for Google.
TPUs are also cheaper because GPUs need to be more general purpose whereas TPUs are designed with a focus on LLM workloads meaning there's not wasted silicon. Nothing's there that doesn't need to be there. The potential downside would be if a significantly different architecture arises that would be difficult for TPUs to handle and easier for GPUs (given their more general purpose). But even then Google could probably pivot fairly quickly to a different TPU design.
The T in TPU stands for tensor, which in this context is just a fancy matrix. These days both are optimised for matrix algebra, i.e. general ML workloads, not just LLMs.
If LLMs become unfashionable they’ll still be good for other ML tasks like image recognition.
Nothing in principle. But Huang probably doesn't believe in hyper specializing their chips at this stage because it's unlikely that the compute demands of 2035 are something we can predict today. For a counterpoint, Jim Keller took Tenstorrent in the opposite direction. Their chips are also very efficient, but even more general purpose than NVIDIA chips.
How is Tenstorrent h/w more general purpose than NVIDIA chips? TT hardware is only good for matmuls and some elementwise operations, and plain sucks for anything else. Their software is abysmal.
Of course there's the general purpose RISC V CPU controller component but also, each NPU is designed in troikas that have one core reading data in, one core performing the actual kernel work, and the third core forwarding data out.
For users buying H200s for AI workloads, the "ASIC" tensor cores deliver the overwhelming bulk of performance. So they already do this, and have been since Volta in 2017.
To put it into perspective, the tensor cores deliver about 2,000 TFLOPs of FP8, and half that for FP16, and this is all tensor FMA/MAC (comprising the bulk of compute for AI workloads). The CUDA cores -- the rest of the GPU -- deliver more in the 70 TFLOP range.
So if data centres are buying nvidia hardware for AI, they already are buying focused TPU chips that almost incidentally have some other hardware that can do some other stuff.
I mean, GPUs still have a lot of non-tensor general uses in the sciences, finance, etc, and TPUs don't touch that, but yes a lot of nvidia GPUs are being sold as a focused TPU-like chip.
Is it the Cuda cores that run the vertex/fragment/etc shaders in normal GPUs? Where does the ray tracing units fit in? How much of a modern Nvidia GPU is general purpose vs specialized to graphics pipelines?
A datacenter GPU has next to nothing left related to graphics. You can't use them to render graphics. It's a pure computational kernel machine.
That's exactly what Nvidia is doing with tensor cores.
Except the native width of Tensor Cores are about 8-32 (depending on scalar type), whereas the width of TPUs is up to 256. The difference in scale is massive.
I think Hopper's native matmul tile is 64x64, and Blackwell is 128x128.
see this blog for a reference on Blackwell:
https://hazyresearch.stanford.edu/blog/2025-03-15-tk-blackwe...
If it turns out to be useful, Nvidia can't just tweak a parameter in their verilog and declare victory?
If not, what's fundamentally difficult about doing 32 vs 256 here?
Nobody cares about width; they care about TFLOPs.
That's pretty much what they've been doing incrementally with the data center line of GPUs versus GeForce since 2017. Currently, the data center GPUs now have up to 6 times the performance at matrix math of the GeForce chips and much more memory. Nvidia has managed to stay one tape out away from addressing any competitors so far.
The real challenge is getting the TPU to do more general purpose computation. But that doesn't make for as good a story. And the point about Google arbitrarily raising the prices as soon as they think they have the upper hand is good old fashioned capitalism in action.
the entire organisation has been built over the last 25 years to produce GPUs
turning a giant lumbering ship around is not easy
For sure, I did not mean to imply they could do it quickly or easily, but I have to assume that internally at Nvidia there's already work happening to figure out "can we make chips that are better for AI and cheaper/easier to make than GPUs?"
Isn't that a bit like Kodak knowing that digital cameras were a thing but not wanting to jeopardize their film business?
It’s not binary. It’s not existential. What’s at stake for Nvidia is its HUGE profit margins. 5 years from now, Nvidia could be selling 100x as many chips. But its market cap could be a fraction of what it is now if competition is so intense that its making 5% profit margin instead of 90%.
More like 900% right now.
> what prevents Nvidia from doing the same thing and iterating on their more general-purpose GPU towards a more focused TPU-like chip as well, if that turns out to be what the market really wants.
Nothing prevents them per se, but it would risk cannibalising their highly profitable (IIRC 50% margin) higher end cards.
My personal guess would be what drives the cost and size of these chips is the memory bandwidth and the transcievers required to support it. Since transcievers/memory controllers are on the edge of the chip, you get a certain minimum circumference for a given bandwidth, which determines your min surface area.
It might be even 'free' to fill it with more complicated logic (especially one that allows you write clever algorithms that let you save on bandwidth).
They lose the competitive advantage. They have nothing more to offer than what Google has in-house.
I have never understood why, in these discussions, nobody brings up other specialized silicon providers like Groq, SambaNova, or my personal favorite, Cerebras.
Cerebras CS-3 specs:
• 4 trillion transistors
• 900,000 AI cores
• 125 petaflops of peak AI performance
• 44GB on-chip SRAM
• 5nm TSMC process
• External memory: 1.5TB, 12TB, or 1.2PB
• Trains AI models up to 24 trillion parameters
• Cluster size of up to 2048 CS-3 systems
• Memory B/W of 21 PB/s
• Fabric B/W of 214 Pb/s (~26.75 PB/s)
Comparing GPU to TPU is helpful for showcasing the advantages of the TPU in the same way that comparing CPU to Radeon GPU is helpful for showcasing the advantages of GPU, but everyone knows Radeon GPU's competition isn't CPU, it's Nvidia GPU!
TPU vs GPU is new paradigm vs old paradigm. GPUs aren't going away even after they "lose" the AI inference wars, but the winner isn't necessarily guaranteed to be the new paradigm chip from the most famous company.
Cerebras inference remains the fastest on the market to this day to my knowledge due to the use of massive on-chip SRAM rather than DRAM, and to my knowledge, they remain the only company focused on specialized inference hardware that has enough positive operating revenue to justify the costs from a financial perspective.
I get how valuable and important Google's OCS interconnects are, not just for TPUs or inference, but really as a demonstrated PoC for computing in general. Skipping the E-O-E translation in general is huge and the entire computing hardware industry would stand to benefit from taking notes here, but that alone doesn't automatically crown Google the victor here, does it?
I have read in the past that ASICs for LLMs are not as simple a solution compared to cryptocurrency. In order to design and build the ASIC you need to commit to a specific architecture: a hashing algorithm for a cryptocurrency is fixed but the LLMs are always changing.
Am I misunderstanding "TPU" in the context of the article?
LLMs require memory and interconnect bandwidth so needs a whole package that is capable of feeding data to the compute. Crypto is 100% compute bound. Crypto is a trivially parallelized application that runs the same calculation over N inputs.
Regardless of architecture (which is anyways basically the same for all LLMs), the computational needs of modern neural networks are pretty generic, centered around things like matrix multiply, which is what the TPU provides. There is even TPU support for some operations built into PyTorch - it is not just a proprietary interface that Google use themselves.
It’s true that architectures change, but they are built from common components. The most important of those is matrix multiplication, using a relatively small set of floating point data types. A device that accelerates those operations is, effectively, an ASIC for LLMs.
We used to call these things DSPs
This is quite accurate considering Google TPUs are VLIW machines.
What is the difference between a DSP and Asic? Is a GPU a DSP?
DSP is simply a compute architecture that focuses on mutliply and accumulate operations on particular numerical formats, often either fixed point q15/q31 type values or floats f16/f32.
The basic operation that a NN needs accelerating is... go figure multiply and accumulate with the added activation function.
See for example how the Intel NPU is structured here: https://intel.github.io/intel-npu-acceleration-library/npu.h...
A DSP contains analog to digital and digital to analog converters plus DMA for fast transfers to main memory and fixed function blocks for finite impulse response and infinite pulse response filters.
The fact that they also support vector operations or matrix multiplication is kind of irrelevant and not a defining characteristic of DSPs. If you want to go that far, then everything is a DSP, because all signals are analog.
I've worked on DSP's with none of those things. Well, they did have DMA.
See here https://intel.github.io/intel-npu-acceleration-library/npu.h...
Maybe also note that Qualcomm has renamed their Hexagon DSP to Hexagon NN. Likely the change was adding activation functions but otherwise its a VLIW architecture with accelerated MAC operations, aka a DSP architecture.
ASICs bake one algorithm into the chip. DSPs are programmable, like GPUs or CPUs. The thing that historically set them apart were MAC/FMA and zero overhead loops. Then there are all the nice to haves, like built in tables of FFT twiddle factors, helpers for 1D convolution, vector instructions, fixed point arithmetic, etc.
What makes a DSP different from a GPU is the algorithms typically do not scale nicely to large matrices and vectors. For example, recursive filters. They are also usually much cheaper and lower power, and the reason they lost popularity was because Arm MCUs got good enough and economy of scale kicked in.
I've written code for DSPs both in college and professionally. It's much like writing code for CPUs or MCUs (it's all C or C++ at the end of the day). But it's very different from writing compute shaders or designing an ASIC.
"Application-specific" doesn't necessarily mean unprogrammable. Bitcoin miners aren't programmable because they don't need to be. TPUs are ASICs for ML and need to be programmable so they can run different models. In theory, you could make an ASIC hardcoded for a specific model, but given how fast models evolve, it probably wouldn't make much economic sense.
Cryptocurrency architectures also change - Bitcoin is just about the lone holdout that never evolves. The hashing algorithm for Monero is designed so that a Monero hashing ASIC is literally just a CPU, and it doesn't even matter what the instruction set is.
If Google won, it would cannibalize its current ad-driven business and replace it with something that is extremely expensive to run and difficult to make profit from. A Pyrrhic win essentially.
Hardly a Pyrrhic win. When the rest of the market is burning money, whoever burns money the slowest while still remaining competitive will win.
But that would happen regardless of who won, better to at least dominate the new paradigm and figure out how to extract value from it. I also suspect that once the value generation is figured out they will cease offering these APIs to anyone, if you had a golden goose would you rent it?
They could go all dark mirror and inject ads directly into the responses.
I mean, focus is a thing that Google has always struggled with. But I kind of doubt that customers who need online marketing (ads) are going to convert en masse to users who rent cloud TPUs instead.
- ASIC won the crypto mining battle in the past, it's orders of magnitude faster
- Google is not owning the technology but builds a cohesive cloud around it, Tesla, Meta work on their own asic ai chips and I guess others
- A signal is already given: Softbank sold it's entire Nvidia stock and berkshire added google on their portfolio.
Microsoft "has" a lot of companies data, and google is probably building the most advanced ai cloud.
However, I can't think they had a cloud which was light-years ahead of aws 15 years ago and now GCP is no 3, they also released opensource gpt models more than 5 years ago that constituted the foundation for openai closed sourced models.
Given the importance of scale for this particular product, any company placing itself on "just" one layer of the whole story is at a heavy disadvantage, I guess. I'd rather have a winning google than openai or meta anyway.
> I'd rather have a winning google than openai or meta anyway.
Why? To me, it seems better for the market, if the best models and the best hardware were not controlled by the same company.
I agree, it would be the best of bad cases, in a sense. I have low trust in OpenAI due to its leadership, and in Meta, because, well, Meta has history, let's say.
I think you are disagreeing.
Yep :)
5 days ago: https://news.ycombinator.com/item?id=45926371
Sparse models have same quality of results but have less coefficients to process, in case described in the link above sixteen (16) times as less.
This means that these models need 8 times less data to store, can be 16 and more times faster and use 16+ times less energy.
TPUs are not all that good in the case of sparse matrices. They can be used to train dense versions, but inference efficiency with sparse matrices may be not all that great.
TPUs do include dedicated hardware, SparseCores, for sparse operations.
https://docs.cloud.google.com/tpu/docs/system-architecture-t...
https://openxla.org/xla/sparsecore
SparseCores appear to be block-sparse as opposed to element-sparse. They use 8- and 16-wide vectors to compute.
Here's another inference-efficient architecture where TPUs are useless: https://arxiv.org/pdf/2210.08277
There is no matrix-vector multiplication. Parameters are estimated using Gumbel-Softmax. TPUs are of no use here.
Inference is done bit-wise and most efficient inference is done after application of boolean logic simplification algorithms (ABC or mockturtle).
In my (not so) humble opinion, TPUs are example case of premature optimization.
They are on their 7th generation now, so presumably the architecture is being updated as needs require.
The part that surprised me is how much TPUs gain from the systolic array design. It basically cuts down the constant memory shuffling that GPUs have to do, so more of the chip’s time is spent actually computing.
The downside is the same thing that makes them fast: they’re very specialized. If your code already fits the TPU stack (JAX/TensorFlow), you get great performance per dollar. If not, the ecosystem gap and fear of lock-in make GPUs the safer default.
At this stage, it is somewhat clear that it doesn't really matter who's ahead in the race, cause everyone else is super close behind...
If an outsider is ever allowed to have a simplified mental model of why nvidia was unbeatable, here's mine:
- they were way ahead, and they didn't make any big mistakes
- they weren't waiting for others to catch up. They were aggressively improving
- memory bandwidth is almost always the bottleneck. Hence systolic array is "overrated". Furthermore, interconnect is the new bottleneck now
- cuda offers the most flexibility in the world of ever changing model requirements
Will Google sell TPUs that can be plugged into stock hardware, or custom hardware with lots of TPUs? Our customers want all their video processing to happen on site, and don't want their video or other data to touch the cloud, so they're not happy about renting cloud TPUs or GPUs. Also it would be nice to have smart cameras with built-in TPUs.
Why don't your customers trust Google Cloud?
It's not Google Cloud per se, it's any cloud. There are a million reasons not to trust (or spend money on) any cloud. They want all their video and data on premises and completely under their control.
Well, there's also an ever increasing list of reasons not to trust anything from Google. They're even pretty similar reasons.
With its AI offerings, can Google suck the oxygen out of AWS? AWS grew big because of compute. The AI spend will be far larger than compute. Can Google launch AI/Cloud offerings with free compute bundled? Use our AI, and we'll throw in compute for free.
Interesting thought. If that strategy worked too well, I could see the government going after them for monopolistic "bundling"
The funniest thing about this story is that NVIDIA has essentially become a TPU company. Look at the Hopper and Blackwell architectures: Tensor Cores are taking up more space, the Transformer Engine has appeared, and NVLink has started to look like a supercomputer interconnect. Jensen Huang isn't stupid. He saw the threat of specialized ASICs and just built the ASIC inside the GPU. Now we have a GPU that is 80% matrix multiplier but still keeps CUDA compatibility. Google tried to kill the GPU, but instead forced the GPU to mutate into a TPU
There's an issue with building a swiss knife chip that supports everything back to the 80s, it works great until it doesn't (Intel)
It's a cool subject and article and things I only have a general understanding of (considering the place of posting).
What I'm sure about is having a programming unit more purposed to a task is more optimal than a general programming unit designed to accommodate all programming tasks.
More and more of the economics of programming boils down to energy usage and invariably towards physical rules, the efficiency of the process has the benefit of less energy consumed.
As a Layman is makes general sense. Maybe a future where productivity is based closer on energy efficiency rather than monetary gain pushes the economy in better directions.
Cryptocurrency and LLMs seem like they'll play out that story over the next 10 years.
You can't really buy a TPU, you have to buy the entire data center that includes the TPU plus the services and support. In Google Colab, I often don't prefer the TPU either because the documentation for the AI isn't made for it. While this could all change in the long term, I also don't see these changes in Google's long term strategy. There's also the problem with Google's graveyard which isn't mentioned in the long term of the original article. Combined with these factors, I'm still skeptical about Google's lead on AI.
How much of current GPU and TPU design is based around attn's bandwith hungry design? The article makes it seem like TPUs aren't very flexible so big model architecture changes, like new architectures that don't use attn, may lead to useless chips. That being said, I think it is great that we have some major competing architectures out there. GPUs, TPUs and UMA CPUs are all attacking the ecosystem in different ways which is what we need right now. Diversity in all things is always the right answer.
Google has always had great tech - their problem is the product or the perseverance, conviction, and taste needed to make things people want.
This is a bizarre argument to make for AI, since Google started working on TPUs in 2013 (12 years ago) and Sundar started publicly banging on about being an AI-first company in 2016. They missed the first boat on LLMs, but Google has been invested in AI for way longer than any of the competition.
https://aibusiness.com/companies/google-ceo-sundar-pichai-we...
Their incentive structure doesn't lead to longevity. Nobody gets promoted for keeping a product alive, they get promoted for shipping something new. That's why we're on version 37 of whatever their chat client is called now.
I think we can be reasonably sure that search, Gmail, and some flavor of AI will live on, but other than that, Google apps are basically end-of-life at launch.
It's also paradoxically the talent in tech that isolates them. The internal tech stack is so incredibly specialized, most Google products have to either be built for internal users or external users.
Agree there are lots of other contributing causes like culture, incentives, security, etc.
haha remember Steve Yegge's Platform Rant? [1]
nothing changed...
[1] Ref: he mistakenly posted what was meant to be an internal memo, publicly on G+. He quickly took it down but of course The Internet Never Forgets https://gist.github.com/chitchcock/1281611
It's telling that basically all of Google's successful projects were either acquisitions or were sponsored directly by the founders (or sometimes, were acquisitions that were directly sponsored by the founders). Those are the only situations where you are immune from the performance review & promotion process.
They've actually had many very successful projects that make the few products and acquisitions you are thinking of work. It's true most of their end products don't work or get abandoned but it stretches their infrastructure in ways that works out well in the long run
I should probably have said "products" rather than "projects". There's a fair bit of extremely good engineering that goes on in the infrastructure side, but when it comes to consumer products, if one of the founders isn't explicitly sponsoring it it gets killed.
They get killed since ROI of putting more people on search or ads is almost always higher than ROI on the new projects, not because the founders doesn't like them.
Google released their latest chat app 8 years ago.
Odd way to describe the most used product in the history of the world.
Fuschia or me?
> The GPUs were designed for graphics [...] However, because they are designed to handle everything from video game textures to scientific simulations, they carry “architectural baggage.” [...] A TPU, on the other hand, strips away all that baggage. It has no hardware for rasterization or texture mapping.
With simulations becoming key to training models doesn't this seem like a huge problem for Google?
Any chance of a bit of support for jax-metal, or incorporating apple silicon support into Jax?
deepseek kind of innovated on this using off-the-shelf components right ?
to quote from their paper "In order to ensure sufficient computational performance for DualPipe, we customize efficient cross-node all-to-all communication kernels (including dispatching and combining) to conserve the number of SMs dedicated to communication. The implementation of the kernels is codesigned with the MoE gating algorithm and the network topology of our cluster."
I wish we had more options for a dedicated/stand-alone TPU for end users. I recently bought a 2019 Coral, which as far as I know is my only option.
Coral really has little to do with modern TPUs.
That and the fact they can self-fund the whole AI venture and don't require outside investment.
The most fun fact about all the developments post-ChatGPT is that people apparently forgot that Google was doing actual AI before AI meant (only) ML and GenAI/LLMs, and they were top players at it.
Arguably main OpenAI raison d'être was to be a counterweight to that pre-2023 Google AI dominance. But I'd also argue that OpenAI lost its way.
And they forgot to pay those people so most of them left.
To be fair, they weren't increasing Ads revenue.
They literally gave away their secret sauce to OpenAI and pretended like it wasn’t a big opportunity.
Just as expected from a big firm with slower organizational speed. They can afford to make those mistakes.
That and they were harvesting data way before it was cool, and now that it is cool, they're in a privileged position since almost no-one can afford to block GoogleBot.
They do voluntarily offer a way to signal that the data GoogleBot sees is not to be used for training, for now, and assuming you take them at their word, but AFAIK there is no way to stop them doing RAG on your content without destroying your SEO in the process.
But they also collect the data without causing denial of service, and respect robots.txt, which is more than you can say of most LLM scrapers...
Do people still get organic search traffic from google?
Wow, they really got folks by the short hairs if that is true...
All this assumes that LLMs are the sole mechanism for AI and will remain so forever: no novel architectures (neither hardware nor software), no progress in AI theory, nothing better than LLMs, simply brute force LLM computation ad infinitum.
Perhaps the assumptions are true. The mere presence of LLMs seems to have lowered the IQ of the Internet drastically, sopping up financial investors and resources that might otherwise be put to better use.
That's incorrect. TPUs can support many ML workloads, they're not exclusive to LLMs.
TPUs predate LLMs by a long time. They were already being used for all the other internal ML work needed for search, youtube, etc.
It's all about the right mind set at the very top level. At the beginning of the PC era, nobody would bet IBM to lose. Same in the dawn of internet, all money was on MS. so it happened to Nokia and Ericsson.
Google is a giant without a direction. The ads money is so good that it just doesn't have the gut to leave it on the table.
A Tensor-based PCI adapter card?
TAKE MY MONEY!!!
Yes, but what's at the finish line? The bottom?
Then Groq should reign emperor?
[dead]
[dead]
Right because people would love to get locked into another even more expensive platform.
That's mentioned in the article, but is the lock-in really that big? In some cases, it's as easy as changing the backend of your high-level ML library.
That is like how every ORM promises you can just swap out the storage layer.
In practice it doesnt quite work out that way.
That's what it is on paper. But in practice you trade one set of hardware idiosyncrasies for another and unless you have the right people to deal with that, it's a hassle.
On top, when you get locked into Google Cloud, you’re effectively at the mercy of their engineers to optimize and troubleshoot. Do you think Google will help their potential competitors before they help themselves? Highly unlikely considering their actions in the past decade plus.
Given my Fitbit's inability to play nice with my pixel phone, I have zero faith in Google engineers.
What else would one expect when their core value is hiring generalists over specialists* and their lousy retention record?
*Pay no attention to the specialists they acquihire and pay top dollar... And even they don't stick around.
I thin k you can only run on google cloud not aws bare metal azure etc
https://killedbygoogle.com
That's actually one of the reasons why Google might win.
Nvidia is tied down to support previous and existing customers while Google can still easily shift things around without needing to worry too much about external dependencies.
It's all small products which didn't receive traction.
Google Hangouts wasn't small. Google+ was big and supposedly "the future" and is the canonical example of a huge misallocation of resources.
Google will have no problem discontinuing Google "AI" if they finally notice that people want a computer to shut up rather than talk at them.
> Google+ was big
how you define big? My understanding they failed to compete with facebook, and decided to redirect resources somewhere else.
At the time Google+ was started and shortly after, leadership (larry page at that time) focused the attention of the company on it. There was a social bonus (that you'd get if you integrated your product), there were large changes to existing systems to support Google+, and the company made it quite clear it thought that social was the direction to go and that Google+ was going to be an enormous product.
I and a lot of other googlers were really confused by all of this because at the time we were advocating that Google put more effort into its nascent cloud business (often to get the reply "but we already have appengine" or "cloud isn't as profitable as ads") and that social, while getting a lot of attention, wasn't really a good business for google to be in (with a few exceptions like Orkut and Youtube, Google's attempts at social have been pretty uninspired).
There were even books written at the time that said Google looked lazy and slow and that Meta was going to eat their lunch. But shortly after Google+ tanked, Google really began to focus on Cloud (in a way that pissed off a lot of Googlers in the same way Google+ did- by taking resources and attention from other projects). Now, Meta looks like its going to have a challenging future while Google is on to achieving what Larry Page originally intended: a reliable revenue stream that is reinvested into development of true AI.
Google completely fumbled Google+ by doing a slow invite only launch.
The hype when it was first coming to market was intense. But then nobody could get access because they heavily restricted sign ups.
By the time it was in "open beta" (IIRC like 6-7 mos later), the hype had long died and nobody cared about it anymore.
In my recollection, what killed g+ was forcing your YouTube account to become your g+ account, with your public name attached to the trashpit YouTube comments used to be. Everybody protested using g+, but the "Google account for everything" stuck around anyways.
They put a lot of effort into it, but it never had much usage.
Orkut was HUGE in Brazil.
In Finnish, it pretty much means orgasms, so it wasn't. Though that's probably not the sole reason, of course.
It's not though. Chromecast, g suite legacy, podcast, music, url shortener,... These weren't small products.
Chromecast is "gone" because it bridged the gap of dumb tvs needing streaming capabilities. Now almost every tv sold has some kind of smart feature or can stream natively so Chromecast aren't needed.
chromecast is alive, podcast, music were migrated to youtube app, url shortener is not core business and just side hustle for google. Not familiar with g suite legacy.
Wait until Apple's ChromeBook competitor shows up to eat their lunch just like switching to another proprietary stack with no dev ecosystem will die out. Sure they'll go after big ticket accounts, also take a guess at what else gets sanctioned next.
Isn't an iPad with a keyboard or the air essentially a Chromebook competitor?
The only lunch that will be eaten is Apple's own, since it would probably cannibalize their own sales of the MacBook air
This is the “Microsoft will dominate the Internet” stage.
The truth is the LLM boom has opened the first major crack in Google as the front page of the web (the biggest since Facebook), in the same way the web in the long run made Windows so irrelevant Microsoft seemingly don’t care about it at all.
Exactly, ChatGPT pretty much ate away ad volume & retention if th already garbage search results weren't enough. Don't even get me started on Android & Android TV as an ecosystem.
That's not the story that GOOGs quarterly earning reports tell(ad revenue up 12% YoY)
most likely because they got more aggressive with campaign against adblock in chrome and more ads in youtube.
They can only privatize the AI race.
If Google wins, we all lose.
[dead]
How high are the chances that as soon as China produces their own competitive TPU/GPU, they'll invade Taiwan in order to starve the West in regards to processing power, while at the same time getting an exclusive grip on the Taiwanese Fabs?
China will invade Taiwan when they start losing, not when they're increasingly winning.
As long as "tomorrow" is a better day to invade Taiwan than today is, China will wait for tomorrow.
Their demographics beg to differ.
If demographics were a big deal, it'd be part of the same "better to invade today or tomorrow" calculation.
Zeihan's predictions on China have been fabulously wrong for 20+ years now.
The US would destroy TSMC before letting China have it. China also views military conquest of Taiwan as less than ideal for a number of reasons, so I think right now it's seen as a potential defensive move in the face of American aggression.
Not very. Those fabs are vulnerable things, shame if something happens to them. If China attacks, it would be for various other reasons and processors are only one of many considerations, no matter how improbable it might sound to an HN-er.
What if China becomes self-sufficient enough to no longer rely on Taiwanese Fabs, and hence having no issues with those Fabs getting destroyed. That would put China as the leader once and for all.
First, the US has advanced fab capabilities and in case of a need can develop them further. On the other side, China will suffer a Russia style blockback while caught up in a nasty war with Taiwan.
Totally possible, but the second order effects are much more complex than "leader once for all". The path for victory for China is not war despite the west, but a war when the west would not care.
The best path for victory for China is probably no war at all. War is wasteful and risky.
In a world where those starting wars would suffer their consequences the most, wars would be a bad idea. This is not such a world.
I don't agree.
Even the US suffers (their veterans do, anyway) but that's the country that in general least suffers from their constant involvement in warfare. They have this industry down to a "T". However, you cannot generalize from a nation that can project military force almost anywhere in the globe, with little fear of repercussion back home; most countries cannot afford this. China certainly cannot.
So what about the rest? Internecine conflicts are outrageously wasteful, and sadly common in the modern age. Russia's war with Ukraine has turned incredibly wasteful and costly, and Russians are suffering (and dying) regardless of whatever Putin says.
I think China is not generally oriented towards waging war. They do have their military, military projects, and their nationalistic things (what I learn from Wikipedia is called "irredentism"), but generally they seem to be trying to become an economic world power. War would mess and interfere with that. War is too fucking risky.
If they have the fabs but ASML doesn't send them their new machines, they will just end up in the same situation as now, just one generation later. If China wants to compete, they need to learn how to make the EUV light and mirrors.
Seems low at the moment with the concept of G2 being floated as generic understanding of China's ascension to where Russia used to be effectively recreating bipolar semi cold war world order. Mind, I am not saying impossible, but there are reasons China would want to avoid this scenario ( probably one of the few things US would not tolerate and would likely retaliate ).
Imo having the best logic process nodes is not necessary to win at AI - having the most memory bandwidth is - and China has SOTA HBMs.
I'd guess most of their handicap comes from their hardware and software not being as refined as the US's
The fabs would be destroyed in such a situation. The wesr would absolutely play that card in negotiations.
Highly unlikely. Despite the rampant anti-Chinese FUD that's so prevalent in the media (and, sadly, here on HN), China isn't really in the habit of invading other lands.
The plot twist here is that China doesn't view Taiwan as foreign.
But China also doesn't see war as the best path forward in Taiwan (they want to return it to the mainland, not lay waste to it). The grandparent comment is unfairly downvoted in my opinion, the fact remains modern China is far less likely to be involved in military campaigns than, say, the US.
In my 20+ years of following NVIDIA, I have learned to never bet against them long-term. I actually do not know exactly why they continually win, but they do. The main issue they have a 3-4 year gap between wanting a new design pivot and realizing it (silicon has a long "pipeline"), it can seem that they may be missing a new trend or swerve in the demands of the market, it is often simply because there is this delay.
You could have said the same thing about Intel for ~50 years.
Depends on the top management though. I imagine Nvidia will keep doing well while Jensen Huang is running things.
Fair, but the 75% margins can be reduced to 25% with healthy competition. The lack of competition in the frontier chips space was always the bottleneck to commoditization of computation, if such a thing is even possible
Turkeys bet on tomorrow 364 days of the year.
I told you a thousand times, you have to sell your pumpkin stock before Halloween, before!