Oracle's recent force majeure invocation shines a spotlight on what seems to be an endemic problem: datacenter builders have been signing commitments to bring a certain amount of capacity online by a certain date, without first conducting sufficient due diligence about whether they would be able to secure all the resources needed to meet those commitments.
This means they are now caught in a mad scramble to source all sorts of stuff, including RAM, potentially as a matter of life and death.
What's less clear to me, though, is whether datacenter builders will be able to make good on these purchase agreements. It seems these projects have every opportunity to fail, and the path to success keeps getting narrower. What are the odds that chip manufacturers ultimately get caught holding a bag of chips that nobody can buy in one hand, and a bag of IOUs from bankrupt customers in the other?
Yeah there was just a story about how there are hundreds of billions of dollars in "non-cancellable" debt obligations to build AI data centers. But they're building far beyond current demand, I feel like the collapse is inevitable.
I'm sensing sarcasm but is it not true that the scale of obligations is unsustainable?
As for preparation, I'm not sure. I have a 401k that I'm worried about. I'm too young to really remember the dotcom bubble, let alone have any stake in the stock market at the time.
When the bubble bursts it will depress valuations across the board, including the valuations of the good companies that will survive to eventually own significant market share. Buy those at those depressed prices and wait a decade.
The trick will be learning to tell the difference between the Google's and the pet.com's of the AI era. The trick during the dotcom situation was to look for companies that had actual gross profit and were reinvesting it, rather than companies that only had theoretical profit based on nonsense like market share and eyeballs.
Take Amazon as an example. They were $107 a share in 1999. By late 2001, it had fallen to under $6. Now they are $250. It still took almost 10 years to recover.
Is there anyone in the AI sector that is making profit and re-investing it? I guess that really only applies to Microsoft, Apple, Google, etc. that other lines of business to prop up their AI divisions, vs. companies like OpenAI and Anthrophic. I wonder where NVIDIA will land...
Google is making money on it. However, that's coming at the expense of the very free web they built it from. So they have a different problem that may be unsolvable.
Worst part is that some of the companies that survived the dot com era are the new "...but on the internet" companies except now it's "...but with chat bot"
It's half sarcasm, half my frustration in the way ai crash proponents behave in general
if you proclaim that there's an inevitable doom - propose preparations *people you're talking to* can take
when I hear about nuclear doom, I see preppers discussing bunkers and how rich people buy land in Argentina to evacuate to
when I hear about stock market crash, I see people discussing some famous super-investor moving his money into coca-cola, because food would always have a demand
even bloody "the end is nigh" apocalypse junkies continue with "go to church" (or smth similar) in the very next sentence
what can YOU suggest ME as the next action? not for the government to do something, not for the whole market - how do you suggest ME to prepare? what do you suggest *I* will experience?
at best, our approximation of ai bust is the dotcom - and that happened beyond current generation's memory. At best surviving context is "and the ruin left a lot of cheap fiber laid out all around the world", so... we'll get a lot of cheap ai then? good
Nobody Knows. The market can remain irrational longer than you can remain solvent.
Having cash in reserve to deploy in the aftermath of a crash is golden. If that happens you look great, if that doesn't happen for the next 5 years, you miss out on the interim gains.
Keep investing, diversify, have some reserves. Try to be less anxious about things. Exercise, sleep, be grateful for things you have.
Not every inevitability can be mitigated. We're each and every one going to die. The sun will die. The universe will wind down. Embrace it, don't pretend you can avoid it.
Except we don't get cheap AI this go round. It's just a bunch of e-waste that can't even be repurposed into server in the closet back at the office because each server needs about 100kW of power.
There are signs that we are, and are not, in an AI bubble.
Compared to the dotcom boom, firms are well positioned in terms of capital - they have revenue and profits.
However, there's still the absurd speculative valuations, risk, and high expenditure (on said data centres). Also, Andrew Bailey from the Bank of England, personal opinion - https://www.bbc.co.uk/news/articles/cv8e30enrkxyo
The real problem is not that they are buying all the hardware, but that they are buying it and keeping it in bloody warehouses without bringing it online. Because they haven't got DC capacity for it.
Has anyone else here been trying to rent some cloud gpus for reasonable money? I have. I have signed up to a dozen "providers" that advertise everything from tesla cards to b200 only to find stuff is either out of capacity or double/triple the price.
Remember when software was eating the world? The gravy train ran out on commodity hardware and now we are paying the price for almost 3 decades of underinvestment.
My health insurance company recently rolled out a major revamp of their website, and it now uses GiB as if they were KiB.
At my knitting group last night one of the conversation topics was how everyone's computer feels really unstable these days. One person stopped volunteering at a local charity because, after a recent website rewrite, she has too much trouble with the page locking up when she's trying to sign up for hours.
Last weekend I got talking to another parent at my kid's gymnastics class about how her computer's really struggling since they rolled out some "modernization" rewrites of internal business applications, and it's making it hard to get her work done.
What I'm getting at is, if we do have to get used to memory being more expensive, that might be very disruptive. Programmers will have to do a major about-face on how they've been writing software in the post-AI era.
On the other hand, memory optimization is so much easier with AI. Profiling an app, reworking a function to be twice as long but more efficient, A/B testing libraries across browsers to check which uses the least extra memory. All of these things are fairly easy but so tedious, which makes them perfect for AI.
> What I'm getting at is, if we do have to get used to memory being more expensive, that might become very disruptive because software developers will have to do a major about-face on how they've been writing software in the post-AI era.
I'm guessing you meant to say "pre-AI era."
But don't kid yourself. What that means is right before they prompt "and don't make mistakes," they'll prompt "use less memory."
How so? Memory bandwidth and capacity has been steadily increasing. On desktop for example, 10 years ago an i7-5960X was the absolute top of the line and it had a limit of 64GB, with most motherboards in practice being limited to 32-48GB, and this was an unthinkable amount of RAM at the time for desktop. Now the top end desktop chips support 192-256GB, and a year ago (before the RAM price spike) you could realistically buy that much if you wanted for a reasonable price.
Has it? The i7-5960X was 8C/16T, Ryzen 9 9950X (and variants) are 16C/32T. Even the Threadripper PRO 9000 WX is 96C/192T and supports a staggering 2TB of RAM.
Not sure I buy that? There's a hardware/process argument that DRAM scaling is hitting a wall (for largely the same reason that logic scaling is, even though the fabs are different).
But DRAM has always been almost completely dominated by consumer device needs, and consumer requirements leveled off hard maybe a decade ago. Once you had a 4G system, things were mostly OK. Pervasive VM use and general bloat pushed that up by maybe a factor of two or three since, but that's it.
AI is different. It wants that capacity for different reasons and puts it in different places. And almost none of us really saw that coming more than ~18 months ago or so. Certainly not to the extend that you'd conclude "we've been ignoring it".
Yes it is, they're talking about induced demand. There are lots of resources where, when usage of the resource becomes more efficient, people end up using more rather than less.
If training and inference is hardware constrained, and you can train and server better and bigger models on the same hardware with memory optimizations, that's exactly what I would expect companies to do.
> I have this wild theory that this has not just todo with AI but with weapons production overall.
Why? Weapons are low-volume compared to commercial products, and they probably use trailing edge or weird bespoke components.
I was digging around the other day, and at least the initial versions of the Tomahawk cruise missile had the computing power of an original IBM PC. That just shows how little computing power is needed for a lot of advanced weapons capabilities.
... for what were then 'advanced' weapons capability in high-cost units produced and expended in relatively small numbers with high yield payloads.
The modern drone warfare version of expendable, high volume, high technology small units with two way communication via fibre-optic and RF jamming resistant multiple channels, and the ability to do on-device video object identification and terminal flight control when active electronic counter measures degrade operator control...
That seems like exactly the kind of gear using commodity DDR4/5 sodimm and dimm modules and literally burning them up at then end of nearly every sortie. Even if they are using much smaller units (smartphone grade RAM) it's still cooking the limited output of the global memory fab capacity, and from the fields covered in fibre-optic blankets, I think it's safe to say there is a significant volume of electronics equipment being consumed.
So yeah, AI might be ordering memory and putting it in datacenter, but eventually that material will be available for e-waste recovery (repurpose first, then recycle later). Both are competing for the same limited fab capacity.
or you can wait until one of those frontier labs IPO goes bust in a really bad way where it pumps 10x on day 1 and then everyone dumps every share out there and half a million people are holding bags. that ll surely wipe out all the AI hype
What was the benefit of all this AI stuff again? Because all I see is middle and lower class being fucked over, repeatedly. Increasing prices of consumer electronics, job losses, suffering quality of goods and services, consolidation of power in favor of elites.
Think about the app sitting on some developer's laptop, necer getting deployed anywhere else! Or the thousands getting uploaded to app stores to slurp and sell data!
Oracle's recent force majeure invocation shines a spotlight on what seems to be an endemic problem: datacenter builders have been signing commitments to bring a certain amount of capacity online by a certain date, without first conducting sufficient due diligence about whether they would be able to secure all the resources needed to meet those commitments.
This means they are now caught in a mad scramble to source all sorts of stuff, including RAM, potentially as a matter of life and death.
What's less clear to me, though, is whether datacenter builders will be able to make good on these purchase agreements. It seems these projects have every opportunity to fail, and the path to success keeps getting narrower. What are the odds that chip manufacturers ultimately get caught holding a bag of chips that nobody can buy in one hand, and a bag of IOUs from bankrupt customers in the other?
Yeah there was just a story about how there are hundreds of billions of dollars in "non-cancellable" debt obligations to build AI data centers. But they're building far beyond current demand, I feel like the collapse is inevitable.
sure, collapse is inevitable, alright
how does one prepare for the moment it actually happens? what would it look like?
what were the immediate consequences of dotcom for the normal folk?
I'm sensing sarcasm but is it not true that the scale of obligations is unsustainable?
As for preparation, I'm not sure. I have a 401k that I'm worried about. I'm too young to really remember the dotcom bubble, let alone have any stake in the stock market at the time.
Save money now.
When the bubble bursts it will depress valuations across the board, including the valuations of the good companies that will survive to eventually own significant market share. Buy those at those depressed prices and wait a decade.
The trick will be learning to tell the difference between the Google's and the pet.com's of the AI era. The trick during the dotcom situation was to look for companies that had actual gross profit and were reinvesting it, rather than companies that only had theoretical profit based on nonsense like market share and eyeballs.
Take Amazon as an example. They were $107 a share in 1999. By late 2001, it had fallen to under $6. Now they are $250. It still took almost 10 years to recover.
Is there anyone in the AI sector that is making profit and re-investing it? I guess that really only applies to Microsoft, Apple, Google, etc. that other lines of business to prop up their AI divisions, vs. companies like OpenAI and Anthrophic. I wonder where NVIDIA will land...
Google is making money on it. However, that's coming at the expense of the very free web they built it from. So they have a different problem that may be unsolvable.
Worst part is that some of the companies that survived the dot com era are the new "...but on the internet" companies except now it's "...but with chat bot"
It's half sarcasm, half my frustration in the way ai crash proponents behave in general
if you proclaim that there's an inevitable doom - propose preparations *people you're talking to* can take
when I hear about nuclear doom, I see preppers discussing bunkers and how rich people buy land in Argentina to evacuate to
when I hear about stock market crash, I see people discussing some famous super-investor moving his money into coca-cola, because food would always have a demand
even bloody "the end is nigh" apocalypse junkies continue with "go to church" (or smth similar) in the very next sentence
what can YOU suggest ME as the next action? not for the government to do something, not for the whole market - how do you suggest ME to prepare? what do you suggest *I* will experience?
at best, our approximation of ai bust is the dotcom - and that happened beyond current generation's memory. At best surviving context is "and the ruin left a lot of cheap fiber laid out all around the world", so... we'll get a lot of cheap ai then? good
> at best, our approximation of ai bust is the dotcom - and that happened beyond current generation's memory.
I'm so old now they think I'm dead. Honestly, it might be kind of a relief.
you're not dead - you just stopped being current
just part of the riverbank :)
Nobody Knows. The market can remain irrational longer than you can remain solvent.
Having cash in reserve to deploy in the aftermath of a crash is golden. If that happens you look great, if that doesn't happen for the next 5 years, you miss out on the interim gains.
Keep investing, diversify, have some reserves. Try to be less anxious about things. Exercise, sleep, be grateful for things you have.
Not every inevitability can be mitigated. We're each and every one going to die. The sun will die. The universe will wind down. Embrace it, don't pretend you can avoid it.
Except we don't get cheap AI this go round. It's just a bunch of e-waste that can't even be repurposed into server in the closet back at the office because each server needs about 100kW of power.
There are signs that we are, and are not, in an AI bubble.
Compared to the dotcom boom, firms are well positioned in terms of capital - they have revenue and profits.
However, there's still the absurd speculative valuations, risk, and high expenditure (on said data centres). Also, Andrew Bailey from the Bank of England, personal opinion - https://www.bbc.co.uk/news/articles/cv8e30enrkxyo
Discussed yesterday https://news.ycombinator.com/item?id=49920932
Was at the computer store yesterday and the RAM I bought for $130 last year is now over $800.
Yeah I checked on DDR4 prices too recently and it made me pretty sad. Glad I upgraded to 64 GB in 2023 though. Another homelab dream deferred
The real problem is not that they are buying all the hardware, but that they are buying it and keeping it in bloody warehouses without bringing it online. Because they haven't got DC capacity for it.
Has anyone else here been trying to rent some cloud gpus for reasonable money? I have. I have signed up to a dozen "providers" that advertise everything from tesla cards to b200 only to find stuff is either out of capacity or double/triple the price.
Remember when software was eating the world? The gravy train ran out on commodity hardware and now we are paying the price for almost 3 decades of underinvestment.
I got a 64gb 6000mt/s kit for 250 recently. Not a humble brag.
How?
$250 angle grinder
Yeah, sure.
He didn't say it's a working kit.
More like they're praying it does while waiting for more options to vest
yeah, at some point memory footprint becomes an architectural constraint again, not just a number you throw more hardware at.
No more two-space copying garbage collectors.
But still it is about GiB instead of KiB.
My health insurance company recently rolled out a major revamp of their website, and it now uses GiB as if they were KiB.
At my knitting group last night one of the conversation topics was how everyone's computer feels really unstable these days. One person stopped volunteering at a local charity because, after a recent website rewrite, she has too much trouble with the page locking up when she's trying to sign up for hours.
Last weekend I got talking to another parent at my kid's gymnastics class about how her computer's really struggling since they rolled out some "modernization" rewrites of internal business applications, and it's making it hard to get her work done.
What I'm getting at is, if we do have to get used to memory being more expensive, that might be very disruptive. Programmers will have to do a major about-face on how they've been writing software in the post-AI era.
On the other hand, memory optimization is so much easier with AI. Profiling an app, reworking a function to be twice as long but more efficient, A/B testing libraries across browsers to check which uses the least extra memory. All of these things are fairly easy but so tedious, which makes them perfect for AI.
> What I'm getting at is, if we do have to get used to memory being more expensive, that might become very disruptive because software developers will have to do a major about-face on how they've been writing software in the post-AI era.
I'm guessing you meant to say "pre-AI era."
But don't kid yourself. What that means is right before they prompt "and don't make mistakes," they'll prompt "use less memory."
You may joke but if it can be measured, it can be iterated upon.
It's about 1TiB of HBM going into a GPU server eating about 3TiB of DDR that could have gone to consumer electronics.
This is driven by AI and at a certain point you just need a minimum amount of memory to load all the floating points representing Neural Net weights.
The memory capacity wall has been coming at us at high speed in plain sight for decades. We've been ignoring it.
Doesn’t seem like it was coming at a high speed if it took decades to manifest.
How so? Memory bandwidth and capacity has been steadily increasing. On desktop for example, 10 years ago an i7-5960X was the absolute top of the line and it had a limit of 64GB, with most motherboards in practice being limited to 32-48GB, and this was an unthinkable amount of RAM at the time for desktop. Now the top end desktop chips support 192-256GB, and a year ago (before the RAM price spike) you could realistically buy that much if you wanted for a reasonable price.
The corresponding thread count has increased by much more than 4x.
Has it? The i7-5960X was 8C/16T, Ryzen 9 9950X (and variants) are 16C/32T. Even the Threadripper PRO 9000 WX is 96C/192T and supports a staggering 2TB of RAM.
Not sure I buy that? There's a hardware/process argument that DRAM scaling is hitting a wall (for largely the same reason that logic scaling is, even though the fabs are different).
But DRAM has always been almost completely dominated by consumer device needs, and consumer requirements leveled off hard maybe a decade ago. Once you had a 4G system, things were mostly OK. Pervasive VM use and general bloat pushed that up by maybe a factor of two or three since, but that's it.
AI is different. It wants that capacity for different reasons and puts it in different places. And almost none of us really saw that coming more than ~18 months ago or so. Certainly not to the extend that you'd conclude "we've been ignoring it".
/s/execpt/create/
Executives expecting $$ as usual
[dupe] https://news.ycombinator.com/item?id=49920932
That's assuming CXMT or another Chinese company doesn't pull something out of the bag.
Watch as memory manufacturers learn nothing from the car industry, and cry for tariffs when the Chinese inevitably annihilate them.
Or possibly more likely get the Chinese manufacturers labeled as some kind of national security threat and forbid importing their products.
Can someone please explain to me how the recent llm cache breakthrough doesn't alleviate this memory shortage issue? https://intl.cloud.baidu.com/en/article/8937874
Why use less memory when you can just have more LLM?
This is not a very useful comment
Yes it is, they're talking about induced demand. There are lots of resources where, when usage of the resource becomes more efficient, people end up using more rather than less.
See Jevons paradox. https://en.wikipedia.org/wiki/Jevons_paradox
Not everyone is using baidu?
If training and inference is hardware constrained, and you can train and server better and bigger models on the same hardware with memory optimizations, that's exactly what I would expect companies to do.
I have this wild theory that this has not just todo with AI but with weapons production overall.
> I have this wild theory that this has not just todo with AI but with weapons production overall.
Why? Weapons are low-volume compared to commercial products, and they probably use trailing edge or weird bespoke components.
I was digging around the other day, and at least the initial versions of the Tomahawk cruise missile had the computing power of an original IBM PC. That just shows how little computing power is needed for a lot of advanced weapons capabilities.
> how little compute power used to be needed ...
... for what were then 'advanced' weapons capability in high-cost units produced and expended in relatively small numbers with high yield payloads.
The modern drone warfare version of expendable, high volume, high technology small units with two way communication via fibre-optic and RF jamming resistant multiple channels, and the ability to do on-device video object identification and terminal flight control when active electronic counter measures degrade operator control...
That seems like exactly the kind of gear using commodity DDR4/5 sodimm and dimm modules and literally burning them up at then end of nearly every sortie. Even if they are using much smaller units (smartphone grade RAM) it's still cooking the limited output of the global memory fab capacity, and from the fields covered in fibre-optic blankets, I think it's safe to say there is a significant volume of electronics equipment being consumed.
So yeah, AI might be ordering memory and putting it in datacenter, but eventually that material will be available for e-waste recovery (repurpose first, then recycle later). Both are competing for the same limited fab capacity.
Not just that. Everybody is rearming and the run for the first AI enabled fully autonomous robot is also going on.
https://m.youtube.com/watch?v=TYaj0QnZx9s
Exactly. It might actually be a long term benefit to humanity that memory fab production is constrained at this juncture.
Demand outstrips supply. Efficient software is great, but it doesn’t directly resolve insufficient supply of hardware.
or you can wait until one of those frontier labs IPO goes bust in a really bad way where it pumps 10x on day 1 and then everyone dumps every share out there and half a million people are holding bags. that ll surely wipe out all the AI hype
What was the benefit of all this AI stuff again? Because all I see is middle and lower class being fucked over, repeatedly. Increasing prices of consumer electronics, job losses, suffering quality of goods and services, consolidation of power in favor of elites.
Think about the app sitting on some developer's laptop, necer getting deployed anywhere else! Or the thousands getting uploaded to app stores to slurp and sell data!