We configured 6 machines each having 2x5090 for local Qwen3.8 in the past months. Prices are very high.
The performance of local AI for coding, given correct setup & harness, is very good and on par for 95% of use cases compared to closed models.
Local usage for end-users is unmetered and token speeds are very fast and keep improving every few weeks due to optimizations being done in the community.
Data remains local which makes EU compliance checkboxes way easier since data never touches anything related to the CLOUD act.
> The performance of local AI for coding, given correct setup & harness, is very good and on par for 95% of use cases compared to closed models
While I don't know your specific use cases, when interpreted generally, this seems trivially disproven by putting eg. websites made by Opus 5.5 and the local model side by side.
I have merely a laptop 5090 (24 GB) and I can run Qwen3.8 Flash IQ3_XXS at 110-140 tok/s on Strata, 128K context. If Artificial Analysis' benchmarks mean anything, it sits somewhere around GPT 5.6 Terra/Sol. I use both that and Codex/Sol 6.1 most days of the week, for different projects. I don't feel like I'm particularly compromised when working locally - I very rarely think "ah man I wish I could just point Codex at this". The biggest problem is speed - it needs more reasoning time and I can only work single-threaded.
I don't think changing the prompt alone is going to do it, but iteratively providing feedback can get you there.
With GPT-5.4 or GPT-5.5, or of course also by writing the code by hand if we want to go all the way, it was well possible to build nice frontends to our liking.
I still don't want to go back when I can do it better and faster with Opus 5.5.
While I don't doubt your efficacy, the cost of those machines today, especially considering the high cost of electricity in Europe, would be uneconomical versus buying subscriptions. The reason to do this setup is data sovereignty, and there are plenty of cases where this is required.
We are continuously improving the workflow and things improve almost on a weekly basis. Current setup is custom Gitlab project structure + OpenCode + Mattpocock/skills followed by a well defined align/plan/build/review/learn engineering process.
It feels like things changed in the past few months. We are now truly able to do what we need to do without Claude etc, where this was not feasible earlier this year.
I lament not splurging for the 5090 back when I built my machine, I figured it would only increase in value (even used) but it was hard to swallow the price at release which was already expensive and way above MSRP.
Wrote a couple of days ago: I decided to check out what a top gaming PC would cost. $12.5k! Absolutely nuts. I have gamer friends that make bank, the types that have always bought the latest flagship parts at launch, and even they slowing down, because they feel the prices are extortionate.
We're back to early 90s prices.
EDIT: Yes, obviously you don't need a RTX5090 to play games. But it seems to me like the top GPU market is now exclusively for the local LLM users.
it is funny how all this is only driven by ram. From a 5070ti with 16GB of ram at less than 1500€ to a 5090ti with 32GB of ram at 12500€ you get nearly a factor of 10x with the rtx pro at 24GB of ram sitting below 4000€.
As someone whose gaming PC is now over six years old, I am very nervous that my GPU is going to crap out on me, because I won't be able to afford to replace it.
<8GB gpus and below can be found at decent price. While the price are higher than they used to be a year ago for older / lower end levels it is not crazier than the price during the covid/crypto miner period for example which is probably when you built that pc.
The 5090 is a firecracker of a GPU. RAM of course is a part of this; but it's also just... a really good card. I bought it ostensibly for gaming and light AI, but now I use it for Qwen 3.8 27B running in ninfer and in (shameless plug) dreamcoder.ai it absolutely flies. I get ~2-3k per second prefill (important for large contexts), 256k context and ~150 tok/s.
I have 30 years of coding experience. That tiny model is no Opus 5.5, but make no mistake: it's a very strong coder in the hands of a capable developer.
I just did a cost breakdown of owning a GPU vs paying for tokens.
An RTX 5060 Ti 16GB card costs about $800 on Amazon right now. Running 18.4 hours a day ( off peak electric rate schedule ) it can process approximately 109.3 million tokens per month. The card has a 90 day return window. Credit card interest + electricity puts the cost at $80 total out of pocket, to use this card for 3 months and return it.
Paying an AI provider came out to $1,639.50 for the same 3 months of 18.4 hour a day usage.
You know its a bubble when ... $80 vs $1,639.50
I understand that if you're a major corporation, or a researcher that needs 100+ gpus for a large project, unless you have access to your own supercomputer, the economics will be different.
This was for "individual coder at home" scenarios.
tl;dr ... im exploiting jeff bezos, and math is racist. too funny.
I guess people have never heard of "options" before.
If you’re returning a product you used because you wanted to pretend it was a rental at the seller’s expense, bluntly, you’re as much the problem as the seller.
Everyone in the heirachy above you is an exploitative business psychopath. Might as well exploit their calculated greed as they attempt to exploit your empathy.
In fact I'm running qwen3.8-flash-next with a Playwright-mcp docker and it is far better for scraping than my ChatGPT or Claude subscription, for the simple reason that sites aren't lying to Playwright.
I don't understand, your subscription models can also use Playwright, you don't even need to setup an mcp, they'll automatically download playwright-cli and get going.
Yes, but even the smaller models, when used within a harness and orchestrator, can be accurate enough for real work, especially when combined or validated by other smaller models.
The 5060 wasn't the fastest/$, it just provided the most VRAM / $ as of yesterday.
Im installing it in a box with a 4070 Ti to run 2 different models simultaneously. Coder + MoE via paperclip or multica.
That probably is the cheapest per 1 GB vram right now. However, the R9700 a few months ago was only $1350/32GB or $42/1GB VRAM. And everyone knew it, because the card is now $1800 or so with supply issues.
I think the market discovered the reasonable min is 32GB. That runs Qwen3.8 Q6 with 100k+ context all in vram. 24GB does not.
You're counting on most people being willing to abuse a return policy like that. Also, the 3 month return window must've activated 3 days ago. These prices didn't start just then.
Yeah I did something similar but more extensive, because not every model is same plus depend on your options how much you want to degrade model from fp8. Some models is just slow on local hardware and not practical really.
- rate at which tokens are generated
- quality of generated tokens
- ethics of abusing return policies
- long-term viability of abusing return policies
You're comparing running a heavily quantized 10-30B model to running a multi trillion one. Being half the price (unless you expect to only need AI 3 months in your life) is absolutely not a good deal, here.
What it does buy, though, is privacy and freedom (nobody can decide to "change the terms" to milk you more).
I remember when the top GPU was 550€ and the overclocked, custom-cased, water-cooling ready version cost 660€ tops. Any GPU price out there even $800 for 5060Ti from january 2025 is not only out of any sane band, but look as if we're in a ultra-recession, yet other markets aren't affected nearly as extreme.
I bought two 2TB SSDs from TeamGroup last year, about $100/ssd. I put them side by side in a raid setup for my home server. (I wanted raid, and they were less than half the price of Samsung, etc.). I'm on a budget, so I went with the cheap ones and raid reduduncy vs one expensive one with no reduduncy.
The exact same ones are now pushing $300/ssd and Samsung's (which was $300 18 months ago) is now $500.
Yeah, and Qwen 3B that you run on 16GB of VRAM and 10K context length surely is comparable to whatever subscription you're comparing to, that also gives access to 3B parameter models? You might want to take actual use cases into account before you try this scam in real life.
I too did a cost breakdown of buying my GPU for ~10K EUR more than a year ago. It came out "Invaluable" as I use it for analyzing my diary, financials and other private things I cannot send to random 3rd parties.
But even the best of models you can run with 96GB VRAM really doesn't come close to the hosted SOTA models, sadly.
Thankfully, the environment we are all forced to live in, doesn't care what we think about it, and adapting to it accurately at minimal expense is the difference between surviving and thriving.
What people don't like is the environment, and they just decided to stone my post to death for adapting to it, instead of doing the same.
People are just being people.
I'm a much easier target than that Sam guy, who locked up 40% of the planets GPU's using purchase orders, with no obligation to follow through.
He's an untouchable elite. I'm much easier to mug.
We are all opportunists.
The entire thinking behind the post, was spawned by the article the other day, about the "video game construction kit" guy, bragging about spending $3.1 million a month on tokens in an article, with $0 revenue.
> What scam are you speaking of ? The return window ?
Your calculation seems to have depended on "Buy this hardware with the explicit intention of returning it" process, which is a scam. If you buy hardware you do so with the intent of keeping it, otherwise you find renters, if you don't like scams.
So if you don't plan to return it, wouldn't the cost at minimum be the cost of the GPU??? Why count on returning it if you don't plan to do so?
Firstly you are assuming $0.048$/hr for GPU, which is highly misleading and you can easily sell it for double(vast.ai is $0.12) as you are forgetting lot of cost like CPU, cooling etc.
Also even if you can somehow fit a few 100 billion parameter model(like gpt 6 luna or deepseek flash) in 16GB, and magically assume you can run at 200 tokens/sec according to your calculation, the blended cost of it would be around $100, not $1639.
You're comparing apples to oranges. 109 million tokens is ~nothing, easily a day's worth of casual use. Compare price per token and cloud providers are probably much closer. But even that isn't a fair comparison between cloud AI providers and abusing retailer return policies just yet, because usable local models even remotely comparable to the frontier don't fit in 16gb of vram.
We configured 6 machines each having 2x5090 for local Qwen3.8 in the past months. Prices are very high.
The performance of local AI for coding, given correct setup & harness, is very good and on par for 95% of use cases compared to closed models.
Local usage for end-users is unmetered and token speeds are very fast and keep improving every few weeks due to optimizations being done in the community.
Data remains local which makes EU compliance checkboxes way easier since data never touches anything related to the CLOUD act.
> The performance of local AI for coding, given correct setup & harness, is very good and on par for 95% of use cases compared to closed models
While I don't know your specific use cases, when interpreted generally, this seems trivially disproven by putting eg. websites made by Opus 5.5 and the local model side by side.
I have merely a laptop 5090 (24 GB) and I can run Qwen3.8 Flash IQ3_XXS at 110-140 tok/s on Strata, 128K context. If Artificial Analysis' benchmarks mean anything, it sits somewhere around GPT 5.6 Terra/Sol. I use both that and Codex/Sol 6.1 most days of the week, for different projects. I don't feel like I'm particularly compromised when working locally - I very rarely think "ah man I wish I could just point Codex at this". The biggest problem is speed - it needs more reasoning time and I can only work single-threaded.
Imagine thinking that the look of a website is the right metric to judge the performance of AI models.
If your job is to produce websites that look good it's an excellent way to judge performance of AI models _for your workflow_.
What's the problem?
The problem is you can change the prompt a little and get correct results on a local AI too.
Frontend has been solved for years.
I don't think changing the prompt alone is going to do it, but iteratively providing feedback can get you there.
With GPT-5.4 or GPT-5.5, or of course also by writing the code by hand if we want to go all the way, it was well possible to build nice frontends to our liking.
I still don't want to go back when I can do it better and faster with Opus 5.5.
It is one metric to judge them. And if you do frontend stuff, it's probably a fairly relevant one.
While I don't doubt your efficacy, the cost of those machines today, especially considering the high cost of electricity in Europe, would be uneconomical versus buying subscriptions. The reason to do this setup is data sovereignty, and there are plenty of cases where this is required.
>> given correct setup & harness
I am geninuely interested what stack you are using here. Could you please elaborate?
We are continuously improving the workflow and things improve almost on a weekly basis. Current setup is custom Gitlab project structure + OpenCode + Mattpocock/skills followed by a well defined align/plan/build/review/learn engineering process.
It feels like things changed in the past few months. We are now truly able to do what we need to do without Claude etc, where this was not feasible earlier this year.
I lament not splurging for the 5090 back when I built my machine, I figured it would only increase in value (even used) but it was hard to swallow the price at release which was already expensive and way above MSRP.
Wrote a couple of days ago: I decided to check out what a top gaming PC would cost. $12.5k! Absolutely nuts. I have gamer friends that make bank, the types that have always bought the latest flagship parts at launch, and even they slowing down, because they feel the prices are extortionate.
We're back to early 90s prices.
EDIT: Yes, obviously you don't need a RTX5090 to play games. But it seems to me like the top GPU market is now exclusively for the local LLM users.
it is funny how all this is only driven by ram. From a 5070ti with 16GB of ram at less than 1500€ to a 5090ti with 32GB of ram at 12500€ you get nearly a factor of 10x with the rtx pro at 24GB of ram sitting below 4000€.
As someone whose gaming PC is now over six years old, I am very nervous that my GPU is going to crap out on me, because I won't be able to afford to replace it.
<8GB gpus and below can be found at decent price. While the price are higher than they used to be a year ago for older / lower end levels it is not crazier than the price during the covid/crypto miner period for example which is probably when you built that pc.
Same boat here. I'm stuck with my 1070ti.
The 5090 is a firecracker of a GPU. RAM of course is a part of this; but it's also just... a really good card. I bought it ostensibly for gaming and light AI, but now I use it for Qwen 3.8 27B running in ninfer and in (shameless plug) dreamcoder.ai it absolutely flies. I get ~2-3k per second prefill (important for large contexts), 256k context and ~150 tok/s.
I have 30 years of coding experience. That tiny model is no Opus 5.5, but make no mistake: it's a very strong coder in the hands of a capable developer.
Moore Threads Lushan GPU will be out by the end of this year with 64GB RAM.
I just did a cost breakdown of owning a GPU vs paying for tokens.
An RTX 5060 Ti 16GB card costs about $800 on Amazon right now. Running 18.4 hours a day ( off peak electric rate schedule ) it can process approximately 109.3 million tokens per month. The card has a 90 day return window. Credit card interest + electricity puts the cost at $80 total out of pocket, to use this card for 3 months and return it.
Paying an AI provider came out to $1,639.50 for the same 3 months of 18.4 hour a day usage.
You know its a bubble when ... $80 vs $1,639.50
I understand that if you're a major corporation, or a researcher that needs 100+ gpus for a large project, unless you have access to your own supercomputer, the economics will be different.
This was for "individual coder at home" scenarios.
tl;dr ... im exploiting jeff bezos, and math is racist. too funny.
I guess people have never heard of "options" before.
I optioned a video card for 90 days. My bad.
That comparison is nonsensical, it exploits a vulnerability in the retail system in order to not pay for the hardware at all.
On the other hand, if you do include the cost of the hardware it doesn't seem change the overall conclusion.
I'm not exploiting vulnerabilities, im calculating "risk" on the transaction.
You're trading goodwill for greed.
They told me I could return it.
They gave me a 90 day evaluation window.
They did this because it benefits them.
I paid 35% above retail for the card.
And I'm the greedy one ? really ?
Someone needs redpilled on how the monkey planet really works, which was the entire point of this post.
We can choose to spend months crying about card prices and waiting for christmas, or ...
We can choose to analyse the market, calculate risk for our projects, and move forward confidently after doing so.
Or, we can stand on the sidelines and throw rocks at the marathon contestants because they pooped on our lawn.
Choose wisely.
You're a dishonest person who seized a mechanism that improves our consumer experience for your personal gain.
Yes
Edit: I do like that you came back to edit a whole bunch more into your reasoning. So I'll do the same.
I like that you describe yourself as someone who shits on other people's lawns.
I fully believe that you randomly shit on other people's lawns but this marathon you've imagined yourself in is pure delusion.
Thank you for the kind words.
I wish you much success in life.
[flagged]
If you’re returning a product you used because you wanted to pretend it was a rental at the seller’s expense, bluntly, you’re as much the problem as the seller.
Thank you for contributing to current year.
Calculating the financial risk of a transaction before committing to it is a problem to you ?
MBA school calls it responsible corporate stewardship.
No, as I said, it’s committing said transaction and then reversing it at the expense of the other party that I take issue with.
As for MBA school, don’t really give a shit how it is colorfully worded to attempt to excuse it.
Everyone in the heirachy above you is an exploitative business psychopath. Might as well exploit their calculated greed as they attempt to exploit your empathy.
Yes, that does continue the cycle.
Seems like a great way to help the species get a bit lower still.
That card will only run a tiny model though, no?
Exactly. A lot of stupid tokens has applications, but it's not a plug-in replacement for a subscription for most.
... yet.
In fact I'm running qwen3.8-flash-next with a Playwright-mcp docker and it is far better for scraping than my ChatGPT or Claude subscription, for the simple reason that sites aren't lying to Playwright.
I don't understand, your subscription models can also use Playwright, you don't even need to setup an mcp, they'll automatically download playwright-cli and get going.
Sites don't lie to Claude operated Playwright either.
Not to invalidate your point, just saying it's a bit of a straw man.
Sure, but that's not what the subscription uses.
You can just tell it to use playwright instead of the browser skill. This has nothing do with the subscription but with how tool discovery works.
Yes, but even the smaller models, when used within a harness and orchestrator, can be accurate enough for real work, especially when combined or validated by other smaller models.
The 5060 wasn't the fastest/$, it just provided the most VRAM / $ as of yesterday.
Im installing it in a box with a 4070 Ti to run 2 different models simultaneously. Coder + MoE via paperclip or multica.
I see it at $770/GPU, or $48/1GB vram
That probably is the cheapest per 1 GB vram right now. However, the R9700 a few months ago was only $1350/32GB or $42/1GB VRAM. And everyone knew it, because the card is now $1800 or so with supply issues.
I think the market discovered the reasonable min is 32GB. That runs Qwen3.8 Q6 with 100k+ context all in vram. 24GB does not.
Depending on RAM - it can run a big one veeeeeeeery slowly
> to use this card for 3 months and return it.
You're counting on most people being willing to abuse a return policy like that. Also, the 3 month return window must've activated 3 days ago. These prices didn't start just then.
Yeah I did something similar but more extensive, because not every model is same plus depend on your options how much you want to degrade model from fp8. Some models is just slow on local hardware and not practical really.
localaicost.com
Missing from this analysis:
> it can process approximately 109.3 million tokens per month.
quality determined by model(s) and model size(s), not card size.
no return policy has been abused, you imagined this.
the plan was for 1 card today.
RIF.
To be fair we don't know which direction that arbitrage will collapse to.
> to use this card for 3 months and return it.
You can also keep it and return a bag of flour weighing the same.
Or even go to people houses and rob them at gun point to steal their hardware.
You're comparing running a heavily quantized 10-30B model to running a multi trillion one. Being half the price (unless you expect to only need AI 3 months in your life) is absolutely not a good deal, here.
What it does buy, though, is privacy and freedom (nobody can decide to "change the terms" to milk you more).
I remember when the top GPU was 550€ and the overclocked, custom-cased, water-cooling ready version cost 660€ tops. Any GPU price out there even $800 for 5060Ti from january 2025 is not only out of any sane band, but look as if we're in a ultra-recession, yet other markets aren't affected nearly as extreme.
I bought two 2TB SSDs from TeamGroup last year, about $100/ssd. I put them side by side in a raid setup for my home server. (I wanted raid, and they were less than half the price of Samsung, etc.). I'm on a budget, so I went with the cheap ones and raid reduduncy vs one expensive one with no reduduncy.
The exact same ones are now pushing $300/ssd and Samsung's (which was $300 18 months ago) is now $500.
The prices are insane.
You are exploiting a loophole. Calculate the differential if you use one of those "don't buy electronics, just rent them!" services.
This is such a boneheaded comparison.
It’s akin to saying “the new iPhone costs $2.5k, it weighs 250 g, I can get a piece of scrap aluminium for $30, checkmate liberals.”
And imagine if you just stole electricity and GPU. It would be $0 vs 1639.50... The bubble becomes even more obvious!
Yeah, and Qwen 3B that you run on 16GB of VRAM and 10K context length surely is comparable to whatever subscription you're comparing to, that also gives access to 3B parameter models? You might want to take actual use cases into account before you try this scam in real life.
I too did a cost breakdown of buying my GPU for ~10K EUR more than a year ago. It came out "Invaluable" as I use it for analyzing my diary, financials and other private things I cannot send to random 3rd parties.
But even the best of models you can run with 96GB VRAM really doesn't come close to the hosted SOTA models, sadly.
> before you try this scam in real life.
What scam are you speaking of ? The return window ?
I don't plan on returning it, unless the economics of upgrading in 89 days favor doing so.
The point was to calculate "my risk" on the transaction.
As for the results of the calculation, I didn't invent math or risk.
For $80 total "risk", I can get 10+ programmers worth of work for 3 months. A much better gamble than paying per token.
I am seeing nobody liking what you said, but you are not wrong.
Thankfully, the environment we are all forced to live in, doesn't care what we think about it, and adapting to it accurately at minimal expense is the difference between surviving and thriving.
What people don't like is the environment, and they just decided to stone my post to death for adapting to it, instead of doing the same.
People are just being people.
I'm a much easier target than that Sam guy, who locked up 40% of the planets GPU's using purchase orders, with no obligation to follow through.
He's an untouchable elite. I'm much easier to mug.
We are all opportunists.
The entire thinking behind the post, was spawned by the article the other day, about the "video game construction kit" guy, bragging about spending $3.1 million a month on tokens in an article, with $0 revenue.
> What scam are you speaking of ? The return window ?
Your calculation seems to have depended on "Buy this hardware with the explicit intention of returning it" process, which is a scam. If you buy hardware you do so with the intent of keeping it, otherwise you find renters, if you don't like scams.
So if you don't plan to return it, wouldn't the cost at minimum be the cost of the GPU??? Why count on returning it if you don't plan to do so?
Firstly you are assuming $0.048$/hr for GPU, which is highly misleading and you can easily sell it for double(vast.ai is $0.12) as you are forgetting lot of cost like CPU, cooling etc.
Also even if you can somehow fit a few 100 billion parameter model(like gpt 6 luna or deepseek flash) in 16GB, and magically assume you can run at 200 tokens/sec according to your calculation, the blended cost of it would be around $100, not $1639.
You're comparing apples to oranges. 109 million tokens is ~nothing, easily a day's worth of casual use. Compare price per token and cloud providers are probably much closer. But even that isn't a fair comparison between cloud AI providers and abusing retailer return policies just yet, because usable local models even remotely comparable to the frontier don't fit in 16gb of vram.
Not sure that Kantian ethics supports the population mass buying Amazon cards for 90 day periods and then returning them en-masse.
This seems more like a sociopathic "hack" than an actual comment on the economy of selfhost vs inference-as-a-service.