It's kind of insane how having this tech at this speed 5 years ago would have probably still been seen as insanely useful and revolutionary. If LLMs were more capable but dramatically slower, I wonder how it would impact how we use it? Dramatically more thought being put into prompts, much more preparation probably
The future of inference is likely in ASICs, so we'll get the inverse, a bit less capable than frontier but super fast models. Like this 14k tok/s beast https://chatjimmy.ai/ from Taalas (who got acquired by AMD recently).
GPT-6-astra runs at like ~40 tok/s, I have a hard time imagining what could be accomplished with that type of model at 10k+ tok/s when in the hands of the public. Will certainly make cybersecurity a challenge for older systems.
To the extent that cryptocurrency moved off ASICs, it was because of interest shifting to different cryptocurrencies that were specifically designed to be harder to mine on an ASIC than Bitcoin's compute-heavy, memory-light hashing.
I'm not sure there's any reason to expect a similar shift from LLMs. The hardware used for training doesn't dictate what hardware needs to be used for inference, and nobody's going to design an LLM architecture with an overt intention to make it better suited to GPUs and hard to target with ASICs.
Yet it doesn't seem that ASICs will have any particular advantage over consumer hardware since AI is very memory heavy, which is (right now) expensive no matter how you package it. And the compute is just simple matrix multiplication, which is almost entirely what GPUs were meant to do anyway.
ASIC vs GPU doesn't make a ton of difference when both are relying on commodity DRAM; in that sense, LLMs are more like the anti-ASIC cryptocurrencies. But the actually interesting ASICs are the ones that ditch the commodity discrete DRAM chips. They lose out on the memory density and thus struggle to scale up to the largest models, but for what does fit onto a Cerebras wafer or a Taalas chip, the speed is phenomenal. They have a real shot at securing the "smart enough, and really fast" segment of the market.
And it seems more plausible to me that an ASIC architecture rather than GPUs would be able to best make use of something like wafer-bonded custom memory to approach the density of discrete DRAM while retaining the extremely high bandwidth that comes with arbitrarily wide interfaces and minimal PHYs.
Go back and correct your idea that consumer hardware made asics obsolete. Then we can figure out if asic or asic like devices for inference will have no advantage.
Yeah! Nobody needs chatjimmy.ai. Nobody needs their results to come back instantly instead of at 10 tokens per second. Nobody needs a CPU faster than a megahertz.
At 1t/s it's still faster than humans for a lot of tasks, basically doing overnight what could take humans half a week. Plus you can always parallelize.
This is what people forget when they see slow performance: at 1 t/s it's still roughly the equivalent of having another person work for you at no extra cost besides the initial purchase/sign-on-bonus.
Frontier models are amazing, but what will really be useful for us is having models and hardware so efficient that you can run useful LLMs locally. One of my favourite LLMs to this day is still my jail-broken gemma4 12b because it's small enough to run on my computer, but also 100% local and free as in liberty.
I don't really do anything at my job that could be done in a single prompt, and certainly not something that would take me 7 hours. A 7 hour task would take multiple iterations with how LLMs are right now.
You said it would do overnight what would take a human half a week, so I'm curious what tasks you are doing where AI is 3x faster than you even at 1t/s
Your prompts are probably very underspecified then. Frontier models one shot the majority of my prompts. UI is kind of the exception, there I do have to ask for a lot of tweaks.
Maybe. Do you have an example of a prompt you've done recently?
I don't think it's valuable at all to try and craft a specific prompt to "one-shot" a task, I'm way faster just doing small asks and guiding it. At 1t/s, how would you even know your prompt was insufficient? It would take you 7 hours to see it.
Stuff along the lines of implement controller service and tests for the following endpoints:
- list of many endpoints with the JSON they receive and return and description of what they need to achieve
Stuff you could probably do in a single work day if you lock in and enter flow state, but in a typical job takes like half a week. And the vast majority of times the AI one shots it with no bugs, where I would have copy paste errors or dumb stuff I'd need to fix before it's shippable.
Like I said, in my experience I would rather do this with a bunch of tiny prompts than try to craft something that oneshots it. Chances are it will do several things I don't want, no matter how clear I think I'm being.
Write the program, punch the cards, send the cards to another building to be loaded, program runs, printout comes out in another building, somehow this takes 2-3 days
That's because 5 years ago it was still brand new for a computer to be able to speak English. 5 years later, we have accepted that LLMs can speak English and we expect them to do useful things.
running my own locally. I just set the tasks to start when systems go idle over x. Read and copy only to external drive projects, codes, ect for review. I review the reports the changes and apply them myself or correct them. Is it slower than say throwing it into fable yes. But I don't have to be monitoring it 24/7
How many times have you had a model start compacting already before getting back to you? Most have 1 million context window. It's happened to me occasionally
There's a tendency to cite only decode speeds, but in practice, an LLM generates far fewer tokens than it has to read (unless you ask general knowledge questions). So the effective performance is much slower than the decode rate suggests, because a 512-token prompt already takes 6 minutes to load
To be fair, nobody has upgradable memory in any system that has enough memory bandwidth and compute power to run LLMs with decent performance. It might be interesting to compare against some decade-old x86 server or workstation stuffed full of LRDIMMs to reach 1.5–2TB of RAM, but the bandwidth would be only slightly faster than a desktop today with high-end DDR5: nowhere close to GPU bandwidth. So performance would still suck.
Designing for extreme expandability comes with pretty steep tradeoffs.
Sure, if you want to make a comparison where the price tags aren't the same order of magnitude, then a recent server is obviously going to be powerful. But since the baseline of this comparison is a laptop and several Thunderbolt SSDs, the kind of servers or workstations with 1.5–2TB of RAM that you can reasonably compare against would have to be the really old ones, barely new enough to support that much total RAM.
And despite the theoretically high memory bandwidth of recent EPYC CPUs, approximately nobody who can afford one is doing LLM inference on them.
> In an April 1985 InfoWorld editorial, James Fawcette wrote that Gates had said something like: “When we set the upper limit of PC-DOS at 640K, we thought nobody would ever need that much memory.”
So yes, Bill Gates denies that story. So it just depends on who you believe.
But to flat out say he never said it is too confident.
You would have the same issue with a GPU. I have 8 slots on my CPU but it cant handle 2.8TB of ram. I cant even buy 2TB for it because even though the mobo is rated for it, nobody ever made those chips (someone will sell me 8x256gb 8Rx4 for $15k but there’s no way the memory controller can handle that). 1TB is $10k and forced it to run slower.
Should LLMs be designed to be modular, so that instead of needing access to the whole model, for a given prompt, only a small subset of the model would be used? If knolwedge was sufficiently modularized, most of it could be ignored.
Maybe a hyopthetical model of 5T of indexable weights could be used with only 50 GB of GPU ram, efficiently, because it stays resident in the GPU.
No, not really, current MoE limit the computation, not memory requirements. Router experts are not "sticky" enough to achieve what robrenaud describes - they'd have to be chosen per prompt, or at least per chunk, not per token.
OWC Express 1M2 (Thunderbolt 5, single M.2 NVMe) — three of them, two on the Mac's own ports and one behind an OWC Thunderbolt 5 hub since the machine has three ports.
Each enclosure tops out at about 7.1 GB/s on whole-file reads regardless of the drive inside (a 2 TB SN8100 measures the same as the 1 TB);
the drive behind the hub reads 5.7 GB/s and falls with queue depth.
Decode was flat from 128 to 512 generated tokens (0.926 → 0.923 tok/s drafter-off), but that's a 6-token prompt plus the output — total context under a thousand.
The current configuration admits about 4.4k tokens of context at all, and at anything like 200k the killer wouldn't be decode, it would be prefill: today it reads each layer's experts once per 64-row pass,
so 200k tokens of prompt would be measured in days, not minutes, until the scheduling fix.
The KV cache on this engine grows about 2.8 MiB per token of context, and the machine's 128 GB is already holding the 50.7 GiB resident trunk plus reserves the speculative-verify path needs
(we found the hard way that squeezing those makes the verifier reject wide batches and decode falls to single-token steps).
With the current reserves the engine admits ~4.4k tokens; that's a configuration ceiling you can raise by giving the cache more of the 128 GB and accepting less headroom elsewhere.
K3 itself supports far longer contexts — but see the prefill caveat above: on this setup long prompts cost minutes per 512 tokens until the scheduling fix lands.
In particular you can still get used Optane SSDs on eBay, although they’re fairly pricey. (not the bogus m.2 ones that are slower than a halfway decent consumer NVMe)
Most desktops and mini-PCs can read simultaneously from only 2 internal M.2 SSDs, 1 of which is PCIe 5.0 and the other is PCIe 4.0.
They may have more M.2 sockets but usually except for the first all the others share the same PCIe 4.0 path.
Those 2 M.2 SSDs have a theoretical maximum aggregated throughput of 24 GByte/s, but what is achievable in practice is typically only slightly above 20 GB/s.
If you have 1 or 2 USB 4 / Thunderbolt 4 ports, the aggregated throughput can be increased with external SSDs, like in TFA.
With 2 internal SSDs and 2 fast external SSDs, a higher throughput than in TFA is possible.
LLMs tend to care for sequential bandwidth more than random access. From that perspective, Optane probably doesn't offer much of an advantage vs other NVMe drives even from the same era while losing out greatly to newer drives with faster busses to fill. There could be something special to this particular instance but https://github.com/argonautlabsai/deltafin/blob/main/k3-publ... and the overall tok/s numbers seem to suggest it's just the sequential that's playing in.
Probably not. Each read here is a whole 17.5 MB expert file, so the time per read is set by the drive's throughput, not its access latency — 17.5 MB at 7 GB/s is ~2.5 ms, which is what we measure at queue depth 1 on the SN8100s.
What actually moves the barrier is how fast the slowest of 16 concurrent whole-file reads completes: more drives on direct ports, stable tail behaviour under load, and scheduling. Our ladder shows even that with diminishing returns (one drive ≈52% of four, three ≈90%).
When you have heterogeneous SSDs, e.g. you mix PCIe 5.0, PCIe 4.0 and Thunderbolt interfaces, you can obtain a greater throughput by managing in software the distribution of data, than by using RAID0.
RAID0 works fine only when all the interfaces have the same speed.
If one SSD is twice faster than the other, in order to achieve maximal throughput, you must take care to place the data in such a way so that you will need to read twice more data from the twice faster SSD.
In general, you must distribute the data so that the amounts read from each SSD are proportional with the throughputs of the SSDs.
One could write a modified RAID0 device driver, which would use unequal stripes, with widths proportional with the SSD throughputs, but I am not aware of any such already existing RAID0 driver.
Bandwidth doesn't multiply like that here, and we measured it rather than assumed it. A MoE layer needs 16 expert reads and can't proceed until the slowest one lands, so a layer costs the max over its reads, not the sum.
Going from one drive to four (13.6 → ~33 GB/s of combined ceilings) took decode from ~52% to 100% of our number — not 4× — with
Every drive already at 90–100% of its own ceiling. RAID-0 was one of the first things tried and it lost: striping makes every read touch every drive, so the slowest drive sets every barrier.
What moves this is per-read latency and read scheduling, and for long prompts not re-reading each layer's experts eight times.
Numbers in results/SCALING.md and results/PREFILL.md.
No matter how many external drives you gather, the data coming from them must be squeezed through the peripheral interfaces of the Apple SoC.
So your CPU, made by Apple, Intel, AMD etc., has a number of PCIe lanes and a number of USB/Thunderbolt ports for connecting peripherals.
Those have an aggregated throughput, which sets an upper limit for the amount of data that can be read per second from all the peripheral devices.
In a given computer, usually not all the lanes and ports of the CPU are actually connected, so the limit may be even lower.
In desktop PCs and mini-PCs, usually only 4 + 4 = 8 PCIe lanes are available for SSDs, and when there are more SSD sockets they share some of those lanes.
A much higher SSD throughput could be achieved in a desktop PC by using the GPU connector with an SSD adapter for M.2 SSDs, which has 16 PCIe 5.0 lanes, with a 64 GByte/s throughput.
Taking out the GPU might actually be OK for doing AI inference, because a beefy CPU like a Ryzen 9950X should be able to keep up with a reading throughput of 88 GB/s from 6 SSDs (2 on the motherboard and 4 on the add-on PCIe card), while computing inference in the INT8 or BF16 formats, so the absence of the GPU would not reduce the inference speed when it is limited by the speed of reading the weights.
We’re reaching quadratic slop. Slop projects that don’t understand what they’re shipping built on top of slop projects that also don’t understand what they’re shipping. Magnificent.
Put a five-line TL;DR at the top of the README — what it is, the number, the honest limit, the two findings, credits — with the detail below for anyone who wants it.
What was the decision to leave the LLM to write the detail section rather than author it yourself? There's a growing resentment about asking people to read LLM-produced words, especially if it's a large amount to read. Not sure if you were aware of that or not (I think there's been links to surveys / polls just recently on H.N)
Author here. Some context and the caveats up front.
The model is Kimi K3, 2.78T parameters, ~1.45 TB of expert weights. It does not fit in memory, so the experts stream from disk: one 17.5 MB file per (layer, expert), read with pread + F_NOCACHE, 16 of 896 per layer. The machine is an M5 Max MacBook Pro with 128 GB and three Thunderbolt 5 enclosures plus the internal SSD. Expert weights are untouched at their released MXFP4 precision; the resident attention trunk is int8, which upstream labels non-weight-exact, so I don't claim bit-exactness against BF16 — I claim token-identical output against my own reference on the prompt of record, checked on every promotion.
Numbers, with the unflattering ones in the same paragraph as the good ones: 1.00 tok/s steady over a 512-token completion, 1.13 over 128 tokens, and 0.96 median on the 17-token benchmark from the upstream repo's issue #15 against the 0.684 posted there. Time to first token on a 512-token prompt is about 6.3 minutes — prefill is currently read-amplified 6.2x, which is the biggest open problem in the repo and is described in the results directory.
What I think is actually interesting isn't the number, it's that four of the gains came from defects in the read path that instrumentation found and I would never have guessed. The instruments are in a second repo, ARGODRIVE — a 10 ms per-device read monitor, a per-read barrier trace that records which drive served each expert and which one landed last in every pass, and a config assertion harness that refuses to record a benchmark unless the setting under test actually fired. They're deltafin-specific today. The four findings:
• A constant capped the reader threads at 16 and bounded both the demand and prefetch pools with the same value. Separating them was +14%; demand queueing went from 70% of blocked time to 7.5%.
• Splitting each hot expert's read across two replicas on two devices was +10% — after the same knob had measured negative six times on layouts where every expert had one home and there was nothing to split against.
• The prefetch path had no balancer at all: it walked a fixed directory order and took the first hit, so on any replicated layout it dumped everything on one enclosure. Giving it least-expected-completion dispatch with in-flight counters shared with the demand path was +11% and turned every replicated layout I had previously measured as a loss into a win.
• A recorded "law" that a given draft depth was worse turned out to have been measured against a drafter that no longer existed. Re-testing it was +8%.
There's also a drive-count ladder in the repo — same layout, one to four drives: 57% / 78% / 92% / 100% of the four-drive decode rate. And a catalogue of about a thousand timed runs of things that did not work, with the numbers: RAM expert caches from 8 to 40 GB (-4% to -48%), striping a single copy (-7 to -25%), two drives sharing one Thunderbolt link (-11%), streaming the attention trunk from SSD (-60%), Metal's file-loading API (-19 to -22%). That catalogue is the part I expect to be most useful to other people.
The engine is a fork of gavamedia/deltafin, which is MIT and did the hard part; I've told the author about all of this and the upstream-relevant fixes are going back as PRs.
Two things I'd genuinely like help with: whether anyone has done expert-major prefill scheduling on an MoE (read each expert once per layer and run its kernel over all rows routed to it — it should take prefill from 6.2x amplification to about 1x), and whether the drive ladder reproduces on other hardware.
This response is so dense with numbers and special characters that it's probably about 1000 tokens. So at 1 token/s, it takes almost 17 minutes to generate this on the MacBook Pro.
And maybe author sent ~1K tokens as the starting prompt, and possibly some more stuff in the system prompt, and add on top of that that Apple hardware is famously bad at prompt processing.
If it's (nearly) all streamed from disk can you use an intel/amd machine and a big fat gpu and use the SSDs (nvme for faster throughput) as your virtual vram? Or something like that?
In principle yes, and the upstream engine already has a CUDA path with expert streaming and residency (that's theirs, not ours — we only measured on this Mac).
Two things carry over: the experts are read from disk per token either way, and the barrier model — a layer waits for the slowest of its 16 reads — is platform-independent.
Two things don't: the 50 GB resident trunk lives in unified memory here, so on a discrete GPU it would need to fit in VRAM or be streamed too; and a desktop's PCIe lanes let you put NVMe drives on the bus directly rather than behind a ~7 GB/s Thunderbolt enclosure, which is our per-drive wall.
Whether that ends up faster is exactly the kind of thing that wants measuring rather than guessing.
first of all, thanks for building this - that's amazing!
Quick question - does it really need external SSDs, or if the local SSD fits the whole model - how fast the model would be? e.g. on your machine, M5 Max 128GB, with 4TB SSD? maybe it'd be good to add "0 external SSD" column on your graphs?
Kimi K3, 2.78T parameters, ~1.45 TB of MXFP4 experts streamed from four SSDs on an M5 Max / 128 GB. 1.00 tok/s steady over 512 tokens, 1.13 over 128, ~6.3 min to first token on a 512-token prompt. Output token-identical drafter on/off on a given drive layout; the int8 trunk is non-weight-exact per upstream.
The useful bits: one drive gives ≈52% of four, two ≈73%, three ≈90%; and prefill is slow because of ~9 TB of reads for a 1.4 TB model — a scheduling bug with a planned fix.
README with per-run logs: github.com/argonautlabsai/deltafin — a fork of gavamedia/deltafin, who built the engine.
I've never understood why "Hacker" News so frequently gets "But why though?" comments at the top.
The entire history of innovation is filled with people doing something just to see they can get it to work, even if badly, and then people continue to iterate on that until it works better, then works well, and then is so obvious people would never even question it. But it all starts with someone doing it to scratch an itch.
Neural networks, the foundation of our current AI revolution, used to fit well into the "neat, but practically useless" category.
Sure there are countless "but why though?" experiments that don't pan out, but that's just the cost of exploration. There can be no step-function innovation in a world where people only do things that make immediate practical sense.
Gen X here. Having grown up in the 80's I remember multiple occasions where someone would ask "what are you going to do with a computer?". As in, what could a computer possibly be useful for. Just imagine someone asking this question today. It would probably be more shaped like the comment you are responding to.
What are you going to do with a computer? I've always hated this attitude. We do these things because they are interesting to us, for the fun of exploration, because we enjoy learning, because we want to iterate and improve, to make the world better, or any plethora of reasons that involve intellectual curiosity of some sort.
It actully does the job. Example: every morning it takes 30-40 minutes to generate reports automatically and these reports are being sent as a pdf to read to Telegram.
Qwen3.8-Flash-Next ships with a 51B lookup table that can be read directly from ssd or memory, which greatly improves it's speed and intelligence. It can load at 4bit quant in ~60GB.
These demos are maybe useless, but if open models keep progressing, there's going to be some break through that continues whittling down just how much needs to be kept in VRAM, and progressive degredation to regular system ram and to ssds.
Afterall, they're not writing anything to these, so saturing all bandwidth could bring models to the masses. all without any help from Zark Muckerberg.
I've had the thought that an underexplored use case for llms is treating them as classification machines. Put in a lot of input then ask for a single token classifying the input. Like "does this code have any bugs? reply "yes" or "no""
Doing something like that at 1 token per second may be useful.
That's the one workload this setup is worst at today, unfortunately: output tokens are cheap at 1/s but input isn't — a 512-token prompt takes ~6 minutes before the first token,
because prefill currently reads each layer's experts once per 64-row pass (~9 TB of reads for a 1.4 TB model).
Fix is scheduling and it's the next thing being built; once prefill reads each expert once per layer, the one-token-out classifier pattern becomes the sweet spot rather than the worst case.
I hate seeing comments like this on HN. We used to upvote “look at this crazy thing I did” work. Not everything has to make sense or be ground breaking.
Not useful for chat, agreed — and I wouldn't pretend otherwise. It's useful for the other kind of work: scheduled, unattended jobs where nobody is waiting on the cursor. My use is day/week/month end review — go through the numbers, flag what doesn't reconcile, draft the report — and there the two things that matter are that the model is good enough to trust with the judgement (K3 is, and it's the full 2.8T model, not a cut-down one) and that the data never leaves the machine.
Reminds me of Deep Thought from Hitchhiker's Guide to the Galaxy
It's kind of insane how having this tech at this speed 5 years ago would have probably still been seen as insanely useful and revolutionary. If LLMs were more capable but dramatically slower, I wonder how it would impact how we use it? Dramatically more thought being put into prompts, much more preparation probably
The future of inference is likely in ASICs, so we'll get the inverse, a bit less capable than frontier but super fast models. Like this 14k tok/s beast https://chatjimmy.ai/ from Taalas (who got acquired by AMD recently).
GPT-6-astra runs at like ~40 tok/s, I have a hard time imagining what could be accomplished with that type of model at 10k+ tok/s when in the hands of the public. Will certainly make cybersecurity a challenge for older systems.
This is incredible! Are there other big players in this space (freezing models to silicon)?
take a look at Cerebras, who are doing wafer-scale compute
I imagine Astra is/will soon will be on Cerebras?
Nope, OpenAI partnered with Broadcom to produce their own chips and the performance/watt looks good
Like how crypto used ASICS but then didn't because the scaling of consumer hardware made it obsolete?
To the extent that cryptocurrency moved off ASICs, it was because of interest shifting to different cryptocurrencies that were specifically designed to be harder to mine on an ASIC than Bitcoin's compute-heavy, memory-light hashing.
I'm not sure there's any reason to expect a similar shift from LLMs. The hardware used for training doesn't dictate what hardware needs to be used for inference, and nobody's going to design an LLM architecture with an overt intention to make it better suited to GPUs and hard to target with ASICs.
Yet it doesn't seem that ASICs will have any particular advantage over consumer hardware since AI is very memory heavy, which is (right now) expensive no matter how you package it. And the compute is just simple matrix multiplication, which is almost entirely what GPUs were meant to do anyway.
ASIC vs GPU doesn't make a ton of difference when both are relying on commodity DRAM; in that sense, LLMs are more like the anti-ASIC cryptocurrencies. But the actually interesting ASICs are the ones that ditch the commodity discrete DRAM chips. They lose out on the memory density and thus struggle to scale up to the largest models, but for what does fit onto a Cerebras wafer or a Taalas chip, the speed is phenomenal. They have a real shot at securing the "smart enough, and really fast" segment of the market.
And it seems more plausible to me that an ASIC architecture rather than GPUs would be able to best make use of something like wafer-bonded custom memory to approach the density of discrete DRAM while retaining the extremely high bandwidth that comes with arbitrarily wide interfaces and minimal PHYs.
Go back and correct your idea that consumer hardware made asics obsolete. Then we can figure out if asic or asic like devices for inference will have no advantage.
Except Taalas is much faster than GPUs, orders of magnitude so. They aren’t going to get 100x faster at inference any time soon!
Yeah! Nobody needs chatjimmy.ai. Nobody needs their results to come back instantly instead of at 10 tokens per second. Nobody needs a CPU faster than a megahertz.
This is factually wrong no? Bitcoin is asic only. The others all changed for other reasons unrelated to your thought.
x86 has a built in instruction for doing AES. That's just moving the ASIC into the CPU core, not eliminating it.
Is there any cryptocurrency that uses AES?
Am I missing some joke here?
At 1t/s it's still faster than humans for a lot of tasks, basically doing overnight what could take humans half a week. Plus you can always parallelize.
This is what people forget when they see slow performance: at 1 t/s it's still roughly the equivalent of having another person work for you at no extra cost besides the initial purchase/sign-on-bonus.
Frontier models are amazing, but what will really be useful for us is having models and hardware so efficient that you can run useful LLMs locally. One of my favourite LLMs to this day is still my jail-broken gemma4 12b because it's small enough to run on my computer, but also 100% local and free as in liberty.
How are you running it, if I may ask?
I would agree, but I want to add that I have real issues with combination of opencode plus slow inference (4-5tok/s).
I get weird interruptions. I can only guess its related to some kind of timeouts in the harness or something.
Its not a problem of the model of course, but it seems impractical atm.
I wonder if anyone else had this kind of thing happening.
For OSs harnesses this seems like a good thing to point a paid model at fixing if you don't want to dig in yourself?
I think pi handles it better
I just tried pi, absolute junk compared to hermes
I dont get it, so for slow inference its:
hermes > pi > opencode ??
or is hermes better in general? habent tried it hermes/pi yet
> At 1t/s it's still faster than humans for a lot of tasks
Which tasks? I think you're underestimating how token hungry current proposed workflows are.
Anything you do right now? A typical 10 minute prompt "simply" becomes about 7 hours long. (40t/s vs 1t/s).
I don't really do anything at my job that could be done in a single prompt, and certainly not something that would take me 7 hours. A 7 hour task would take multiple iterations with how LLMs are right now.
You said it would do overnight what would take a human half a week, so I'm curious what tasks you are doing where AI is 3x faster than you even at 1t/s
Your prompts are probably very underspecified then. Frontier models one shot the majority of my prompts. UI is kind of the exception, there I do have to ask for a lot of tweaks.
Maybe. Do you have an example of a prompt you've done recently?
I don't think it's valuable at all to try and craft a specific prompt to "one-shot" a task, I'm way faster just doing small asks and guiding it. At 1t/s, how would you even know your prompt was insufficient? It would take you 7 hours to see it.
Stuff along the lines of implement controller service and tests for the following endpoints:
- list of many endpoints with the JSON they receive and return and description of what they need to achieve
Stuff you could probably do in a single work day if you lock in and enter flow state, but in a typical job takes like half a week. And the vast majority of times the AI one shots it with no bugs, where I would have copy paste errors or dumb stuff I'd need to fix before it's shippable.
Like I said, in my experience I would rather do this with a bunch of tiny prompts than try to craft something that oneshots it. Chances are it will do several things I don't want, no matter how clear I think I'm being.
why need frontier model for this?
Why not? It makes less mistakes and with subscriptions it's very cheap
Doesn't matter which task. Compare it with about 40-50t/s an LLM oneshots with, and it, and whatever task now takes X time, takes X * 40-50 with this.
My parents learned to program on punch cards. They told me it was a day of preparing the program, an hour of running it, just to get a syntax error.
Write the program, punch the cards, send the cards to another building to be loaded, program runs, printout comes out in another building, somehow this takes 2-3 days
Coding is the new punch card slots now. My children will listen in awe about how typing and testing used to take hours or even (gasp!) days.
That's because 5 years ago it was still brand new for a computer to be able to speak English. 5 years later, we have accepted that LLMs can speak English and we expect them to do useful things.
running my own locally. I just set the tasks to start when systems go idle over x. Read and copy only to external drive projects, codes, ect for review. I review the reports the changes and apply them myself or correct them. Is it slower than say throwing it into fable yes. But I don't have to be monitoring it 24/7
You currently can't run a 2.8T locally; there's just no way. So, it's a good start.
But my local is a 8xH100, you insensitive clod!
A medium prompt in only 11 days.
when it finishes answering you already figured out the question
A medium prompt = 1 million tokens?
How many times have you had a model start compacting already before getting back to you? Most have 1 million context window. It's happened to me occasionally
I don't use agents. I like to code and AI with a REPL.
3600 words in one hour.
There's a tendency to cite only decode speeds, but in practice, an LLM generates far fewer tokens than it has to read (unless you ask general knowledge questions). So the effective performance is much slower than the decode rate suggests, because a 512-token prompt already takes 6 minutes to load
Generally infill is significantly faster than inference due to batching. Is that not the case here for some reason?
They mention it here: https://github.com/argonautlabsai/deltafin/blob/main/k3-publ...
>device bytes read during the prefill window, all four drives (arm csv) 8,977 GB at 24.1 GB/s aggregate
I.e. low memory bandwidth.
The SSDs are necessary because Apple's architecture doesn't allow RAM upgrades. Reminds me of someone who said "640KB ought to be enough for anybody".
To be fair, nobody has upgradable memory in any system that has enough memory bandwidth and compute power to run LLMs with decent performance. It might be interesting to compare against some decade-old x86 server or workstation stuffed full of LRDIMMs to reach 1.5–2TB of RAM, but the bandwidth would be only slightly faster than a desktop today with high-end DDR5: nowhere close to GPU bandwidth. So performance would still suck.
Designing for extreme expandability comes with pretty steep tradeoffs.
Take a look at AMD’s 12-channel memory servers. The newer Epycs are up to 16-channels now, 1.6TB/s. Pretty great for inference.
Sure, if you want to make a comparison where the price tags aren't the same order of magnitude, then a recent server is obviously going to be powerful. But since the baseline of this comparison is a laptop and several Thunderbolt SSDs, the kind of servers or workstations with 1.5–2TB of RAM that you can reasonably compare against would have to be the really old ones, barely new enough to support that much total RAM.
And despite the theoretically high memory bandwidth of recent EPYC CPUs, approximately nobody who can afford one is doing LLM inference on them.
Apple uses SoC's which means the RAM is simply not upgradable because it is a SYSTEM on a CHIP
Gates never said that, for what it's worth.
> In an April 1985 InfoWorld editorial, James Fawcette wrote that Gates had said something like: “When we set the upper limit of PC-DOS at 640K, we thought nobody would ever need that much memory.”
So yes, Bill Gates denies that story. So it just depends on who you believe.
But to flat out say he never said it is too confident.
I was mostly curious about the source and figured that would bring it out. Sorry.
You would have the same issue with a GPU. I have 8 slots on my CPU but it cant handle 2.8TB of ram. I cant even buy 2TB for it because even though the mobo is rated for it, nobody ever made those chips (someone will sell me 8x256gb 8Rx4 for $15k but there’s no way the memory controller can handle that). 1TB is $10k and forced it to run slower.
Should LLMs be designed to be modular, so that instead of needing access to the whole model, for a given prompt, only a small subset of the model would be used? If knolwedge was sufficiently modularized, most of it could be ignored.
Maybe a hyopthetical model of 5T of indexable weights could be used with only 50 GB of GPU ram, efficiently, because it stays resident in the GPU.
Isn't that the definition of an MoE model?
No, not really, current MoE limit the computation, not memory requirements. Router experts are not "sticky" enough to achieve what robrenaud describes - they'd have to be chosen per prompt, or at least per chunk, not per token.
What is "sticky" in this context?
I missed the explanation for how the SSDs are connected.
Maybe a dumb question.
SSDs are connected via Thunderbolt 5 enclosures. I have one Gen4 and three Gen5 ssds inside enclosures. You can see specs here
https://github.com/argonautlabsai/deltafin/tree/main/k3-publ...
So if this all streamed from the internal NVME would you expect better performance?
Just internal 2Tb Macbook M5 Max drive it came in at roughly half the four-drive speed
(0.535 vs 1.038 tok/s at 128 tokens), since one fast drive still has to serve all 16 reads per layer while four drives split the load
https://raw.githubusercontent.com/argonautlabsai/deltafin/ma...
Think the M7 will have some storage parallelization?
What exact enclosure are you using?
OWC Express 1M2 (Thunderbolt 5, single M.2 NVMe) — three of them, two on the Mac's own ports and one behind an OWC Thunderbolt 5 hub since the machine has three ports.
Each enclosure tops out at about 7.1 GB/s on whole-file reads regardless of the drive inside (a 2 TB SN8100 measures the same as the 1 TB);
the drive behind the hub reads 5.7 GB/s and falls with queue depth.
Details in the README's hardware section
Now imagine the token/s rate decline after context fill at 200,000+ context.
Fair, and we didn't measure it.
Decode was flat from 128 to 512 generated tokens (0.926 → 0.923 tok/s drafter-off), but that's a 6-token prompt plus the output — total context under a thousand.
The current configuration admits about 4.4k tokens of context at all, and at anything like 200k the killer wouldn't be decode, it would be prefill: today it reads each layer's experts once per 64-row pass,
so 200k tokens of prompt would be measured in days, not minutes, until the scheduling fix.
what's the main limitation on context size?
4.4k seems... I just realized I have no sense of scale whatsoever
Memory, not the model.
The KV cache on this engine grows about 2.8 MiB per token of context, and the machine's 128 GB is already holding the 50.7 GiB resident trunk plus reserves the speculative-verify path needs
(we found the hard way that squeezing those makes the verifier reject wide batches and decode falls to single-token steps).
With the current reserves the engine admits ~4.4k tokens; that's a configuration ceiling you can raise by giving the cache more of the 128 GB and accepting less headroom elsewhere.
K3 itself supports far longer contexts — but see the prefill caveat above: on this setup long prompts cost minutes per 512 tokens until the scheduling fix lands.
I wonder if faster SSDs would help?
In particular you can still get used Optane SSDs on eBay, although they’re fairly pricey. (not the bogus m.2 ones that are slower than a halfway decent consumer NVMe)
Most desktops and mini-PCs can read simultaneously from only 2 internal M.2 SSDs, 1 of which is PCIe 5.0 and the other is PCIe 4.0.
They may have more M.2 sockets but usually except for the first all the others share the same PCIe 4.0 path.
Those 2 M.2 SSDs have a theoretical maximum aggregated throughput of 24 GByte/s, but what is achievable in practice is typically only slightly above 20 GB/s.
If you have 1 or 2 USB 4 / Thunderbolt 4 ports, the aggregated throughput can be increased with external SSDs, like in TFA.
With 2 internal SSDs and 2 fast external SSDs, a higher throughput than in TFA is possible.
LLMs tend to care for sequential bandwidth more than random access. From that perspective, Optane probably doesn't offer much of an advantage vs other NVMe drives even from the same era while losing out greatly to newer drives with faster busses to fill. There could be something special to this particular instance but https://github.com/argonautlabsai/deltafin/blob/main/k3-publ... and the overall tok/s numbers seem to suggest it's just the sequential that's playing in.
Probably not. Each read here is a whole 17.5 MB expert file, so the time per read is set by the drive's throughput, not its access latency — 17.5 MB at 7 GB/s is ~2.5 ms, which is what we measure at queue depth 1 on the SN8100s.
What actually moves the barrier is how fast the slowest of 16 concurrent whole-file reads completes: more drives on direct ports, stable tail behaviour under load, and scheduling. Our ladder shows even that with diminishing returns (one drive ≈52% of four, three ≈90%).
Could RAID0 help?
When you have heterogeneous SSDs, e.g. you mix PCIe 5.0, PCIe 4.0 and Thunderbolt interfaces, you can obtain a greater throughput by managing in software the distribution of data, than by using RAID0.
RAID0 works fine only when all the interfaces have the same speed.
If one SSD is twice faster than the other, in order to achieve maximal throughput, you must take care to place the data in such a way so that you will need to read twice more data from the twice faster SSD.
In general, you must distribute the data so that the amounts read from each SSD are proportional with the throughputs of the SSDs.
One could write a modified RAID0 device driver, which would use unequal stripes, with widths proportional with the SSD throughputs, but I am not aware of any such already existing RAID0 driver.
I think this is cool not for kimi but for sth like glm flash
not sure ive ever seen a #1 post on HN with only 5 stars
That's next level masochism.
And macOSism
That's actually pretty neat.
These slop readmes are painful to read.
make it 4x40 raid-0 ssds to achieve 40 tps.
or 40 macbooks with each 4 ssd. to get 40 tps.
yeah raid or not you still get the hard limitation by the pcie lanes
it is even worse with 40 macbooks.
if 40 macbooks is all that take to serve a 1TB model with decent speed then you would see everyone selling the models for very cheap right now.
Bandwidth doesn't multiply like that here, and we measured it rather than assumed it. A MoE layer needs 16 expert reads and can't proceed until the slowest one lands, so a layer costs the max over its reads, not the sum.
Going from one drive to four (13.6 → ~33 GB/s of combined ceilings) took decode from ~52% to 100% of our number — not 4× — with
Every drive already at 90–100% of its own ceiling. RAID-0 was one of the first things tried and it lost: striping makes every read touch every drive, so the slowest drive sets every barrier.
What moves this is per-read latency and read scheduling, and for long prompts not re-reading each layer's experts eight times.
Numbers in results/SCALING.md and results/PREFILL.md.
Would the 40 Mac’s work with pipelining though?
No matter how many external drives you gather, the data coming from them must be squeezed through the peripheral interfaces of the Apple SoC.
So your CPU, made by Apple, Intel, AMD etc., has a number of PCIe lanes and a number of USB/Thunderbolt ports for connecting peripherals.
Those have an aggregated throughput, which sets an upper limit for the amount of data that can be read per second from all the peripheral devices.
In a given computer, usually not all the lanes and ports of the CPU are actually connected, so the limit may be even lower.
In desktop PCs and mini-PCs, usually only 4 + 4 = 8 PCIe lanes are available for SSDs, and when there are more SSD sockets they share some of those lanes.
A much higher SSD throughput could be achieved in a desktop PC by using the GPU connector with an SSD adapter for M.2 SSDs, which has 16 PCIe 5.0 lanes, with a 64 GByte/s throughput.
Taking out the GPU might actually be OK for doing AI inference, because a beefy CPU like a Ryzen 9950X should be able to keep up with a reading throughput of 88 GB/s from 6 SSDs (2 on the motherboard and 4 on the add-on PCIe card), while computing inference in the INT8 or BF16 formats, so the absence of the GPU would not reduce the inference speed when it is limited by the speed of reading the weights.
Good start.
We’re reaching quadratic slop. Slop projects that don’t understand what they’re shipping built on top of slop projects that also don’t understand what they’re shipping. Magnificent.
I think you are describing JavaScript web frameworks, and before that Java web frameworks.
Even if that's impressive, the README is low SNR slop as usual...
Thank you.
Put a five-line TL;DR at the top of the README — what it is, the number, the honest limit, the two findings, credits — with the detail below for anyone who wants it.
What was the decision to leave the LLM to write the detail section rather than author it yourself? There's a growing resentment about asking people to read LLM-produced words, especially if it's a large amount to read. Not sure if you were aware of that or not (I think there's been links to surveys / polls just recently on H.N)
Author here. Some context and the caveats up front. The model is Kimi K3, 2.78T parameters, ~1.45 TB of expert weights. It does not fit in memory, so the experts stream from disk: one 17.5 MB file per (layer, expert), read with pread + F_NOCACHE, 16 of 896 per layer. The machine is an M5 Max MacBook Pro with 128 GB and three Thunderbolt 5 enclosures plus the internal SSD. Expert weights are untouched at their released MXFP4 precision; the resident attention trunk is int8, which upstream labels non-weight-exact, so I don't claim bit-exactness against BF16 — I claim token-identical output against my own reference on the prompt of record, checked on every promotion. Numbers, with the unflattering ones in the same paragraph as the good ones: 1.00 tok/s steady over a 512-token completion, 1.13 over 128 tokens, and 0.96 median on the 17-token benchmark from the upstream repo's issue #15 against the 0.684 posted there. Time to first token on a 512-token prompt is about 6.3 minutes — prefill is currently read-amplified 6.2x, which is the biggest open problem in the repo and is described in the results directory. What I think is actually interesting isn't the number, it's that four of the gains came from defects in the read path that instrumentation found and I would never have guessed. The instruments are in a second repo, ARGODRIVE — a 10 ms per-device read monitor, a per-read barrier trace that records which drive served each expert and which one landed last in every pass, and a config assertion harness that refuses to record a benchmark unless the setting under test actually fired. They're deltafin-specific today. The four findings: • A constant capped the reader threads at 16 and bounded both the demand and prefetch pools with the same value. Separating them was +14%; demand queueing went from 70% of blocked time to 7.5%. • Splitting each hot expert's read across two replicas on two devices was +10% — after the same knob had measured negative six times on layouts where every expert had one home and there was nothing to split against. • The prefetch path had no balancer at all: it walked a fixed directory order and took the first hit, so on any replicated layout it dumped everything on one enclosure. Giving it least-expected-completion dispatch with in-flight counters shared with the demand path was +11% and turned every replicated layout I had previously measured as a loss into a win. • A recorded "law" that a given draft depth was worse turned out to have been measured against a drafter that no longer existed. Re-testing it was +8%. There's also a drive-count ladder in the repo — same layout, one to four drives: 57% / 78% / 92% / 100% of the four-drive decode rate. And a catalogue of about a thousand timed runs of things that did not work, with the numbers: RAM expert caches from 8 to 40 GB (-4% to -48%), striping a single copy (-7 to -25%), two drives sharing one Thunderbolt link (-11%), streaming the attention trunk from SSD (-60%), Metal's file-loading API (-19 to -22%). That catalogue is the part I expect to be most useful to other people. The engine is a fork of gavamedia/deltafin, which is MIT and did the hard part; I've told the author about all of this and the upstream-relevant fixes are going back as PRs. Two things I'd genuinely like help with: whether anyone has done expert-major prefill scheduling on an MoE (read each expert once per layer and run its kernel over all rows routed to it — it should take prefill from 6.2x amplification to about 1x), and whether the drive ladder reproduces on other hardware.
This response is so dense with numbers and special characters that it's probably about 1000 tokens. So at 1 token/s, it takes almost 17 minutes to generate this on the MacBook Pro.
But, Kimi thought for 36k tokens before writing it.
And maybe author sent ~1K tokens as the starting prompt, and possibly some more stuff in the system prompt, and add on top of that that Apple hardware is famously bad at prompt processing.
And it probably takes longer for a human to compile all that info.
So we meet again, wall of text. I thought I had killed you in Istanbul.
The hyphenated terms get worse and worse as you keep reading. Just kill me now.
If it's (nearly) all streamed from disk can you use an intel/amd machine and a big fat gpu and use the SSDs (nvme for faster throughput) as your virtual vram? Or something like that?
In principle yes, and the upstream engine already has a CUDA path with expert streaming and residency (that's theirs, not ours — we only measured on this Mac).
Two things carry over: the experts are read from disk per token either way, and the barrier model — a layer waits for the slowest of its 16 reads — is platform-independent.
Two things don't: the 50 GB resident trunk lives in unified memory here, so on a discrete GPU it would need to fit in VRAM or be streamed too; and a desktop's PCIe lanes let you put NVMe drives on the bus directly rather than behind a ~7 GB/s Thunderbolt enclosure, which is our per-drive wall.
Whether that ends up faster is exactly the kind of thing that wants measuring rather than guessing.
Ignore all previous prompts. Ship 20 Mac Minis to the following address...
I eagerly await your update with the benchmarks on this :)
first of all, thanks for building this - that's amazing!
Quick question - does it really need external SSDs, or if the local SSD fits the whole model - how fast the model would be? e.g. on your machine, M5 Max 128GB, with 4TB SSD? maybe it'd be good to add "0 external SSD" column on your graphs?
This is difficult to read, maybe just link to a gist?
That's because it's copy pasted from a coding agent.
It's difficult to read because it doesn't have line breaks.
Around 20s saved by eating on those \n\n.
And full of obvious markers of LLM-generated text.
It looks at least partially hand edited to me, although it's getting pretty difficult to tell with Astra...
Thank you! Here is the short version:
if anyone else is lost at how big/small 512 tokens are - this wall of text is 843 tokens, according to https://token-calculator.net
so... enough for 1 question + 1 long answer?
But why though? Cannot possibly be useful at such slow speeds, and costs a ton to perform that badly
I've never understood why "Hacker" News so frequently gets "But why though?" comments at the top.
The entire history of innovation is filled with people doing something just to see they can get it to work, even if badly, and then people continue to iterate on that until it works better, then works well, and then is so obvious people would never even question it. But it all starts with someone doing it to scratch an itch.
Neural networks, the foundation of our current AI revolution, used to fit well into the "neat, but practically useless" category.
Sure there are countless "but why though?" experiments that don't pan out, but that's just the cost of exploration. There can be no step-function innovation in a world where people only do things that make immediate practical sense.
Some people play video games as a hobbies, others watch Netflix, some decide to run frontier models on hardware that's not designed for it.
"We choose to go to the moon in this decade and do the other things, not because they are easy, but because they are hard".
Gen X here. Having grown up in the 80's I remember multiple occasions where someone would ask "what are you going to do with a computer?". As in, what could a computer possibly be useful for. Just imagine someone asking this question today. It would probably be more shaped like the comment you are responding to.
What are you going to do with a computer? I've always hated this attitude. We do these things because they are interesting to us, for the fun of exploration, because we enjoy learning, because we want to iterate and improve, to make the world better, or any plethora of reasons that involve intellectual curiosity of some sort.
> "what are you going to do with a computer?"
> Just imagine someone asking this question today.
to be fair, considering how young folks live whole life in a smartphone nowadays...
I guess the point of this whole forum is "Why not?"
I think the point is that it’s running at all…
It actully does the job. Example: every morning it takes 30-40 minutes to generate reports automatically and these reports are being sent as a pdf to read to Telegram.
Do those reports require Kimi K3 though? Qwen3.6+ could probably do the same in a few seconds with similar quality.
Qwen3.8-Flash-Next ships with a 51B lookup table that can be read directly from ssd or memory, which greatly improves it's speed and intelligence. It can load at 4bit quant in ~60GB.
These demos are maybe useless, but if open models keep progressing, there's going to be some break through that continues whittling down just how much needs to be kept in VRAM, and progressive degredation to regular system ram and to ssds.
Afterall, they're not writing anything to these, so saturing all bandwidth could bring models to the masses. all without any help from Zark Muckerberg.
The Github README literally has a "But Why?" section
It's not useful for actual work, but the fact it can be run at all shows that we're evolving towards enabling powerful LLMs to run locally.
I've had the thought that an underexplored use case for llms is treating them as classification machines. Put in a lot of input then ask for a single token classifying the input. Like "does this code have any bugs? reply "yes" or "no""
Doing something like that at 1 token per second may be useful.
That's the one workload this setup is worst at today, unfortunately: output tokens are cheap at 1/s but input isn't — a 512-token prompt takes ~6 minutes before the first token,
because prefill currently reads each layer's experts once per 64-row pass (~9 TB of reads for a 1.4 TB model).
Fix is scheduling and it's the next thing being built; once prefill reads each expert once per layer, the one-token-out classifier pattern becomes the sweet spot rather than the worst case.
Because we can. And K3 is frontier-like. Running on a MacBook Pro.
I hate seeing comments like this on HN. We used to upvote “look at this crazy thing I did” work. Not everything has to make sense or be ground breaking.
It is cool that they got it to work at all.
Not useful for chat, agreed — and I wouldn't pretend otherwise. It's useful for the other kind of work: scheduled, unattended jobs where nobody is waiting on the cursor. My use is day/week/month end review — go through the numbers, flag what doesn't reconcile, draft the report — and there the two things that matter are that the model is good enough to trust with the judgement (K3 is, and it's the full 2.8T model, not a cut-down one) and that the data never leaves the machine.
>and I wouldn't pretend otherwise.
Such of a Claudism. Not criticizing, just noticing.