The old DeepSeek V4.1 Flash has been my workhorse model for the past month (after running tests comparing w/ Qwen 3.8 Flash Next, GLM 5.3 Flash, MiMo 2.6 Flash (including the MOPD fix)) and I've run tens of billions of tokens through it and pushed it fairly hard in both private internal benchmarking and in my own workloads to figure out where it's strong/weak.
Partly because I run it on very fast local infra (200+ tok/s) and partly because I'm guaranteed it's not being changed on me, I've generally preferred it over proprietary subscription models for daily tasks, as it seems about as capable at mid-level tasks (I reserve Astra/Opus mostly for planning/review).
I've replaced the previous model with 1017, and while it's early, a few observations:
- It's CoT has changed dramatically. I find the heavy Astra distillation claims to be plausible
- It has strong benchmarks scores, and from what I've seen online it's much better at 3D games and such, but that's not so relevant for my work. On my internal evals it is more token efficient, but it actually doesn't score any higher than the previous model
- It's only been a couple days, but it does seem a bit more focused and reliable. DS4 0731 and DS4.1 have been a bit sloppy (as in not-careful, although they did tend to fall into Claudisms as well), and 1017 seems to be a bit sharper.
- I'm running it through some more challenging work, but I'm not sure if it's really at GLM-5.3 level, especially for cyber/RE/hardware poking, which GLM has excelled at.
> because I'm guaranteed it's not being changed on me
This is something underrated. With an open weight model I can evaluate the performance for my use case, and know for a fact the model will stay the same over time. No arbitrary “model welfare” rules being introduced later on, no undocumented change to capabilities or performance
Somehow GPT 6.1 Sol (medium) is cheaper per task ($0.21 vs $0.26) and faster per task (2.9 vs 4.9 minutes) vs v4.1 Flash 1017 (max). I hope AA adds other reasoning effort results of Deepseek so it'd be more realistic to compare it with other models, because IMO using models at max effort is very unrealistic for most people, they then take a huge amount of time and tokens for a small performance uplift.
Yeah, this always bugged me. Then again, unlike American models, surely Chinese models are far too expensive for them to test them at every difficulty ;)
I don’t know about 1017 but according to my limited testing of 0910 and previous v4-flash-0731 on some of my use cases, max simply used a lot more tokens (almost twice as many IIRC) for little to no benefit over high. So yeah I really don’t see a reason to use max.
For DS41F (both the original and 1017) I've found low (25/100) reasoning to be the optimal for my tasks at least (max doesn't tend to better, and if you can't do it w/ low imo you should just switch to a smarter model). At low it's about 1/3-1/2 max's token count (and price and speed).
A bit like influencers of the benchmarking world. Milking it while all this trends. Are there financial/tie-up disclosures that are audited for benchmarks?
AA prices are always kind of hand wavy for open models. I mean if you rent a cheap gpu and run inference yourself...or heck, what about a cpu instance with a bunch of ram, you can do that with llama.cpp for under a $1/hour. Ok not that, so are we talking about the median spot price per token from a basket of inference providers? Which ones? They actually vary quite a bit, even on openrouter, for this particular deepseek checkpoint it's between $0.15 and $1.20 ... that's like 8x spread...
Do the math yourself, it's actually $0.08 not the AA claimed $0.10 (and that's the more expensive of the two pricing tiers). That's a pretty large error bar.
Also many of the providers are now doing "peak" and "offpeak" pricing (deepseek.com itself for instance) so it can differ dramatically. These "peak" times depend on where the service is most popular. You can basically ride "off-peak" pricing by cascading from global provider to global provider.
The majority of the time taken in an inference call is doing the inference so the extra latency of traveling across an ocean is actually a fairly small wall-clock tax here. People still seem to be partial to nearby servers for ai-related workloads but really, try some decent ones 10,000 km away, you probably won't notice.
I know AA thinks highly of the price metric (I've been to their events, even have their hat, they brag about it) but I have learned to ignore it as noise.
It's an especially strange metric with open-weights decision models. I mean I can run those on my laptop ... cost is very abstract there.
But for closed-source models with only a handful of providers (sometimes just 1) great, the price is a very reasonable number.
For the open models a more useful thing would be like "100 tok/s spot price build" and then it's using pcpartpicker or something in some kind of partnership and can query a build for buying a machine right now that can achieve a certain benchmark number and maybe something for less interactive workloads like "5 tok/s". Of course getting some base unit of power cost in there might be important as well since the power bill is the majority of what you pay for when you rent a commodity gpu.
this matches my experience tbh. 6.1 sol on medium is quite good at getting things right the first time without overthinking and i prefer it over xhigh or max usually. so i'm not surprised it ends up cheaper.
What I really want to know for models like this is if I can have them run on really lame consumer hardware (consider 128gb of ram, 2080ti) with extremely low TPS, but with reasonable enough intelligence that you can walk away and expect something to bake really nicely while you sleep.
Perhaps I'm losing touch with the edge, but it doesn't seem like this is quite possible with the hardware baseline I'm talking about...
"Model like this" not sute about ds v4.1 but qwen3.8 flash next is currently running on 10gb 3080 + 64 gb ddr5 ram at respectable 40-70t/s with 1200pp at max context
I use DwarfStar DeekSeek V4 Flash on my Mac Studio with 128gb of RAM and it's pretty good - decent TPS and coding ability. Apparently the V4.1 model can also be run by streaming weights from the SSD - I've not tried this yet because it results in a lower TPS, but eventually this version of 4.1 should also be usable like this.
expert streaming with cache-conditional routing is very powerful (you can run 4.1 Flash on two Strix Halos and a 9070XT at around 50-70TPS decode in ~full quality), and that setup might be able to at least do a few tok/s with it.
Nvidia stopped manufacturing the 5090, so you and people like us cannot run these models on our computers and are forced to use American SOTA so they don't lose the game...
Mimo from Xiaomi has safety refusals on things compltely unrelated to safety. It's good when it works but you need another model to continue the work when Mimo craps out.
It does. Even looking at cases where a both a better and a worse model accomplish a given goal the difference between a solution and a good solution is important. And will save you time by not having to iterate as much with it.
For this model in particular, even if it didn’t get stronger and it simply thought less that is a big win on its own. DS4 can be served at a hit tok/s, but it takes a while because it thinks for so long.
Really wild that Deepseek's model is 536+16B (passive+active) parameter model and its really comparable to Mimo v2.6 Pro which is the current open source SOTA model which is at (978+42)
This is half the size of Mimo and comes pretty damn close to it and I feel like lots of tasks can be done by it.
I really like its speed as well but its quite verbose from the benchmarks, IMO, this should be tackled in the near future as well. I wonder if specific fine-tuning can happen just to make the model token-efficient.
Also, slightly offtopic though deekseep has good tps but Nvidia's nemotron recent 120B model also has some really crazy tps. I wonder if we are gonna see some really good tps on general hardware.
The best part is that the amount of current innovation happening within the space means that a lot of good ideas are bound to be absorbed by one or the other and we will get many of these improvements handed to us :-D
I think that such engineering is also the engineering of the trade-offs and on what axis to play on. You can be SOTA but with very low speed, extreme price and other factors. I think its interesting to see how each engineering lab approaches these trade-offs and what they pick. But whatever they pick, its best because its more choices for us to pick upon from (especially if its open weights)
At risk of being blunt: Why the surprise that a model that activates 12B parameters is faster than one that activates 16B parameters which is faster than one that activates 42B parameters?
It feels like that would be the expected result, all else equal.
The old DeepSeek V4.1 Flash has been my workhorse model for the past month (after running tests comparing w/ Qwen 3.8 Flash Next, GLM 5.3 Flash, MiMo 2.6 Flash (including the MOPD fix)) and I've run tens of billions of tokens through it and pushed it fairly hard in both private internal benchmarking and in my own workloads to figure out where it's strong/weak.
Partly because I run it on very fast local infra (200+ tok/s) and partly because I'm guaranteed it's not being changed on me, I've generally preferred it over proprietary subscription models for daily tasks, as it seems about as capable at mid-level tasks (I reserve Astra/Opus mostly for planning/review).
I've replaced the previous model with 1017, and while it's early, a few observations:
- It's CoT has changed dramatically. I find the heavy Astra distillation claims to be plausible
- It has strong benchmarks scores, and from what I've seen online it's much better at 3D games and such, but that's not so relevant for my work. On my internal evals it is more token efficient, but it actually doesn't score any higher than the previous model
- It's only been a couple days, but it does seem a bit more focused and reliable. DS4 0731 and DS4.1 have been a bit sloppy (as in not-careful, although they did tend to fall into Claudisms as well), and 1017 seems to be a bit sharper.
- I'm running it through some more challenging work, but I'm not sure if it's really at GLM-5.3 level, especially for cyber/RE/hardware poking, which GLM has excelled at.
200+ tok/s on local infra? That's crazy. How much did you pay for your hardware?
CoT = Chain of Thought (I had to look it up)
Thanks for the insights. If you edit your post to include the definition, I'll delete mine.
> because I'm guaranteed it's not being changed on me
This is something underrated. With an open weight model I can evaluate the performance for my use case, and know for a fact the model will stay the same over time. No arbitrary “model welfare” rules being introduced later on, no undocumented change to capabilities or performance
Somehow GPT 6.1 Sol (medium) is cheaper per task ($0.21 vs $0.26) and faster per task (2.9 vs 4.9 minutes) vs v4.1 Flash 1017 (max). I hope AA adds other reasoning effort results of Deepseek so it'd be more realistic to compare it with other models, because IMO using models at max effort is very unrealistic for most people, they then take a huge amount of time and tokens for a small performance uplift.
I wonder what AA's funding and affiliations are. I get the sense that they have their hand on the scale at least a bit.
Yeah, this always bugged me. Then again, unlike American models, surely Chinese models are far too expensive for them to test them at every difficulty ;)
I don’t know about 1017 but according to my limited testing of 0910 and previous v4-flash-0731 on some of my use cases, max simply used a lot more tokens (almost twice as many IIRC) for little to no benefit over high. So yeah I really don’t see a reason to use max.
For DS41F (both the original and 1017) I've found low (25/100) reasoning to be the optimal for my tasks at least (max doesn't tend to better, and if you can't do it w/ low imo you should just switch to a smarter model). At low it's about 1/3-1/2 max's token count (and price and speed).
A bit like influencers of the benchmarking world. Milking it while all this trends. Are there financial/tie-up disclosures that are audited for benchmarks?
AA prices are always kind of hand wavy for open models. I mean if you rent a cheap gpu and run inference yourself...or heck, what about a cpu instance with a bunch of ram, you can do that with llama.cpp for under a $1/hour. Ok not that, so are we talking about the median spot price per token from a basket of inference providers? Which ones? They actually vary quite a bit, even on openrouter, for this particular deepseek checkpoint it's between $0.15 and $1.20 ... that's like 8x spread...
They have some math here https://artificialanalysis.ai/methodology but ok how are the prices weighted among the providers... and even if you do that 7:2:1 math it doesn't match the providers. Specifically the model in question: https://artificialanalysis.ai/models/deepseek-v4-1-flash-101... doesn't match the first listed provider https://deepinfra.com/deepseek-ai/DeepSeek-V4.1-Flash-1017
Do the math yourself, it's actually $0.08 not the AA claimed $0.10 (and that's the more expensive of the two pricing tiers). That's a pretty large error bar.
Also many of the providers are now doing "peak" and "offpeak" pricing (deepseek.com itself for instance) so it can differ dramatically. These "peak" times depend on where the service is most popular. You can basically ride "off-peak" pricing by cascading from global provider to global provider.
The majority of the time taken in an inference call is doing the inference so the extra latency of traveling across an ocean is actually a fairly small wall-clock tax here. People still seem to be partial to nearby servers for ai-related workloads but really, try some decent ones 10,000 km away, you probably won't notice.
I know AA thinks highly of the price metric (I've been to their events, even have their hat, they brag about it) but I have learned to ignore it as noise.
It's an especially strange metric with open-weights decision models. I mean I can run those on my laptop ... cost is very abstract there.
But for closed-source models with only a handful of providers (sometimes just 1) great, the price is a very reasonable number.
For the open models a more useful thing would be like "100 tok/s spot price build" and then it's using pcpartpicker or something in some kind of partnership and can query a build for buying a machine right now that can achieve a certain benchmark number and maybe something for less interactive workloads like "5 tok/s". Of course getting some base unit of power cost in there might be important as well since the power bill is the majority of what you pay for when you rent a commodity gpu.
this matches my experience tbh. 6.1 sol on medium is quite good at getting things right the first time without overthinking and i prefer it over xhigh or max usually. so i'm not surprised it ends up cheaper.
What I really want to know for models like this is if I can have them run on really lame consumer hardware (consider 128gb of ram, 2080ti) with extremely low TPS, but with reasonable enough intelligence that you can walk away and expect something to bake really nicely while you sleep.
Perhaps I'm losing touch with the edge, but it doesn't seem like this is quite possible with the hardware baseline I'm talking about...
"Model like this" not sute about ds v4.1 but qwen3.8 flash next is currently running on 10gb 3080 + 64 gb ddr5 ram at respectable 40-70t/s with 1200pp at max context
I use DwarfStar DeekSeek V4 Flash on my Mac Studio with 128gb of RAM and it's pretty good - decent TPS and coding ability. Apparently the V4.1 model can also be run by streaming weights from the SSD - I've not tried this yet because it results in a lower TPS, but eventually this version of 4.1 should also be usable like this.
expert streaming with cache-conditional routing is very powerful (you can run 4.1 Flash on two Strix Halos and a 9070XT at around 50-70TPS decode in ~full quality), and that setup might be able to at least do a few tok/s with it.
You sure can, https://github.com/lxe/dsx gets you this deepseek on 2x3090s and 128 gb ram at about 30 t/s
At around 75% of the full model intelligence
You can get something relatively capable and usable with your configuration and Strata: https://github.com/Niko1221/Strata
Nvidia stopped manufacturing the 5090, so you and people like us cannot run these models on our computers and are forced to use American SOTA so they don't lose the game...
Mimo 2.6 pro scores 46 and is half the price (though a lot less throughput). Im not sure why nobody has made the same fuss around it
Hmm if you just compare raw token prices on Openrouter right now, Mimo 2.6 Pro is roughly 4x the price.
Does deepseek v4.1 flash 1017 really take that much longer per task?
Mimo from Xiaomi has safety refusals on things compltely unrelated to safety. It's good when it works but you need another model to continue the work when Mimo craps out.
Just want to note that this model still does not work (read: sucks) for recognizing sheet music.
The only model that works for me is Astra, but it's expensive.
does the exact model matter anymore? pick one and develop context with it, imho, and just get to work.
then again, I'm not a heavy AI user, so maybe I'm missing something.
It does. Even looking at cases where a both a better and a worse model accomplish a given goal the difference between a solution and a good solution is important. And will save you time by not having to iterate as much with it.
For this model in particular, even if it didn’t get stronger and it simply thought less that is a big win on its own. DS4 can be served at a hit tok/s, but it takes a while because it thinks for so long.
Really wild that Deepseek's model is 536+16B (passive+active) parameter model and its really comparable to Mimo v2.6 Pro which is the current open source SOTA model which is at (978+42)
This is half the size of Mimo and comes pretty damn close to it and I feel like lots of tasks can be done by it.
I really like its speed as well but its quite verbose from the benchmarks, IMO, this should be tackled in the near future as well. I wonder if specific fine-tuning can happen just to make the model token-efficient.
Also, slightly offtopic though deekseep has good tps but Nvidia's nemotron recent 120B model also has some really crazy tps. I wonder if we are gonna see some really good tps on general hardware.
The best part is that the amount of current innovation happening within the space means that a lot of good ideas are bound to be absorbed by one or the other and we will get many of these improvements handed to us :-D
I think that such engineering is also the engineering of the trade-offs and on what axis to play on. You can be SOTA but with very low speed, extreme price and other factors. I think its interesting to see how each engineering lab approaches these trade-offs and what they pick. But whatever they pick, its best because its more choices for us to pick upon from (especially if its open weights)
high tps and braindead model is a dangerous and fun thing to watch.
give it browser access and it's a speed-run to a cloudflare bot detection freak-out in minutes.
At risk of being blunt: Why the surprise that a model that activates 12B parameters is faster than one that activates 16B parameters which is faster than one that activates 42B parameters?
It feels like that would be the expected result, all else equal.
It's too bad they couldn't make it smaller, but still very nice improvement.
The new model:
DeepSeek V4.1 Flash 1017 (Max) 552B A16B
Matches
GLM 5.3 753B A40B
MiMo 1T A42B
Nicely done!
my daily model now