The summary "There are still clear limits. Gemini 3.5 Flash remains a better practical choice [than GPT 5.6 Sol] for high-volume detection and counting in our benchmark, especially at its price." seems rather understated !
GPT 5.6 Sol was outperformed on all benchmarks by Gemini 3.5 Flash, apart from a single exception (OCR) where Fable was the winner.
Gemini 3.5 Flash not only outperformed GPT 5.6 Sol, but did so at 1/3 of the cost.
Hi, I’m the author of this blog post. I wrote it about 4 weeks ago, and the VLM world is moving so fast that it’s already kinda outdated. I think Gemini 3.7 Flash might be a better choice now, especially when you factor in the price.
Here’s a comparison of the best low-cost models I put together last week. What’s crazy is that Gemini 3.7 Flash is now 50% off on OpenRouter, and this chart doesn’t even account for that discount. https://x.com/skalskip92/status/2088032652301304121?s=20
Curious why you didn't try Gemini 3 pro? That is the model I've been using for OCR entry of handwritten datasheets (JPGS of datasheets, structured JSON output). At my scale, the cost of 3 pro is basically not an issue, but if there are improvements in quality, I'd definitely be willing to explore other models
In my experience starting with Gemini 2.5 Pro, moving to 3 and 3.1, 3.5 Flash, 3.6 Flash, and finally 3.7 Flash, 3.7 Flash is just as good if not better than 3 especially on high resolution mode (same token count per page as 3.1).
I run complicated, messy PDFs through these models. 2.5 Pro required a lot of kludgy hacks to get it to fully "see," but from 3.1 pro on I've removed many of them and haven't spotted problems.
3.7 Flash scores better than 3.1 pro on most benchmarks, leading me to believe that even if your OCR requires reasoning to interpret text or data, 3.7 Flash is probably going to be better.
3 Pro is quickly approaching one year old. There's almost no reason to benchmark it, especially since a new version of Gemini Pro was supposed to be released mid 2026 and hasn't seen the light of day.
That would make sense if we already knew that, for these kinds of tasks it was significantly worse. The tests that I'm aware of for these tasks show it as still performing near the top.
3 and 3.1 Pro are both marked as deprecated by Google. Even if they're the best Google offers, it would be foolish to choose a model that's explicitly deprecated.
It's not a technical problem, it's a commercial one. If Google can't ship a model to replace the one they deprecated, that tells you everything you need to know about choosing a Gemini model for whatever you're trying to do.
these models aren’t successors and barely have a common ancestor, they are independently baked in the training oven and assigned a semantic version randomly by someone trying to show initiative but not trying to do on the toes of the last guy who got promoted first
So 3 pro is outdated and will likely never exit preview
The “flash” and “lite” models are the real “pro” in colloquial ideas of fleshed out and capability, at this point.
they’re better, faster and cheaper, larger context windows keeping up with the industry and more
Gemini tops their vision evals [0] by a mile, with 4/5 top spots going to variants of it. Qwen is the only other contender, likely due to how good it is for object detection, where it crushes the competition [1].
Gemini is still my top choice within production software for typical data extraction from unstructured data. Gemini Flash Lite feels like a cheat code for speed, and it's really cheap.
Some other Chinese models are also fast and cheap, but a harder sell in a U.S. production environment.
Speaking from experience here, flash lite models have amazing price, speed, and perform far above their size, but are susceptible to very bad instruction following and recall when either complexity or context size inch up. They’ll just forget to apply your instructions to portions of the input, and repeat parts of the input that should be returned verbatim as direct quotes but with subtle changes (breaking urls, for example).
This was 3.5 flash lite, actually, and after prompt tuning. It was very clearly an issue that correlated with input (JSON array) size, the more elements in the batch, the higher the error rate.
3.0 flash (not lite) handled it like a champ though, fwiw.
Hi, I’m the author of this blog. It depends on how strong of a model you need, but in general, Qwen is easily the best among the Chinese models right now.
Googles local gemma models which target roughly the same parameter count range, are known for being a lot better at vision tasks than qwen, no idea if 3.8 has changed that though
Anecdotally, Gemini Flash is the leader for a particular use case of mine and has been since at least version 2.5. But now there's also Luna as the first real competitor thanks to the price cut.
My worry is that this is a zero-sum game and when Gemini catches up on coding, it'll regress to the mean in other areas.
The prompt sensitivity is fascinating. A simple coordinate-format change shifting detection performance this much shows how much interface design still matters.
Gpt is really good in vision stuff, or at least their MoE seems to be really cohesive. From my experience Claude models can be really good at language but the moment they need to look at a picture and decide why the design is not good what parts need improvement it degrades a lot. My easiest benchmark is giving them a screenshot of a feature in my app and tell it "identify non-normative UI blocks and improve readability and consistency". Sol does a great job at re-structuring the page into composable units that build upon each other and the general looks and feels of the app. Claude tends to over-focus one one part while completely forgetting about the rest or the cohesion as a whole.
There's a lot of objective principles and decisions that go into subjective quality; if you don't know the field well, asking LLM for assessment is a good way to discover all that.
Of course. It’s OK, but it tends to generate very cliched “AI” UIs with little originality. Despite the skill spending a lot of time coaching the model into avoiding that!
My exposure to Claude-produced UIs is limited, but I have started to notice certain design trends they tend to have in-common, which might be becoming hallmarks of AI-produced UIs - the same way we've started noticing the clichés of low-effort LLM-generated text.
FWIW, the summary-description[1] of "frontend-design"[2] gives me a few things to pick at:
> create polished code
Methinks only if you're using it with a very popular framework like React. What happens if you ask Claude to make the UI in WinForms or MFC?
> high-impact animations
That's bad UX 101 right there: animations in a UI exist as an affordance to the user, and never for its own sake (e.g. macOS's "genie" animation when you minimize a window to the dock exists so the user knows where they can restore the window from). The only people who actually want "high impact animations" in software are salespeople who want something for demo purposes.
> generic system fonts, predictable purple gradients, and cookie-cutter components.
This screams wanting to be different for the sake of standing-out, not because it results in a better software product; users benefit when their software fits-in with platform conventions: if you refuse to use a stock checkbox <input> or <select> drop-down and instead use your own entirely custom component solely for aesthetic reasons then you are producing worse software. There's nothing wrong with system-fonts, but your site will look ugly after your third-party font-host CDN shuts-down and turns into a walking CSRF factory.
> thoughtful typography with unexpected font pairings
The above fragment set my alarm-bells off. Yikes.
> scroll-triggered interactions
Not every web-page should be an Apple.com product brochure page. This is also a fantastic way to make your webpage horribly inaccessible.
------
The SKILL.md itself[3] grinds my gears too:
> Approach this as the design lead at a small studio known for giving every client a visual identity that could not be mistaken for anyone else's.
Claude has no way of knowing what designs are actually unique or not...
> For web designs, the hero is a thesis. Open with the most characteristic thing in the subject's world, in whatever form makes sense for it: a headline, an image, an animation, a live demo, an interactive moment
...this is exactly what everyone else's web-pages look like!
> For calibration: AI-generated design right now clusters around three looks: (1) a warm cream background (near #F4F1EA) with a high-contrast serif display and a terracotta accent; (2) a near-black background with a single bright acid-green or vermilion accent; (3) a broadsheet-style layout with hairline rules, zero border-radius, and dense newspaper-like columns
...I called this out weeks ago[4], lol.
and I could go on. This is all quite painful to read.
Hi! I’m the author of this blog. GPT-5.6 is much better at vision than previous GPT versions, but it’s still much weaker than Gemini 3.5 Flash or Gemini 3.7 Flash, which was released last week. One interesting approach is to use Gemini through a tool call.
Hi! I’m the author of this blog. I had the same intuition, but together with the OpenAI team we figured out that the issue was image resolution. GPT-5.6 doesn’t handle large images well.
It is funny to me seeing Sol used for what a "traditional" AI model can do already (counting pills).
We have vision models for our pharmacy and I could never imagine taking the latency hit to use a Sol in our robotics, it would be likely 25-50x slower.
Building a dataset is expensive, manual annotation is expensive. Datasets don't exist in every niche.
I remember around 2013-15 people were scoffing at uses of deep learning CNNs for various things, because why don't you just use an SVM on HOG features? Or face detection is solved, just use Viola-Jones.
What if you give the benefit of doubt and assume the author knows about alternatives and uses VLMs for their strengths? They use it to auto-annotate training data for regular deep learning models.
How are we supposed to pay off all these data centers and chips if you’re not willing to burn a microwave burrito worth of electricity for each prescription? Think of the benchmarks
Ironically, the pill counting example selected to showcase "the best vision model" can be easily solved with OpenCV template matching, a technology created 25 years ago.
I'm assuming you mean that this tech became available in OpenCV 25 years ago, but as it turns out, the underlying tech can be traced back much further, at least as far as 1977! :)
https://ieeexplore.ieee.org/document/1674847
G. J. Vanderbrug and A. Rosenfeld, “Two-Stage Template Matching,” IEEE Transactions on Computers, Vol. C-26, No. 4, pp. 384–393, April 1977.
DOI: 10.1109/TC.1977.1674847
The point is that it's general. It can do this task and many other tasks and it doesn't need custom development like OpenCV does. Of course if you only want to count pills and you want it to be cheap/fast you're still better off using OpenCV.
Basic Template matching has severe limitations around scaling, rotation, and perspective. In my experience it greatly underperforms compared to deep network object detectors. My experience- and I imagine others have different experiences- is that SIFT techniques also fail pretty badly with noisy data.
That's correct, and I was specifically referring to the example chosen - where scale and perspective are known. Template rotation is relatively easy as well - but partial obstructions would pose a problem.
Another application where template matching would work brilliantly? Car counting in parking lots using satellite imagery.
Source: I did this [1] using OpenCV and template matching. Outperformed "Cars Overhead with Context" models.
Gemini 3 Flash should really be included in this comparison. Or at least 3.7. In most of my testing, 3.5 and 3.6 were both a downgrade in terms of vision capabilities, relative to 3, and at a much higher cost. 3.7 is slightly better than 3, finally.
For 3, pricing for image tokens was the same as text tokens. Since they don't indicate a difference on 3.7, I would assume the same holds. And as far as I know the number of image tokens is the same for both (depending on the detail level you pick, but it's generally around 1k per image).
So they're about the same, 3.7 is slightly more expensive. At least until the end of the year (when they raise 3.7's pricing).
Anyway, my point was that 3.5 tended to have worse performance and significantly higher costs. 3 and 3.7 are both better and cheaper than 3.5.
The second answer is far more revealing than the first:
OP:
> do you think you did a good job there
ChatGPT:
> I spent 15 minutes, emitted several fake-sounding “tracing the puzzle” progress updates, and then gave a confident permutation without showing that I had actually followed the lines correctly. It reads much more like I guessed than solved it. The only part I did well was obeying the “no Python or tools” instruction.
My observations:
1) Sarcastic tone suggests pre-prompting, or frequent (and therefore stored in memories) denigration of the model in past conversations. I'm leaning the former - it sounds like it was instructed to read admission of defeat.
2) The part about "no Python or tools" is setting the model up for failure.
I mean, this task is, for a human, basically a game of "simulate a line following robot in your head". Pretty sure a VLM could solve that if it was allowed to do the same thing. Off the top of my head, an algorithm like:
1. Identify start and end points
2. Foreach start point, follow next pixel minimizing angle, until endpoint is reached.
3. Report answer
It's literally what every human facing this task does.
EDIT:
My attempt - same image, prompt altered to allow for code (but still no search/external checks), solved in 1/5th of the time, correctly, and (going by thinking trace summaries that I don't think show up in shared chats), basically the same way I'd approach it, by tracing the lines, coloring them as it goes.
INB4: I know this is now not a pure vision check, but it really doesn't make much sense to diss models for failing to solve tasks explicitly designed to teach humans to externalize computation that's hard to do in their heads (i.e. kids, crayons, coloring paths).
Still, if such things are becoming a benchmark for tool-less evaluation, it's only a matter of time until the models learn - much like humans learn in school - to follow algorithms mentally, essentially emulating an ad-hoc computer in their head.
No pre-prompting, although I can't be sure it didn't use memories. "No tools" should theoretically have prevented it from looking up memories. FWIW, Grok and Gemini both failed in a similar way.
The real pain point is that at work, I use Codex and I'm currently working on a project that involves debugging some polyline topology, very similar to the path following puzzle. The vision is completely useless here.
Your VLM idea sounds good. Theoretically, the inverse problem (generating an SVG of a pelican riding a bike) can also be solved with a VLM that plans out how to draw it, not unlike a human planning out a path for their hand to follow.
Dumb question: When your testing "ChatGPT 5.6 Sol" are you testing an actual LLM or some visual pre-processor stack that sits in front of it (along with a maybe a bunch of other such pre-processors) that is bundled into what's call "ChatGPT 5.6 Sol"? I.e. last I checked LLMs had a something like a 30-100K token alphabet to work with and it's hard to imagine how throwing pixels arrays at one directly would work.
I'm ~95% certain that images are tokenized, just like regular text, and fed directly in; that's the 'multimodal' part of these models. Now how this tokenizing works I don't know, and there might be some level of preprocessing, but it's certainly not converting the image into text and feeding it in to a regular LLM.
From our experiments it’s the best video captioning model in the world by a mile. This was not the case a year ago.
When reasoning got introduced a year ago to GPT 5, on average the model performed worse than GPT4-o for short video clip captioning
(Ie hallucinating actions that didn’t happen). The old GPT 5 was extremely finicky in terms of fps sample rate.
The other SOTA LLMs (like Gemini Pro) have clearly been optimized for long video understanding, since they can’t see almost anything sub-second (even if you up the frame sampling rate).
Sol is the first model we’ve seen to accurately caption complex sub-second movements (eg woman suddenly turns heard head to right). It’s robust to different fps sample rates so I can only guess that they trained on videos sampled at different fps.
It's not clear to me from the article, are they asking sol to output bounding box coordinates with some kind of structured outputs?
Anecdotal but I've seen it use python to crop, zoom, and "enhance" (fiddle with sharpness and brightness) images to read sections of handwritten census data from the 1800s. Feels like that there might just be a mismatch of capabilities when it comes to straight outputting coordinates but I bet the model is better at actually finding the answer given any tools available. Which I get is a bit of an apples and oranges situation.
I've also tried to use it to identify an old pair of glasses and it didn't stand a chance, so I do think it's not quite there yet when it comes to some vision tasks.
I currently have fable organize a bunch of 5.6 sol agents when working on my personal projects. This makes me wonder if I should add something along the lines of "For tasks that involve visual analysis, have gemini 3.7 look at images generated."
Overall I've been hooked on using agents from different companies for what they are best at (Thanks to Theo). Fable is expensive, but unmatched for planning and top level organization of other agents. Sol is fast, will persistantly go after goals (sometimes to its detriment), and does well with computer use.
I concur with your conclusion, Qwen 3.8 has exceptional Vision Capabilities. The other commenter mentioned "I thought Qwen 3.8 didn't have Vision", it does, just not on the open weights version, only via official API.
I understand why you would like to use an LLM for vision. I do it myself often enough. I don't understand however, why the pill detection and counting is included in this benchmark. That is a task which you would perform with OpenCV right?
In my personal mini benchmark minicpm-v-4.6 scores amazingly well. Its a 0.8B model which runs fine on many consumer hardware.
Generating datasets to train more efficient models is a common use case for VLMs, especially frontier ones. It makes it much cheaper to create that initial dataset and you can abuse the nondeterminism of LLMs to identify data for human review (if they don’t converge, escalate to a human).
Especially the pill counting example. The best model was shown at 81.1% accuracy, which is a terrible rate for pharmacy scenarios. It seems like implementors would be better off instructing the models to use deterministic tools (like OpenCV) until the models are at 99.99% accuracy (or whatever an acceptable error rate is for pharmacy techs).
OpenCV is no longer hard to use, it just takes longer. Still, a little more complicated than asking LLM to count.
To use an LLM, you just prompt it with an image + text saying "count the pills in this image".
To use OpenCV, ... you just prompt an LLM with an image + text saying "count the pills in this image, using OpenCV instead of eyeballing it".
(I like to throw in "produce intermediary artifacts so I can see the process" for more difficult tasks; this helps the model avoiding making hallucination-prone leaps and gives more opportunities to self-correct. At a cost of extra time and tokens, of course.)
Using OpenCV without an LLM? Nah, not touching that, I don't have free weekends to waste anymore.
So far I haven't seen a single model succeeding at transcribing sheet music, but I just tested it again with 5.6 Sol and it nailed the small test case. Fluently reading music requires multiple years of training for most people, but I feel like accurately following the horizontal lines trips up vision models in particular.
If you're doing any kind of inference that is multi-modal and non-factual, opinions and biases will affect any kind of assessment of a visual that you provide to a model.
For example, a UI / UX professional being asked to appraise a website screenshot may determine that the image in question has "desirable" traits which are inherently not deterministically measurable. Such as, if the interface elements have strong information hierarchy, or if they are deemed to be "fashionable" with current UI trends.
Is 16 px or 14 px a better font-size value for a subheading, in a hypothetical layout? Immediately that kind of decision, where both options are objectively good for 12 px paragraph text, suddenly becomes an issue of taste that cannot be evaluated crudely by an algorithm.
As a design system engineer I usually have to fight against the taste of the designers. (And I consider it natural.)
But, if you have a proper well documented design system and you tell the LLM to use the DS and to avoid styling hacks they can generally do it. Even the dumber ones than Sol 5.6.
Of course only if the design is achievable in the design system.
One of my friends (and BIL) own an architecture firm. They use AI to generate and quickly update renderings but they run into the equivalent of the 6 fingered hand problem. I sent him this article I wonder if the updated models can catch and fix mistakes made by previous models.
I've decided it's "good enough" after I saw it properly quote a string of text that was very roughly highlighted within a nested visual context. It also identified the context correctly (modal inside webapp inside screenshot of user desktop).
I run the free service https://countrx.app/ so i have some idea what goes into counting.
The performance as a general model is indeed really impressive and i think they might actually win compared to fine tuned models.
Their feedback loop of training on user data is incredibly strong.
I've learned that lots of accuracy results depends on threshold configs, which llms should be able to dynamically set.
Or the future will develop in llms using fine-tuned models as tools?
Inference cost and speed does still seem to be below user expectations.
But for being able to one shot with this accuracy... IMPRESSIVE
I really like the combo 5.6 Luna & Sol for price and performance and would be perfectly happy if they stayed here for a moment without mucking about with sidegrades that I think AI evolution has often felt like lately.
It’s gotten so good that I now have infrastructure to render all mermaid/plantuml in my project to png and have AI’s always load both text and image versions. And they are instructed to review the rendering as part of the diagramming cycle (for layout, salience, usefulness, etc). They can now produce useful diagrams that help reach shared architecture understanding.
I recently used it at grocery stores in a foreign country. Photographed the whole aisle and told it to find Y (detergent, softener, glue, sour cream, whatever), at the same time recommend the best Y for whatever reason. Worked marvelously, including the cases where the object wasn't present and it told me there was nothing useful.
I asked then, can you crop the exact image of how does the item look like and where is it in the aisle - did that perfectly as well.
I will add that all frontier models were fine with such tasks from the early 2024's.
I would love more vision benchmarks! Once I asked the model to inspect a completely black picture and it hallucinated a nice wooden kitchen wall. Took me some time to figure out where the kitchen came from...
5.6 Sol looks nice, but the Gemini 3.5 Flash comparison is interesting. It’s cheaper and still came out ahead on detection and counting, which doesn't really give me much of a reason to use Sol since Flash is much cheaper and hence much easier to scale. Not to mention we now have 3.6 Flash too
One of the use cases I've wondered about for AI is giving it a picture of the "spice wall" in a grocery store and asking it to find all jars of e.g. cardamom. This takes me an annoyingly long time to do when I'm shopping, so it would actually be useful.
It's vision capabilities poisoned my cucumber bed, misidentifying the malaise and having me spray them down with water, which only spread the fungus that gemini later informed me was actual cause, which I went and checked myself.
I hope that whatever was lost at GDM in the last few months, didn't include their extra focus on vision capabilities.
In my practice Gemini models are far better than anything on the market in terms of vision, also it's worth to mention that current Gemini flash is 3.7, so it got 2 updates since 3.5 which beat GPT-5.6 Sol in this comparison.
Hi! I’m the author of this blog. I wrote it 4 weeks ago, and it’s already a bit outdated. Gemini 3.7 Flash came out last week, and considering the price, it’s easily the best vision model right now: https://x.com/skalskip92/status/2088032652301304121?s=20
Do you think we’re getting closer to models that actually understand what they’re seeing, or are they just getting really good at recognizing patterns?
All I'm fine with for now is that I can almost exclusively communicate with Sol through collages and my scribblings (all kinds of web page / block screens with all kinds of arrows and text all over the place) This was not practically ppossible in 5.5 and a tragedy in 5.4. Not sure how much weight is codex uploading in higher res carrying here but it's great to work with.
I've been using Reolink for years and been very satisfied with it.
The only quip is the default UI isn't very good. When changing that reaches the top of my priority list, I'll switch it since they don't force you into a walled garden. Plan is to run it through frigate into HomeAssistant and use a UI from them. I've never used frigate before though so it'll be a learning process if plug and play solutions aren't already available
I agree. It did very well on an extremely challenging task.
I asked it to recognize and draw the very faint reflection of what I was wearing, visible in only a tiny black part of a very brightly lit poster behind glass.
In addition, the poster itself also happened to contain similar clothing.
For the last 2 weeks I've been trying to get Codex to "outpaint" a wonderful image it generated as placeholder art for a level background.
After I increased the game's resolution, I asked it to increase the image's size while keeping the same scale and existing content, and gosh, it constantly keeps getting something wrong no matter what I tell it, even on Sol Max with the $100 Pro subscription.
An organically-grown meat-based pixel-artist could have recreated the image and more within 2-3 days, in exchange for food and shelter.
At first I intended to create a tileset and asked it for several variations of what a hypothetical tilemap created from the planned tileset would look like.
The previews it generated were amazing but wouldn't really be possible as a grid-based tilemap, with lots of clusters and overlaps of elements of varying sizes.
So I just decided to use the preview as a static scrolling background, but it's been a pain to get it to add more content around the edges that still tiles with the existing image at the same scale.
I hate these "The best X thing Y has ever released".
Unlike when Apple says "it's the best iphone we've ever made", LLMs are more or less interchangeable. So "OpenAI's best model" means nothing if "Anthropic wipes the floor with them" or "[open weights model] is 10x cheaper for 1% less quality".
As a reader, it feels like these titles are click bait.
It's really quite good! I was amazed recently by its utter inability to read some faded handwritten cyrillic on the back of a wood carving - 3 or 4 words only, reasonably clear letter forms I found recently, and then stepped back a bit and thought about how insane that was as a benchmark - I just expect it to work so reliably on other OCR and translation tasks that it was surprising to encounter such a failure
Hi! I’m the author of this blog. I regularly benchmark new VLM releases. You can check the results for Qwen3.8-Max and Qwen3.8-27B here: https://playground.roboflow.com/evals
Me too. This is an interesting comparison but in my experience Qwen and Gemini have typically been the top contenders for image related tasks. For that reason it would be great to have the comparison here, as I'm not surprised by Gemini's dominance over the other models.
Fuck ack. I'm working on a new benchmark that combines strong visual requirements with tool and coding requirements. I haven't even tested Sol yet, but between Sonnet, Terra & Luna I already see much better results from OpenAI's models. I'm not releasing anything yet as I still have issues in my harness that need to be fixed.
The summary "There are still clear limits. Gemini 3.5 Flash remains a better practical choice [than GPT 5.6 Sol] for high-volume detection and counting in our benchmark, especially at its price." seems rather understated !
GPT 5.6 Sol was outperformed on all benchmarks by Gemini 3.5 Flash, apart from a single exception (OCR) where Fable was the winner.
Gemini 3.5 Flash not only outperformed GPT 5.6 Sol, but did so at 1/3 of the cost.
Hi, I’m the author of this blog post. I wrote it about 4 weeks ago, and the VLM world is moving so fast that it’s already kinda outdated. I think Gemini 3.7 Flash might be a better choice now, especially when you factor in the price.
Here’s a comparison of the best low-cost models I put together last week. What’s crazy is that Gemini 3.7 Flash is now 50% off on OpenRouter, and this chart doesn’t even account for that discount. https://x.com/skalskip92/status/2088032652301304121?s=20
Curious why you didn't try Gemini 3 pro? That is the model I've been using for OCR entry of handwritten datasheets (JPGS of datasheets, structured JSON output). At my scale, the cost of 3 pro is basically not an issue, but if there are improvements in quality, I'd definitely be willing to explore other models
In my experience starting with Gemini 2.5 Pro, moving to 3 and 3.1, 3.5 Flash, 3.6 Flash, and finally 3.7 Flash, 3.7 Flash is just as good if not better than 3 especially on high resolution mode (same token count per page as 3.1).
I run complicated, messy PDFs through these models. 2.5 Pro required a lot of kludgy hacks to get it to fully "see," but from 3.1 pro on I've removed many of them and haven't spotted problems.
3.7 Flash scores better than 3.1 pro on most benchmarks, leading me to believe that even if your OCR requires reasoning to interpret text or data, 3.7 Flash is probably going to be better.
3 Pro is quickly approaching one year old. There's almost no reason to benchmark it, especially since a new version of Gemini Pro was supposed to be released mid 2026 and hasn't seen the light of day.
That would make sense if we already knew that, for these kinds of tasks it was significantly worse. The tests that I'm aware of for these tasks show it as still performing near the top.
I think it definitely makes sense since it's still the best Google has to offer in the "pro" tier.
3 and 3.1 Pro are both marked as deprecated by Google. Even if they're the best Google offers, it would be foolish to choose a model that's explicitly deprecated.
It's not a technical problem, it's a commercial one. If Google can't ship a model to replace the one they deprecated, that tells you everything you need to know about choosing a Gemini model for whatever you're trying to do.
3.1 Pro is not deprecated!
https://ai.google.dev/gemini-api/docs/deprecations
That link shows 3.1 pro listed as deprecated with no replacement model.
No, that's the "preview" version (gemini-3.1-pro-preview) aka the beta/early version before the official release of 3.1 Pro.
The “pro” moniker means nothing
these models aren’t successors and barely have a common ancestor, they are independently baked in the training oven and assigned a semantic version randomly by someone trying to show initiative but not trying to do on the toes of the last guy who got promoted first
So 3 pro is outdated and will likely never exit preview
The “flash” and “lite” models are the real “pro” in colloquial ideas of fleshed out and capability, at this point.
they’re better, faster and cheaper, larger context windows keeping up with the industry and more
They are smaller models, and you can tell. Small models make dumb common-sense mistakes that big models never do. This is the "smell" many talk about.
Do you have cases where you still see 3.1 pro outperforming 3.7 flash?
Yes, for complex questions of biology, physics, and analysis of anomalies.
3.7 Flash is better at coding, sure, but AI is not just for coding.
hasn't been an issue since 3.5 for me, what have you seen, say, in the last two months
For complex questions of biology, physics, and analysis of anomalies, 3.1 Pro is still better than 3.7 Flash for me.
3.7 Flash is better at coding, sure, but AI is not just for coding.
At this point, VLM benchmarks should probably come with an expiration date. A four-week-old leaderboard can already be measuring a different market.
What about Gemma ?
Gemini tops their vision evals [0] by a mile, with 4/5 top spots going to variants of it. Qwen is the only other contender, likely due to how good it is for object detection, where it crushes the competition [1].
[0] https://playground.roboflow.com/evals
[1] https://playground.roboflow.com/evals/object-detection
Yeah I was thinking about giving Luna a go with my PDF data extraction, but I think I‘ll stay on Gemini. It does a very good job.
Gemini is still my top choice within production software for typical data extraction from unstructured data. Gemini Flash Lite feels like a cheat code for speed, and it's really cheap.
Some other Chinese models are also fast and cheap, but a harder sell in a U.S. production environment.
Speaking from experience here, flash lite models have amazing price, speed, and perform far above their size, but are susceptible to very bad instruction following and recall when either complexity or context size inch up. They’ll just forget to apply your instructions to portions of the input, and repeat parts of the input that should be returned verbatim as direct quotes but with subtle changes (breaking urls, for example).
Yes you have to continuously tune the prompts ever so subtly. 3.5 is a lot better than than 3.1 tho.
Important to remember that json schema instructions take precedence over the normal prompt, so move as much into property descriptions as possible.
This was 3.5 flash lite, actually, and after prompt tuning. It was very clearly an issue that correlated with input (JSON array) size, the more elements in the batch, the higher the error rate.
3.0 flash (not lite) handled it like a champ though, fwiw.
Yeah Gemini 3.5 Flash Lite is really good. Which Chinese models can you recommend?
Hi, I’m the author of this blog. It depends on how strong of a model you need, but in general, Qwen is easily the best among the Chinese models right now.
Over the last two weeks, Qwen released two new models. Qwen3.8-Max is totally insane, but it’s only available through the Alibaba Cloud API. I wrote a similar blog covering Qwen3.8-Max: [https://blog.roboflow.com/qwen3-8-max/](https://blog.roboflow.com/qwen3-8-max/)
If you’re looking for something you can run locally, Qwen3.8-27B might be a great option. On Friday, I did a quick comparison between Qwen3.8-Max and Qwen3.8-27B: [https://x.com/skalskip92/status/2088411215441621469?s=20](https://x.com/skalskip92/status/2088411215441621469?s=20)
Googles local gemma models which target roughly the same parameter count range, are known for being a lot better at vision tasks than qwen, no idea if 3.8 has changed that though
Really? Gemma4-31B should be better than Qwen3.8-27B? I'm happy to test that.
I've been using Qwen3.5-9B, hosted locally for PDF data extraction and it performs pretty well when extracting data from tables and infographics
Gemini is honestly an excellent LLM with many capability strengths.
For example, 3.7 Flash is #1 on MMLU Pro and AA’s agentic spreadsheets/docs benchmark, etc. Yes, beating Fable.
Agentic coding is only one dimension.
Anecdotally, Gemini Flash is the leader for a particular use case of mine and has been since at least version 2.5. But now there's also Luna as the first real competitor thanks to the price cut.
My worry is that this is a zero-sum game and when Gemini catches up on coding, it'll regress to the mean in other areas.
thats so helpful - tysm
The prompt sensitivity is fascinating. A simple coordinate-format change shifting detection performance this much shows how much interface design still matters.
Anecdotal, opinion:
Gpt is really good in vision stuff, or at least their MoE seems to be really cohesive. From my experience Claude models can be really good at language but the moment they need to look at a picture and decide why the design is not good what parts need improvement it degrades a lot. My easiest benchmark is giving them a screenshot of a feature in my app and tell it "identify non-normative UI blocks and improve readability and consistency". Sol does a great job at re-structuring the page into composable units that build upon each other and the general looks and feels of the app. Claude tends to over-focus one one part while completely forgetting about the rest or the cohesion as a whole.
Assessing the subjective quality of a thing is in my experience one of the worst ways to use any LLM.
There's a lot of objective principles and decisions that go into subjective quality; if you don't know the field well, asking LLM for assessment is a good way to discover all that.
anthropic frontend-design skill does a great job with it.
Have you actually read the frontend design skill? It’s placebo at best. Very short and barely focused on design: https://github.com/anthropics/skills/blob/main/skills/fronte...
Have you actually tried using it?
Of course. It’s OK, but it tends to generate very cliched “AI” UIs with little originality. Despite the skill spending a lot of time coaching the model into avoiding that!
> UIs with little originality
Sounds like the kind of UI I like. (Take me back to Windows XP...)
What an annoying time for GitHub to go down.
Like every time
My exposure to Claude-produced UIs is limited, but I have started to notice certain design trends they tend to have in-common, which might be becoming hallmarks of AI-produced UIs - the same way we've started noticing the clichés of low-effort LLM-generated text.
FWIW, the summary-description[1] of "frontend-design"[2] gives me a few things to pick at:
> create polished code
Methinks only if you're using it with a very popular framework like React. What happens if you ask Claude to make the UI in WinForms or MFC?
> high-impact animations
That's bad UX 101 right there: animations in a UI exist as an affordance to the user, and never for its own sake (e.g. macOS's "genie" animation when you minimize a window to the dock exists so the user knows where they can restore the window from). The only people who actually want "high impact animations" in software are salespeople who want something for demo purposes.
> generic system fonts, predictable purple gradients, and cookie-cutter components.
This screams wanting to be different for the sake of standing-out, not because it results in a better software product; users benefit when their software fits-in with platform conventions: if you refuse to use a stock checkbox <input> or <select> drop-down and instead use your own entirely custom component solely for aesthetic reasons then you are producing worse software. There's nothing wrong with system-fonts, but your site will look ugly after your third-party font-host CDN shuts-down and turns into a walking CSRF factory.
> thoughtful typography with unexpected font pairings
The above fragment set my alarm-bells off. Yikes.
> scroll-triggered interactions
Not every web-page should be an Apple.com product brochure page. This is also a fantastic way to make your webpage horribly inaccessible.
------
The SKILL.md itself[3] grinds my gears too:
> Approach this as the design lead at a small studio known for giving every client a visual identity that could not be mistaken for anyone else's.
Claude has no way of knowing what designs are actually unique or not...
> For web designs, the hero is a thesis. Open with the most characteristic thing in the subject's world, in whatever form makes sense for it: a headline, an image, an animation, a live demo, an interactive moment
...this is exactly what everyone else's web-pages look like!
> For calibration: AI-generated design right now clusters around three looks: (1) a warm cream background (near #F4F1EA) with a high-contrast serif display and a terracotta accent; (2) a near-black background with a single bright acid-green or vermilion accent; (3) a broadsheet-style layout with hairline rules, zero border-radius, and dense newspaper-like columns
...I called this out weeks ago[4], lol.
and I could go on. This is all quite painful to read.
------
[1] https://claude.com/plugins/frontend-design
[2] https://github.com/anthropics/claude-plugins-official/tree/m...
[3] https://github.com/anthropics/claude-plugins-official/blob/2...
[4] https://news.ycombinator.com/item?id=49187385
i'd say this is something that has gotten orders of magnitude better with recent releases than it used to be, fwiw
Hi! I’m the author of this blog. GPT-5.6 is much better at vision than previous GPT versions, but it’s still much weaker than Gemini 3.5 Flash or Gemini 3.7 Flash, which was released last week. One interesting approach is to use Gemini through a tool call.
This response is not relevant to the this comment
What is a "non-normative UI block"?
Segments of the UI that don't conform to any other existing established design or conventions
areas that look weird
A better headline would be "Gemini 3.5 Flash is the best vision model". It tops almost every single benchmark shown in the article.
Penny sample shown looks like failed EXIF orientation registered by the model/harness. The coins are correctly marked, it's rotated 90 degrees.
Hi! I’m the author of this blog. I had the same intuition, but together with the OpenAI team we figured out that the issue was image resolution. GPT-5.6 doesn’t handle large images well.
OpenAI team sounds like they've misidentified the root cause for this particular case then.
Haha, ya at least to some degree, those boxes are in the right position, but rotated.
Good call out, I noticed the same rotation issue but pointing at EXIF data sounds about right.
It is funny to me seeing Sol used for what a "traditional" AI model can do already (counting pills).
We have vision models for our pharmacy and I could never imagine taking the latency hit to use a Sol in our robotics, it would be likely 25-50x slower.
Hi! I’m the author of this blog.
I’m evaluating these VLMs to figure out which ones are good enough to auto-annotate my data, so I can fine-tune my detector.
I wrote a bit more about this here: https://x.com/skalskip92/status/2080334344061694429?s=20
Did you evaluate any that could be self-hosted (or at least ow models), if so which one is the best you seen?
Take a look here: https://playground.roboflow.com/evals. We have few ~30B.
Thank you!
It seems Qwen is kicking ass, and Fable made me laugh when I saw it all alone on the far right of the graph :))
Agreed, this like asking a chainsaw to carve a wooden spoon. Impressive it can, but definitely not the right tech to scale.
LLM needs to setup an image classifier to use as a tool call.
Building a dataset is expensive, manual annotation is expensive. Datasets don't exist in every niche.
I remember around 2013-15 people were scoffing at uses of deep learning CNNs for various things, because why don't you just use an SVM on HOG features? Or face detection is solved, just use Viola-Jones.
What if you give the benefit of doubt and assume the author knows about alternatives and uses VLMs for their strengths? They use it to auto-annotate training data for regular deep learning models.
Now maybe, but the gap is closing.
How are we supposed to pay off all these data centers and chips if you’re not willing to burn a microwave burrito worth of electricity for each prescription? Think of the benchmarks
Ironically, the pill counting example selected to showcase "the best vision model" can be easily solved with OpenCV template matching, a technology created 25 years ago.
I'm assuming you mean that this tech became available in OpenCV 25 years ago, but as it turns out, the underlying tech can be traced back much further, at least as far as 1977! :)
https://ieeexplore.ieee.org/document/1674847 G. J. Vanderbrug and A. Rosenfeld, “Two-Stage Template Matching,” IEEE Transactions on Computers, Vol. C-26, No. 4, pp. 384–393, April 1977. DOI: 10.1109/TC.1977.1674847
Exactly my point. Template rotation is a trivial operation as well.
The point is that it's general. It can do this task and many other tasks and it doesn't need custom development like OpenCV does. Of course if you only want to count pills and you want it to be cheap/fast you're still better off using OpenCV.
Basic Template matching has severe limitations around scaling, rotation, and perspective. In my experience it greatly underperforms compared to deep network object detectors. My experience- and I imagine others have different experiences- is that SIFT techniques also fail pretty badly with noisy data.
That's correct, and I was specifically referring to the example chosen - where scale and perspective are known. Template rotation is relatively easy as well - but partial obstructions would pose a problem.
Another application where template matching would work brilliantly? Car counting in parking lots using satellite imagery.
Source: I did this [1] using OpenCV and template matching. Outperformed "Cars Overhead with Context" models.
https://abcnews.com/International/satellite-data-suggests-co...
I'm sure a typical frontier model would also be happy to write that opencv script for you, and it would do it well.
That is certainly pretty far from what was possible 25 years ago.
It 5..10 lines of code. :)
Gemini 3 Flash should really be included in this comparison. Or at least 3.7. In most of my testing, 3.5 and 3.6 were both a downgrade in terms of vision capabilities, relative to 3, and at a much higher cost. 3.7 is slightly better than 3, finally.
3 Flash never left "preview" status and is listed as deprecated.
https://ai.google.dev/gemini-api/docs/deprecations
but 3.7 flash is expensive for img inputs no ?
As usual for something so simple, Google's docs seem unclear: https://ai.google.dev/gemini-api/docs/pricing
For 3, pricing for image tokens was the same as text tokens. Since they don't indicate a difference on 3.7, I would assume the same holds. And as far as I know the number of image tokens is the same for both (depending on the detail level you pick, but it's generally around 1k per image).
So they're about the same, 3.7 is slightly more expensive. At least until the end of the year (when they raise 3.7's pricing).
Anyway, my point was that 3.5 tended to have worse performance and significantly higher costs. 3 and 3.7 are both better and cheaper than 3.5.
mystery to me is how the image tokens are calculated? 1MB is 1000 tokens ?
In the third vision bench result, Sol is 100% correct but the expected has 1 error. Seems like an oversight.
In the next bench, Sol looks like it’s correct again but the bboxes are rotated 90 degrees for some reason.
Seems to be due to the detection area being not fully accurate. Green vs red shows the difference between actual and detected
There is an extra green square where no egg is present, so it's a false positive in the expected.
Hi! I’m the author of this blog and benchmark. You’re right. I’ll fix it in the ground-truth dataset. Thanks for pointing it out.
Great, happy that it was helpful
Vision is still embarrassingly bad.
ChatGPT Pro with GPT 5.6-sol: https://chatgpt.com/share/6a834217-ca8c-83e8-a8e8-45d5b8797b...
The puzzle: https://activityvillage-files.s3.eu-west-2.amazonaws.com/s3f...
The second answer is far more revealing than the first:
OP:
> do you think you did a good job there
ChatGPT:
> I spent 15 minutes, emitted several fake-sounding “tracing the puzzle” progress updates, and then gave a confident permutation without showing that I had actually followed the lines correctly. It reads much more like I guessed than solved it. The only part I did well was obeying the “no Python or tools” instruction.
My observations:
1) Sarcastic tone suggests pre-prompting, or frequent (and therefore stored in memories) denigration of the model in past conversations. I'm leaning the former - it sounds like it was instructed to read admission of defeat.
2) The part about "no Python or tools" is setting the model up for failure.
I mean, this task is, for a human, basically a game of "simulate a line following robot in your head". Pretty sure a VLM could solve that if it was allowed to do the same thing. Off the top of my head, an algorithm like:
1. Identify start and end points
2. Foreach start point, follow next pixel minimizing angle, until endpoint is reached.
3. Report answer
It's literally what every human facing this task does.
EDIT:
My attempt - same image, prompt altered to allow for code (but still no search/external checks), solved in 1/5th of the time, correctly, and (going by thinking trace summaries that I don't think show up in shared chats), basically the same way I'd approach it, by tracing the lines, coloring them as it goes.
https://chatgpt.com/share/6a834f76-8240-83ed-acff-0c67af399d...
INB4: I know this is now not a pure vision check, but it really doesn't make much sense to diss models for failing to solve tasks explicitly designed to teach humans to externalize computation that's hard to do in their heads (i.e. kids, crayons, coloring paths).
Still, if such things are becoming a benchmark for tool-less evaluation, it's only a matter of time until the models learn - much like humans learn in school - to follow algorithms mentally, essentially emulating an ad-hoc computer in their head.
No pre-prompting, although I can't be sure it didn't use memories. "No tools" should theoretically have prevented it from looking up memories. FWIW, Grok and Gemini both failed in a similar way.
With Python, it was able to successfully solve it in 9 minutes: https://chatgpt.com/s/t_6a8350ecddfc81919328caf68de74861
The real pain point is that at work, I use Codex and I'm currently working on a project that involves debugging some polyline topology, very similar to the path following puzzle. The vision is completely useless here.
Your VLM idea sounds good. Theoretically, the inverse problem (generating an SVG of a pelican riding a bike) can also be solved with a VLM that plans out how to draw it, not unlike a human planning out a path for their hand to follow.
Dumb question: When your testing "ChatGPT 5.6 Sol" are you testing an actual LLM or some visual pre-processor stack that sits in front of it (along with a maybe a bunch of other such pre-processors) that is bundled into what's call "ChatGPT 5.6 Sol"? I.e. last I checked LLMs had a something like a 30-100K token alphabet to work with and it's hard to imagine how throwing pixels arrays at one directly would work.
I'm ~95% certain that images are tokenized, just like regular text, and fed directly in; that's the 'multimodal' part of these models. Now how this tokenizing works I don't know, and there might be some level of preprocessing, but it's certainly not converting the image into text and feeding it in to a regular LLM.
From our experiments it’s the best video captioning model in the world by a mile. This was not the case a year ago.
When reasoning got introduced a year ago to GPT 5, on average the model performed worse than GPT4-o for short video clip captioning (Ie hallucinating actions that didn’t happen). The old GPT 5 was extremely finicky in terms of fps sample rate.
The other SOTA LLMs (like Gemini Pro) have clearly been optimized for long video understanding, since they can’t see almost anything sub-second (even if you up the frame sampling rate).
Sol is the first model we’ve seen to accurately caption complex sub-second movements (eg woman suddenly turns heard head to right). It’s robust to different fps sample rates so I can only guess that they trained on videos sampled at different fps.
Have you checked versus more recent Gemini models like 3.5 or perhaps 3.7?
It's not clear to me from the article, are they asking sol to output bounding box coordinates with some kind of structured outputs?
Anecdotal but I've seen it use python to crop, zoom, and "enhance" (fiddle with sharpness and brightness) images to read sections of handwritten census data from the 1800s. Feels like that there might just be a mismatch of capabilities when it comes to straight outputting coordinates but I bet the model is better at actually finding the answer given any tools available. Which I get is a bit of an apples and oranges situation.
I've also tried to use it to identify an old pair of glasses and it didn't stand a chance, so I do think it's not quite there yet when it comes to some vision tasks.
I currently have fable organize a bunch of 5.6 sol agents when working on my personal projects. This makes me wonder if I should add something along the lines of "For tasks that involve visual analysis, have gemini 3.7 look at images generated."
Overall I've been hooked on using agents from different companies for what they are best at (Thanks to Theo). Fable is expensive, but unmatched for planning and top level organization of other agents. Sol is fast, will persistantly go after goals (sometimes to its detriment), and does well with computer use.
I actually favor Qwen3.8 and run it locally + use the Token-Plan on AlibabaCloud, when I need faster results. Kind of favor it over GPT5.6 Sol.
Also it seems to be more capable, need to test more, but I think it's at least getting on par and it's fully open-source and open-weights.
Here's some benchmarks:
https://benchlm.ai/compare/gpt-5-6-sol-vs-qwen3-8-max
https://qwen.ai/blog?id=qwen3.8#full-benchmark-table (incredible UI/UX demos)
https://venturebeat.com/technology/qwen3-8-max-arrives-with-...
EDIT: Am I early to the discussion, or is none else using Qwen3.8-max?
I thought Qwen 3.8 max doesn't have vision?
huh, why am I being shadow banned?
Does YC have similar problems like those at wikipedia/reddit? (wikipedia-editor-wars, or reddit-mod-wars)
You aren't being "shadow banned".
I concur with your conclusion, Qwen 3.8 has exceptional Vision Capabilities. The other commenter mentioned "I thought Qwen 3.8 didn't have Vision", it does, just not on the open weights version, only via official API.
I understand why you would like to use an LLM for vision. I do it myself often enough. I don't understand however, why the pill detection and counting is included in this benchmark. That is a task which you would perform with OpenCV right?
In my personal mini benchmark minicpm-v-4.6 scores amazingly well. Its a 0.8B model which runs fine on many consumer hardware.
Generating datasets to train more efficient models is a common use case for VLMs, especially frontier ones. It makes it much cheaper to create that initial dataset and you can abuse the nondeterminism of LLMs to identify data for human review (if they don’t converge, escalate to a human).
Especially the pill counting example. The best model was shown at 81.1% accuracy, which is a terrible rate for pharmacy scenarios. It seems like implementors would be better off instructing the models to use deterministic tools (like OpenCV) until the models are at 99.99% accuracy (or whatever an acceptable error rate is for pharmacy techs).
I think that is because people perceive OpenCV as 'hard to use' and LLMs as easy to use.
OpenCV is no longer hard to use, it just takes longer. Still, a little more complicated than asking LLM to count.
To use an LLM, you just prompt it with an image + text saying "count the pills in this image".
To use OpenCV, ... you just prompt an LLM with an image + text saying "count the pills in this image, using OpenCV instead of eyeballing it".
(I like to throw in "produce intermediary artifacts so I can see the process" for more difficult tasks; this helps the model avoiding making hallucination-prone leaps and gives more opportunities to self-correct. At a cost of extra time and tokens, of course.)
Using OpenCV without an LLM? Nah, not touching that, I don't have free weekends to waste anymore.
I no longer use it but never felt it was particularly complicated, but since the days of resnet there are much faster ways to the goal.
So far I haven't seen a single model succeeding at transcribing sheet music, but I just tested it again with 5.6 Sol and it nailed the small test case. Fluently reading music requires multiple years of training for most people, but I feel like accurately following the horizontal lines trips up vision models in particular.
For a bespoke model that transcribes sheet music images well, check out our system at Soundslice: https://www.soundslice.com/sheet-music-scanner/
It's not an LLM, it's a custom thing we built. Here's a comprehensive list of support for various notation glyphs: https://www.soundslice.com/help/en/creating/pdf-import/294/s...
My anecdotal evidence says its still as blind as any other model, it has no taste, no attention to any sort of detail.
How can a vision model have taste?
If you're doing any kind of inference that is multi-modal and non-factual, opinions and biases will affect any kind of assessment of a visual that you provide to a model.
For example, a UI / UX professional being asked to appraise a website screenshot may determine that the image in question has "desirable" traits which are inherently not deterministically measurable. Such as, if the interface elements have strong information hierarchy, or if they are deemed to be "fashionable" with current UI trends.
> if the interface elements have strong information hierarchy
...but that's an example of a UX/usability matter that can be assessed objectively and non-subjectively.
I disagree.
Is 16 px or 14 px a better font-size value for a subheading, in a hypothetical layout? Immediately that kind of decision, where both options are objectively good for 12 px paragraph text, suddenly becomes an issue of taste that cannot be evaluated crudely by an algorithm.
Replace taste with consistent if that helps you. Can it follow a design system...
As a design system engineer I usually have to fight against the taste of the designers. (And I consider it natural.)
But, if you have a proper well documented design system and you tell the LLM to use the DS and to avoid styling hacks they can generally do it. Even the dumber ones than Sol 5.6.
Of course only if the design is achievable in the design system.
This is not my experience at all.
So, formulaic output…the opposite of taste
Not really. Compliance with the letter of the law doesn't mean the intent is complied with.
One of my friends (and BIL) own an architecture firm. They use AI to generate and quickly update renderings but they run into the equivalent of the 6 fingered hand problem. I sent him this article I wonder if the updated models can catch and fix mistakes made by previous models.
This article is about vision, not image output.
Hence the "catch and fix mistakes" part.
I've decided it's "good enough" after I saw it properly quote a string of text that was very roughly highlighted within a nested visual context. It also identified the context correctly (modal inside webapp inside screenshot of user desktop).
I run the free service https://countrx.app/ so i have some idea what goes into counting.
The performance as a general model is indeed really impressive and i think they might actually win compared to fine tuned models.
Their feedback loop of training on user data is incredibly strong. I've learned that lots of accuracy results depends on threshold configs, which llms should be able to dynamically set.
Or the future will develop in llms using fine-tuned models as tools? Inference cost and speed does still seem to be below user expectations.
But for being able to one shot with this accuracy... IMPRESSIVE
How are you running it for free? Are you self funding or do you have sponsors?
Self funded. It's a custom trained efficient model on CPU so it's borderline free
I really like the combo 5.6 Luna & Sol for price and performance and would be perfectly happy if they stayed here for a moment without mucking about with sidegrades that I think AI evolution has often felt like lately.
I wonder what about UI review, which model is the best?
It’s gotten so good that I now have infrastructure to render all mermaid/plantuml in my project to png and have AI’s always load both text and image versions. And they are instructed to review the rendering as part of the diagramming cycle (for layout, salience, usefulness, etc). They can now produce useful diagrams that help reach shared architecture understanding.
All of your use cases are very advanced.
I recently used it at grocery stores in a foreign country. Photographed the whole aisle and told it to find Y (detergent, softener, glue, sour cream, whatever), at the same time recommend the best Y for whatever reason. Worked marvelously, including the cases where the object wasn't present and it told me there was nothing useful.
I asked then, can you crop the exact image of how does the item look like and where is it in the aisle - did that perfectly as well.
I will add that all frontier models were fine with such tasks from the early 2024's.
"Best iPhone ever" vibes.
I would love more vision benchmarks! Once I asked the model to inspect a completely black picture and it hallucinated a nice wooden kitchen wall. Took me some time to figure out where the kitchen came from...
I usually go to https://arena.ai/leaderboard/vision/pareto for a nice overview of current models.
5.6 Sol looks nice, but the Gemini 3.5 Flash comparison is interesting. It’s cheaper and still came out ahead on detection and counting, which doesn't really give me much of a reason to use Sol since Flash is much cheaper and hence much easier to scale. Not to mention we now have 3.6 Flash too
We have 3.7 Flash now, actually, and it costs just a hair over the old 3 Flash Preview while being better!
One of the use cases I've wondered about for AI is giving it a picture of the "spice wall" in a grocery store and asking it to find all jars of e.g. cardamom. This takes me an annoyingly long time to do when I'm shopping, so it would actually be useful.
It's vision capabilities poisoned my cucumber bed, misidentifying the malaise and having me spray them down with water, which only spread the fungus that gemini later informed me was actual cause, which I went and checked myself.
I hope that whatever was lost at GDM in the last few months, didn't include their extra focus on vision capabilities.
Luna is pretty strong as well. been using it for projects the last two weeks and its strong
Seed Turbo 2.1 is incredibly detailed in describing every physical feature. I use that one for vision tool calls through Venice API.
I didn't expect Gemini 3.5 Flash to top basically every metric in this article.
In my practice Gemini models are far better than anything on the market in terms of vision, also it's worth to mention that current Gemini flash is 3.7, so it got 2 updates since 3.5 which beat GPT-5.6 Sol in this comparison.
Hi! I’m the author of this blog. I wrote it 4 weeks ago, and it’s already a bit outdated. Gemini 3.7 Flash came out last week, and considering the price, it’s easily the best vision model right now: https://x.com/skalskip92/status/2088032652301304121?s=20
Same. I scrolled back up to see if I read the title correctly. It's important to note that it is the best... OpenAI released. Not the best overall.
Gemini has long been the vision champion, but there aren't many benchmarks and coding is where all the hype is.
Demis had a pretty big interest in vision, more so than text, so I hope they don't lose that with all the recent shuffling.
Do you think we’re getting closer to models that actually understand what they’re seeing, or are they just getting really good at recognizing patterns?
All I'm fine with for now is that I can almost exclusively communicate with Sol through collages and my scribblings (all kinds of web page / block screens with all kinds of arrows and text all over the place) This was not practically ppossible in 5.5 and a tragedy in 5.4. Not sure how much weight is codex uploading in higher res carrying here but it's great to work with.
Are any of these vision benchmarks binocular in order to introduce depth perception?
I keep waiting for these AI companies to assemble the parts into a great autonomous driving module.
Does any popular NVR make a good use of LLMs (especially local models) getting decent at vision?
I've been using Reolink for years and been very satisfied with it.
The only quip is the default UI isn't very good. When changing that reaches the top of my priority list, I'll switch it since they don't force you into a walled garden. Plan is to run it through frigate into HomeAssistant and use a UI from them. I've never used frigate before though so it'll be a learning process if plug and play solutions aren't already available
I agree. It did very well on an extremely challenging task.
I asked it to recognize and draw the very faint reflection of what I was wearing, visible in only a tiny black part of a very brightly lit poster behind glass.
In addition, the poster itself also happened to contain similar clothing.
You can see the reference images and its output in my writeup here: https://medium.com/@rviragh/gpt-5-6-sol-very-good-image-reco...
While a human can focus on the reflection easily, this is an enormous challenge for a vision model. It's very impressive.
For the last 2 weeks I've been trying to get Codex to "outpaint" a wonderful image it generated as placeholder art for a level background.
After I increased the game's resolution, I asked it to increase the image's size while keeping the same scale and existing content, and gosh, it constantly keeps getting something wrong no matter what I tell it, even on Sol Max with the $100 Pro subscription.
An organically-grown meat-based pixel-artist could have recreated the image and more within 2-3 days, in exchange for food and shelter.
I'm unsure why you're using an LLM to generate images. Don't we already have models (some made by the same company) that do this?
> it constantly keeps getting something wrong no matter what I tell it
This 100%
did you try segmenting it first?
At first I intended to create a tileset and asked it for several variations of what a hypothetical tilemap created from the planned tileset would look like.
The previews it generated were amazing but wouldn't really be possible as a grid-based tilemap, with lots of clusters and overlaps of elements of varying sizes.
So I just decided to use the preview as a static scrolling background, but it's been a pain to get it to add more content around the edges that still tiles with the existing image at the same scale.
I hate these "The best X thing Y has ever released".
Unlike when Apple says "it's the best iphone we've ever made", LLMs are more or less interchangeable. So "OpenAI's best model" means nothing if "Anthropic wipes the floor with them" or "[open weights model] is 10x cheaper for 1% less quality".
As a reader, it feels like these titles are click bait.
the last image - it's barely visible to human eyes
> GPT 5.6 Sol is the best "vision" model OpenAI ever released
I mean I should hope so, as it is also the latest one
It's really quite good! I was amazed recently by its utter inability to read some faded handwritten cyrillic on the back of a wood carving - 3 or 4 words only, reasonably clear letter forms I found recently, and then stepped back a bit and thought about how insane that was as a benchmark - I just expect it to work so reliably on other OCR and translation tasks that it was surprising to encounter such a failure
Where are the Qwen benchmarks in this? I would be more interesting to see how Qwen performs.
Hi! I’m the author of this blog. I regularly benchmark new VLM releases. You can check the results for Qwen3.8-Max and Qwen3.8-27B here: https://playground.roboflow.com/evals
Me too. This is an interesting comparison but in my experience Qwen and Gemini have typically been the top contenders for image related tasks. For that reason it would be great to have the comparison here, as I'm not surprised by Gemini's dominance over the other models.
Fuck ack. I'm working on a new benchmark that combines strong visual requirements with tool and coding requirements. I haven't even tested Sol yet, but between Sonnet, Terra & Luna I already see much better results from OpenAI's models. I'm not releasing anything yet as I still have issues in my harness that need to be fixed.
Am I the only one who cannot read the date on the blister pack even fully zoom in my phone?
If that is the full quality image given to the model, I think it's not surprising that the model confused with 03/2022.
I'm a little bit disappointed that vision seems to fall before language at scale.
It seems pretty counter intuitive that we can't do vision significantly better with specialized techniques.
Which is to say, still not ready for any production workloads yet. As in, it cannot reliably count the amount of objects in an image.
Still very impressive, but nowhere near the text chat revolution. OpenAI still trying to strike their second lightning
https://news.ycombinator.com/item?id=46444508
Still not quite as good as gemini.