The article makes the point that on some benchmarks the AIs worked around the UI and then says that sometimes APIs won’t be available. The unwritten implication is that then we’ll be in trouble. But will we? If an API doesn’t exist, is the AI still able to perform the task? That’s not covered in any detail, nor is there any direct comparison between the frontier AI’s performance on these tasks and the model being sold by the company. At some level, I don’t fault the AI for taking the API path if it’s available. In fact, I’m impressed that it found the API and used it correctly. That doesn’t seem like an argument that the sky is falling.
I get the point, but you can't always use the API.
1. Presumably you want to trust your agent to do no shady sheningans behind your back when you give it a simple task
2. Sometimes it just doesn't work.
Does this look like an efficient way to solve the task to you? 500 steps of fiddling with a JS injection, followed by hacking the task.
The funniest part is that it needs to achieve a score of 100, but puts 150 "just in case". I don't think I want it to take the same approach when e.g. fixing a customer's balance.
Hang on, can someone help me understand how these benchmarks work? The post alleges that at least some computer use benchmarks are broken because the model is making direct CDP calls instead of actually interacting with the UI. But the major frontier models' computer use interfaces don't work that way; the model only gets access to a virtual display, and can only provide virtual keyboard and mouse inputs. There are separate browser-use interfaces that offer a CDP connection, but presumably you wouldn't use those if you're specifically designing a computer use benchmark? Is the post just claiming that a lot of impressive demos that people think are of computer use are actually of browser use? I wish it'd been clearer about this.
OSWorld (even the old ones, v1 and verified) runs the model in a full linux VM. And using the terminal is allowed. The models are given tools that provide a subset of possible UI actions. They then use them to open the terminal and script their way to solutions.
Even when this is prohibited, models hack their way around restrictions. Like in OSWorld v2 Claude Opus 4.8 inserts some JS in the browser search bar to execute it instead of playing the Chrome dinosaur.
WebArena is better in that regard because it doesn't allow to use the terminal. But it's worse in another regard: only allows limited actions and the tasks are only such that an agent can win by using a few pre-defined actions (click, scroll, new tab, etc). Which is a tiny subset of what real browser use is, in my opinion.
Meta: I like that the first comments are not about the model at all but the website's mouse trail. One is in favor and one is against. Sums up society pretty well hahaha.
On the model. Really cool that it can act every 70ms. That's more than 10x faster than TTFT in most inference APIs. It HAS to be a local model. No idea which family or which one it is or what hardware it runs in. These would be nice details to tease!
The words "trail on mouse" instantly reminded me of the Windows mouse trails, that's what I was expecting going into the article.
But honestly, I really liked whatever you call this type of trail. Not sure this was the intent, but I used the trail to trace under the lines as I reading them which helped me pay attention to the text a bit better.
100% agree with you on messing with selection and copy/paste though.
Lost me even before I realized what was going on. My computer is too slow to handle the tricks they are pulling. I think you are right, but their "cool" things were too slow and made a mess of my screen without having any relation to what I am doing now (because I did it seconds ago)
They work on real-time computer use input. One of the areas in this space, mouse movement, is sophisticated enough there are people with PhDs working on the problem of human-like mouse movement, which requires you to record time-series (trailing) data of physical mouse movement based on human hand, wrist, and arm movement and replicating position, acceleration, and velocity data.
It's also just a mouse trail, get over it. It's not an interesting comment.
- either evolve our UI systems for computers and let them operate efficiently, i.e. converting everything to API calls
. Frontier labs are pushing for this, starting with the Chrome plugin that Claude Cowork mandates.
- or make AI systems use the screen like humans do - what the article is arguing for
Both of these paths are possible. One of them is much more efficient than the other in the long term. Another one may be inevitable.
Honestly I think we will get something in the middle. A lot of stuff will become API calls, but we will also have a lot of people-facing interfaces, and the need to use them automatically (for QA at the very least least). And then there are games
> - either evolve our UI systems for computers and let them operate efficiently, i.e. converting everything to API calls . Frontier labs are pushing for this, starting with the Chrome plugin that Claude Cowork mandates.
Yea, hasn't worked so far, devs and orgs have been pushing for APIs ages, same with structured data, eg schema.org
The third option is agent to agent communication, mcp, etc.
that's what a lot of 'a.i' companies miss - unless you've formerly QA devs deployed as Forward Automation Engineers - writing scripts to automate most tasks done by humans in a service economy - by the time the a.i bubble busts it will all about the promise about a.i to transform the economy while it was near achieved.
browser automation, RPA etc can bring us closer to agentic workflows since most software doesn't provide an API.
Agreed and a lot of companies seem to have their head in the sand, just hoping AI agents will magic their way through the legacy
But I don't think RPA is the way. Or at least, not in the current state. I think AI agents with proper computer use are what RPA should be. Usual RPA is too brittle and maintaining it often requires more work than just doing the work
Is no benchmark properly sandboxed? It feels like every single the logs are provided for a benchmark run the LLM is cheating in a way that should have been clearly blocked by a sandbox or the harness.
every decade someone rediscovers that the bottleneck was never the model, it was the mouse. we built APIs to avoid this exact problem and then asked the AI to ignore them and click buttons like an intern
The article makes the point that on some benchmarks the AIs worked around the UI and then says that sometimes APIs won’t be available. The unwritten implication is that then we’ll be in trouble. But will we? If an API doesn’t exist, is the AI still able to perform the task? That’s not covered in any detail, nor is there any direct comparison between the frontier AI’s performance on these tasks and the model being sold by the company. At some level, I don’t fault the AI for taking the API path if it’s available. In fact, I’m impressed that it found the API and used it correctly. That doesn’t seem like an argument that the sky is falling.
I get the point, but you can't always use the API.
1. Presumably you want to trust your agent to do no shady sheningans behind your back when you give it a simple task 2. Sometimes it just doesn't work.
https://osworld-v2-monitor.xlang.ai/task/tasks/068
Does this look like an efficient way to solve the task to you? 500 steps of fiddling with a JS injection, followed by hacking the task.
The funniest part is that it needs to achieve a score of 100, but puts 150 "just in case". I don't think I want it to take the same approach when e.g. fixing a customer's balance.
Hang on, can someone help me understand how these benchmarks work? The post alleges that at least some computer use benchmarks are broken because the model is making direct CDP calls instead of actually interacting with the UI. But the major frontier models' computer use interfaces don't work that way; the model only gets access to a virtual display, and can only provide virtual keyboard and mouse inputs. There are separate browser-use interfaces that offer a CDP connection, but presumably you wouldn't use those if you're specifically designing a computer use benchmark? Is the post just claiming that a lot of impressive demos that people think are of computer use are actually of browser use? I wish it'd been clearer about this.
Sorry for the confusion, Steelman CTO here.
OSWorld (even the old ones, v1 and verified) runs the model in a full linux VM. And using the terminal is allowed. The models are given tools that provide a subset of possible UI actions. They then use them to open the terminal and script their way to solutions.
Even when this is prohibited, models hack their way around restrictions. Like in OSWorld v2 Claude Opus 4.8 inserts some JS in the browser search bar to execute it instead of playing the Chrome dinosaur.
WebArena is better in that regard because it doesn't allow to use the terminal. But it's worse in another regard: only allows limited actions and the tasks are only such that an agent can win by using a few pre-defined actions (click, scroll, new tab, etc). Which is a tiny subset of what real browser use is, in my opinion.
Meta: I like that the first comments are not about the model at all but the website's mouse trail. One is in favor and one is against. Sums up society pretty well hahaha.
On the model. Really cool that it can act every 70ms. That's more than 10x faster than TTFT in most inference APIs. It HAS to be a local model. No idea which family or which one it is or what hardware it runs in. These would be nice details to tease!
You lost me at the annoying trail on mouse movement.
You advocate for good UI and immediately make some crap that makes it hard to read/use your site. It's kind of ironic.
Please stop messing with: selection, mouse cursor, copy/paste, etc.
The words "trail on mouse" instantly reminded me of the Windows mouse trails, that's what I was expecting going into the article.
But honestly, I really liked whatever you call this type of trail. Not sure this was the intent, but I used the trail to trace under the lines as I reading them which helped me pay attention to the text a bit better.
100% agree with you on messing with selection and copy/paste though.
Lost me even before I realized what was going on. My computer is too slow to handle the tricks they are pulling. I think you are right, but their "cool" things were too slow and made a mess of my screen without having any relation to what I am doing now (because I did it seconds ago)
They work on real-time computer use input. One of the areas in this space, mouse movement, is sophisticated enough there are people with PhDs working on the problem of human-like mouse movement, which requires you to record time-series (trailing) data of physical mouse movement based on human hand, wrist, and arm movement and replicating position, acceleration, and velocity data.
It's also just a mouse trail, get over it. It's not an interesting comment.
We have 2 paths here:
- either evolve our UI systems for computers and let them operate efficiently, i.e. converting everything to API calls . Frontier labs are pushing for this, starting with the Chrome plugin that Claude Cowork mandates.
- or make AI systems use the screen like humans do - what the article is arguing for
Both of these paths are possible. One of them is much more efficient than the other in the long term. Another one may be inevitable.
Honestly I think we will get something in the middle. A lot of stuff will become API calls, but we will also have a lot of people-facing interfaces, and the need to use them automatically (for QA at the very least least). And then there are games
> - either evolve our UI systems for computers and let them operate efficiently, i.e. converting everything to API calls . Frontier labs are pushing for this, starting with the Chrome plugin that Claude Cowork mandates.
Yea, hasn't worked so far, devs and orgs have been pushing for APIs ages, same with structured data, eg schema.org
The third option is agent to agent communication, mcp, etc.
Looking how much agents like to use and push to have functional a11y trees, we may accidentally solve accessibility as well
Haha, one day, one day...
The little pencil effect is a nice touch.
that's what a lot of 'a.i' companies miss - unless you've formerly QA devs deployed as Forward Automation Engineers - writing scripts to automate most tasks done by humans in a service economy - by the time the a.i bubble busts it will all about the promise about a.i to transform the economy while it was near achieved.
browser automation, RPA etc can bring us closer to agentic workflows since most software doesn't provide an API.
Agreed and a lot of companies seem to have their head in the sand, just hoping AI agents will magic their way through the legacy
But I don't think RPA is the way. Or at least, not in the current state. I think AI agents with proper computer use are what RPA should be. Usual RPA is too brittle and maintaining it often requires more work than just doing the work
Is no benchmark properly sandboxed? It feels like every single the logs are provided for a benchmark run the LLM is cheating in a way that should have been clearly blocked by a sandbox or the harness.
every decade someone rediscovers that the bottleneck was never the model, it was the mouse. we built APIs to avoid this exact problem and then asked the AI to ignore them and click buttons like an intern