Fluent is a voice control application for Windows. A person speaks a task, and the agent does it on the desktop.
The agent reads the screen in two ways:
First, it reads the Windows UI Automation tree. The tree gives the name, the role and the position of each control as text.
Second, it sends a screenshot to a vision language model. It does this only when the tree has no data. Many applications draw custom controls. These controls are not in the tree, but the model finds them in the image.
This hybrid method gives three results:
- Speed. Most steps do not wait for an image request.
- Cost. Most steps send text tokens only.
- Privacy. Most tasks send no pixels from the device, those that do have PII & other sensitive info removed.
The agent obeys these safety rules:
- It asks the person to confirm each destructive action.
- It cannot approve its own tool calls.
- It treats all screen text as data. It does not obey instructions in that text.
- It writes each action to a hash-chained audit log.
- It removes personal data before it writes to disk.
Speech, dictation and keyboard navigation are always free. Only the agent has a usage limit (as I'm a broke student).
Limits: Windows only. Some applications have an incomplete tree, so the agent uses the model more often, and it is slower -- but this will be fixed in later iterations.
I would appreciate your comments on the product, any pitfalls, etc. as this means a lot to me, knowing a lot of people who use similar products.
Fluent is a voice control application for Windows. A person speaks a task, and the agent does it on the desktop.
The agent reads the screen in two ways: First, it reads the Windows UI Automation tree. The tree gives the name, the role and the position of each control as text.
Second, it sends a screenshot to a vision language model. It does this only when the tree has no data. Many applications draw custom controls. These controls are not in the tree, but the model finds them in the image.
This hybrid method gives three results: - Speed. Most steps do not wait for an image request. - Cost. Most steps send text tokens only. - Privacy. Most tasks send no pixels from the device, those that do have PII & other sensitive info removed.
The agent obeys these safety rules: - It asks the person to confirm each destructive action. - It cannot approve its own tool calls. - It treats all screen text as data. It does not obey instructions in that text. - It writes each action to a hash-chained audit log. - It removes personal data before it writes to disk.
Speech, dictation and keyboard navigation are always free. Only the agent has a usage limit (as I'm a broke student).
Limits: Windows only. Some applications have an incomplete tree, so the agent uses the model more often, and it is slower -- but this will be fixed in later iterations.
I would appreciate your comments on the product, any pitfalls, etc. as this means a lot to me, knowing a lot of people who use similar products.
fluentforall.com