> JSON retries generated 7x to 14x more output tokens than the args path.
But!
> On PowerShell, the cost gap between args and JSON was 9x. On Bash, it collapsed to 1.5x.
The conclusion of the post is "cli args are better than JSON", but the argument for why only holds up on Windows. This post says more about Powershell than anything else.
Some big findings:
> JSON retries generated 7x to 14x more output tokens than the args path.
But!
> On PowerShell, the cost gap between args and JSON was 9x. On Bash, it collapsed to 1.5x.
The conclusion of the post is "cli args are better than JSON", but the argument for why only holds up on Windows. This post says more about Powershell than anything else.
Yes it's a shame they didn't show examples of how string quoting was causing retries in Powershell vs Bash.
The title of the article is misleading and is the wrong takeaway from the experiment.
So the largest deciding factor for a model is always going to be what it's trained on.
Even if it knows how to string escape, it's not going to do it successfully.
You see this very clear when you tell it to try and execute commands over ssh.
We need protocols that pipe data directly to the shells it's using rather than trust everything through bash tunnels.
ultimately, something like python or yaml will win because they are defacto token efficient and well known to the LLMs.