I don't know why you would run an agent loop without TDD? Should you write code without test coverage? So something must run the tests to be sufficient anyway?
> TLDR; Based on Opus's judgment of the quality of the outcomes, there was no clearly discernable difference based on TDD workflow versus no TDD workflow. On the contrary, more than once Opus ranked the non-TDD workflow solutions slightly higher in design and test quality. There was also no meaningful difference in mutation scores across the solutions.
That's really surprising.
Was there a difference in cost?
Does it matter whether you ask the agent to write the tests first or last? Is one way TDD and the other not?
Without property testing or fuzzing or formal methods, could the code metrics in this experiment have been sufficient?
I don't know why you would run an agent loop without TDD? Should you write code without test coverage? So something must run the tests to be sufficient anyway?
You can have the agent write them after. The article is specifically about TDD, meaning writing a small test that will fail, making it pass, (refactoring), and repeating until the feature is finished.
Was there a difference in cost?
The author claims TDD used 3x tokens, but points out it might not be 3x the cost due to caching. It's near the end of the article.
A strict tests-must-be-written-ahead-of-time TDD advocate might say that writing the tests last is an error like p-hacking (choosing the threshold of statistical significance after the experiment) is an error.
Having always suspected that test-first TDD was most correct like adhering to the Scientific Method is most correct, I suspect that there are unmeasured differences between the results. But perhaps test-first TDD is a sham in AI and IRL
Using Opus to judge the outcome of "good code" isnt exactly the sign of a great experiment.
I don't know why you would run an agent loop without TDD? Should you write code without test coverage? So something must run the tests to be sufficient anyway?
> TLDR; Based on Opus's judgment of the quality of the outcomes, there was no clearly discernable difference based on TDD workflow versus no TDD workflow. On the contrary, more than once Opus ranked the non-TDD workflow solutions slightly higher in design and test quality. There was also no meaningful difference in mutation scores across the solutions.
That's really surprising. Was there a difference in cost?
Does it matter whether you ask the agent to write the tests first or last? Is one way TDD and the other not?
Without property testing or fuzzing or formal methods, could the code metrics in this experiment have been sufficient?
Perhaps there's a TDD gauntlet loop that's as justified as tests for QC; https://news.ycombinator.com/item?id=49261545
A strict tests-must-be-written-ahead-of-time TDD advocate might say that writing the tests last is an error like p-hacking (choosing the threshold of statistical significance after the experiment) is an error.
Having always suspected that test-first TDD was most correct like adhering to the Scientific Method is most correct, I suspect that there are unmeasured differences between the results. But perhaps test-first TDD is a sham in AI and IRL