We built six agent benchmarks (recommendations, tool routing, text-to-SQL, rag, extraction, multi-turn error recovery), held each one fixed, and swapped only the model: Haiku 4.5 → 5.5 and GPT-5.4 nano → GPT-6 Luna.
What surprised us: the lab's own successors lowered the score in 3 of 12 upgrades, and the other lab's new model matched or beat it in 10 of 12. Prompts and tool descriptions also benefited from significant tuning with these upgrades.
Would love to know if folks have similar experiences and happy to answer any questions.
We built six agent benchmarks (recommendations, tool routing, text-to-SQL, rag, extraction, multi-turn error recovery), held each one fixed, and swapped only the model: Haiku 4.5 → 5.5 and GPT-5.4 nano → GPT-6 Luna.
What surprised us: the lab's own successors lowered the score in 3 of 12 upgrades, and the other lab's new model matched or beat it in 10 of 12. Prompts and tool descriptions also benefited from significant tuning with these upgrades.
Would love to know if folks have similar experiences and happy to answer any questions.