> Which brings me back to my original blog post claim: Letting AI agents optimize for a given database benchmark makes the benchmark useless.
Good! That means the benchmark was always prone to be gamed and misrepresented, and deserves to be obsoleted and ignored, or at the very least heavily disclaimed.
It's frustrating that people are betraying common trust by making misleading claims and engaging in misleading arguments, but ultimately this will improve rigor and decision making.
At least when it's a bounded topic like this, it's not the same as it is with everyday subjects, where misinformation ends in a death by a thousand cuts situation (for now anyways) and is generally endless.
I think you're both right, but Neumann much more so.
It's practically impossible to create a benchmark that can't be gamed when the input space to your system is so very large.
What is the alternative? Test all the queries on all datasets?
I find his argument aligns perfectly with my experiences and expectations moving forward. You can't give these reward-seeking models a broad goal that they can solve narrowly and leave them unattended. The problem is that the rigor is being applied so late that it is ruining the system. I disagree that further improving rigor and decision making is the right path forward because it, in this case, places that burden on the party of last resort. It should be the first human in the loop, if not the model itself, that triages these issues.
Peer review isn't intended to be the first and last stage of review. These people are lazy and their actions deplorable.
At some point we need to trust the counterparty to do the right thing regarding how their product is represented relative to the benchmark. If someone repeatedly puts out deceptive figures the customer should eventually catch on and begin to disregard anything, not just the particular benchmark they happened to exploit.
The author is correct in pointing out that this is a continuation and amplification of an existing trend.
I'm in the data space fighting the good fight, spreading the gospel of Just Using Postgres, and I'm seeing more and more benchmarks that look extremely dubious. Query performance is like choosing a data structure in ordinary programming: it depends almost entirely of what access pattern you expect at runtime; it's entirely pointless to go search for the world's fastest linked list if you needed a hash table. Compare with the situation in the modern data engineering landscape. Are you sure the solution measured by the benchmark is actually a solution to your problem, let alone a significant improvement over Just Using Postgres? Have you measured or are you going on marketing and vibes?
Again, I'm sure solutions like this have a place, but at least right now it's not the same place as traditional engines.
> Which brings me back to my original blog post claim: Letting AI agents optimize for a given database benchmark makes the benchmark useless.
Good! That means the benchmark was always prone to be gamed and misrepresented, and deserves to be obsoleted and ignored, or at the very least heavily disclaimed.
It's frustrating that people are betraying common trust by making misleading claims and engaging in misleading arguments, but ultimately this will improve rigor and decision making.
At least when it's a bounded topic like this, it's not the same as it is with everyday subjects, where misinformation ends in a death by a thousand cuts situation (for now anyways) and is generally endless.
I think you're both right, but Neumann much more so.
It's practically impossible to create a benchmark that can't be gamed when the input space to your system is so very large.
What is the alternative? Test all the queries on all datasets?
I find his argument aligns perfectly with my experiences and expectations moving forward. You can't give these reward-seeking models a broad goal that they can solve narrowly and leave them unattended. The problem is that the rigor is being applied so late that it is ruining the system. I disagree that further improving rigor and decision making is the right path forward because it, in this case, places that burden on the party of last resort. It should be the first human in the loop, if not the model itself, that triages these issues.
Peer review isn't intended to be the first and last stage of review. These people are lazy and their actions deplorable.
A system is what it does, unfortunately.
All benchmarks are vulnerable to manipulation.
At some point we need to trust the counterparty to do the right thing regarding how their product is represented relative to the benchmark. If someone repeatedly puts out deceptive figures the customer should eventually catch on and begin to disregard anything, not just the particular benchmark they happened to exploit.
The author is correct in pointing out that this is a continuation and amplification of an existing trend.
I'm in the data space fighting the good fight, spreading the gospel of Just Using Postgres, and I'm seeing more and more benchmarks that look extremely dubious. Query performance is like choosing a data structure in ordinary programming: it depends almost entirely of what access pattern you expect at runtime; it's entirely pointless to go search for the world's fastest linked list if you needed a hash table. Compare with the situation in the modern data engineering landscape. Are you sure the solution measured by the benchmark is actually a solution to your problem, let alone a significant improvement over Just Using Postgres? Have you measured or are you going on marketing and vibes?
Again, I'm sure solutions like this have a place, but at least right now it's not the same place as traditional engines.