> Open internet with cheating detection => cheating is widespread, 20-36% of tasks are flagged for the stronger models, with source code lookup accounting for the majority of the violations.
Therefore:
> blocking internet access entirely is the appropriate default for ProgramBench
The fact that your Anthropic coding assistant has a tendency to search on the Internet code to be inserted into your program may count for an additional copyright violation (besides the possibility of reproducing recognizable fragments of its training data).
(I do not agree that copyright, at least in its current form, should be applicable to computer programs, but it is weird that the same companies who try to exploit copyrights against others also insist on the use of coding assistants that are a workaround against copyright laws, which is the main reason why they can increase programming productivity, because they may cut and paste code that you are not allowed to copy yourself.)
Nice work once again from Ofir Press and team; this seems to be an idea that's in the air.
> Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task
Fwiw, this is very different from what we find in MirrorCode:
> Opus 4.6 successfully reimplements almost every program up to gotree’s size in our benchmark.
I don't have time right now to dig in to what could explain the difference (I'm working hard on getting the full MirrorCode out as soon as possible). But I suspect that the ProgramBench authors are either under-eliciting the AIs, or their tasks are unfair/impossible given the constraints, or both.
I hope to look more into it after releasing MirrorCode, and write up my conclusions.
Problem with these types of benchmarks is that it’s 100% certain the LLM has been trained on all that code already, so they’re all tainted since you don’t know whether it’s just benchmarking recall vs actual reasoning.
I would love to try this out. I have a horrible legacy project that is written in angular by a really amateur developer, full of huge blocks of copy pasted code that has minor modifications in each block. I’ve tried before to get an LLM to rewrite it to something more sensible, but I have not succeeded, usually it just ends up breaking everything.
Is there a guide or some system to follow? What’s the best way to accomplish a task like this?
Surely the biggest difference is that you guys are mostly testing LLMs on simpler utilities, mostly involving higher-level languages, whereas ProgramBench are all very complex C programs (and much older programs with much more comprehensive test cases).
Eg cal is totally routine. I would expect most sophomores to be able to write a perfectly good cal. In fact the only program you tested which actually has anywhere close to the complexity of SQLite or FFmpeg is is Pkl, and it looks like Opus 4.6 totally failed.
I think your results are consistent. You're just measuring different things. Your benchmarks mostly tests LLMs ability to write technically routine programs of moderate length - yes the bioinformatics package involves specialized domain knowledge, but not specialized Go engineering. ProgramBench is harder.
"Models favor monolithic, single-file implementations that diverge sharply from human-written code."
You say! I might have been just an LLM all along without even knowing it since I too prefer single file implementations.
Back in the old VB5/VB6 days Visual Studio had this mode where it showed the different functions in a file almost as if they were separate files. You could not scroll beyond the functions end but you could easily transition between that mode and global file view. I always found that a nice way of working (but admittedly the world was a lot simpler back then).
Also my preference for fewer but longer files is only there when I write the code myself. For working with AI I think smaller files are beneficial for quicker turn around between human and machine.
> Models favor monolithic, single-file implementations that diverge sharply from human-written code.
Well, all of our code is monolithic with some files close 20K lines of code and we do use coding agents - not for the original code but as of late. I've always had that hunch that splitting everything into tiny files does not improve AI coding agent performance although it feels counterintuitive due to model context constraints.
To me the important parts of a program should be clustered together so the implementation is obvious. Scattering the implementation in various files all over the source tree does not help much building the mental model.
That also closely match how software used to be written in the past too.
Kinda surprising to me, since i had some trouble with Cursor & Co. once the file went over ~800 lines. It repeatedly failed to edit it, until i split it up into multiple logical components. As it should have been from the beginning...
Though, it was some time ago, so things might have improved?
VSCode basically any model can edit the 20K file without any issues. The coding harness does not read the entire file at once though. It reads chunks of it so the size does not really matter. What matters is how close are the things the agent needs to make the edit.
Yeah, that was my experience with Grok, whenever I gave it a file with over 400 lines it would just fail to comprehend it or be too lazy to write too much at a time. Splitting stuff up into separate files helped.
> Scattering the implementation in various files all over the source tree does not help much building the mental model.
Yeah, that happens where I work and I hate it. A combination of lint rules and AI reviewer prompts complain about long files and long functions. This means something that could be a 300 line self contained function that could be read linearly, gets split up into 6 functions across 6 files.
It's the illusion of "clean code". If you're casually skimming the code, you feel good. But as soon as you go beyond the surface level it becomes annoying.
> Models favor monolithic, single-file implementations that diverge sharply from human-written code.
This isn't the case if models are prompted to actually plan the file architecture beforehand, it's only the case if they're given a dumb monolithic "code this thing" prompt.
It’s unfortunate that they didn’t eval using subagents/orchestration for such a complex set of tasks (from what I can tell), e.g. analyze program to produce initial spec -> code -> review and rinse&repeat with each of those steps being a separate subagent allocated
I would be interested to see if there’s a significant quantifiable difference.
This might actually be the whole value prop of this benchmark. Forget their initial scores, take open models (so we can be sure the base doesn't change), and test different combinations of harness + prompts + strategies + whatever memthing is popular today. See if the scores improve. Repeat.
It's interesting that Figure 4 shows that Sonnet and Opus have a very clear distinct curve from all other models, even from GPT 5.4. Anthropic superiority I guess.
It's funny, because that task is very diverse. Any LLM will use the codebase given as a template(At least in free-tier models)
My software as a contract of behaviors works like a program bench(I even cross tested buildouts)
Made an entire corpus layout for multi agent multi platform builds to be compared. Even went ahead and ran 50 contracts for an example. It honestly showed improvable areas, and distinct differences between model code.
{contract_name}/
└── submissions/
└── {date}_{os}_{agent}_{model}_{stack}/
├── {contract}.osc.md
├── osc.osc.md
└── results/
└── {contract}.snapshot.json
That's it, compare to the same contract, or find a new contract to use to compare. Lot's of signed/hash pinned files are all you need to reproduce software from nothing, with an LLM.
Programbench is close to that(they have a nice paper/article here. But I don't like the work used. Having software to start with is not a bench of making code but reverse engineering.
It was the same thing with OOP, TDD, agile development, C, C++, Rust, ORMs..
Whenever something impacts a ton of people you will get some who gain a lot from it and some who don't, and they're generally unable to relate to the other side.
Maybe the thing works in some domain and not the other. Maybe the two groups are doing different things. Maybe the context around it is different. Maybe they have a different definition of "better".
I think it helps to keep an open mind and not grow attached to either position, but rather inquire, "well we did X with outcome Y, what did you do instead?"
How long until AI is not even writing code but producing machine code?
Think about it, all these compilers, tooling, what a waste!
I imagine a future where chipset makers will provide a model you can just prompt to "act upon that chipset" and voila, "You're absolutely right! Here is your binary."
We won't be developers, we won't be devops, we'll be rollmops! /s
Coding agents can write ASM. But if you mean writing the actual byte-code that will require a very different approach at a very different level of abstraction that LLMs are not designed to do. Keep in mind that all LLMs are trained first on text and then fine-tuned on code.
My hunch is that it would take years of hundreds of thousands of developers working with machine code, posting stackoverflow questions with machine code, and publishing github repos written on it with documentation. Thats all the free labor LLMs leveraged to use high level langs.
>We won't be developers, we won't be devops, we'll be modelops! /s
I can still see this happening with higher level langs. the thing is the compiler is not replaced in the training data, more likely LLMs will give rise to semideterministic layers on the compilers
I could see nvidia achieving this first with how nice the devex is with CUDA
FWIW I think "LLMs are semideterministic" is something of a red herring. The real difference between LLM codegen and compilers is that compilers output logically the same assembly regardless of the variable names. If you're numerically solving a differential equation the compiler does not care if the floats represent heat through a pipe or dollars through a brokerage. Compilers don't care about semantic meaning, that concern is totally separated.
But even if its putatively implementing the same algorithm, LLMs certainly do not output basically the same finance Python as they would mechanical engineering Python. The style will be a little different. Sometimes the performance/clarity tradeoffs will be different. Sometimes it'll be fairly fancy and object-oriented, other times it'll be more low-level "objects are just dicts."
It's way more than a higher abstraction layer: LLM codegen involves a nontechnical tangling of concerns that doesn't exist with even the hoitiest-toitiest proof-checking compilers. It's a complete sea change. I find it incredibly disconcerting... for the same reason, by the way, that assembly programmers found Fortran and C disconcerting, and continued to reliably find employment for a good 40 years after higher-level languages were invented :) Actually even today. The assembly programmers who got hosed by C tended to be electricians who learned on the job - it's kind of cool to read old manuals from the 70s, carefully (and correctly!) explaining to electricians that a computer program is essentially an ephemeral circuit.
But I think there are specific skills around scientific thinking (learned at a formal college) and engineering carefulness (learned via hard knocks) that aren't going anywhere.
They are - probably more proficient than with some high-level languages. I've used it for embedded stuff, including TI sitara PRU assembly, with great results.
Frontier models can also easily "learn" directly from the manuals; asm is quite easy for them to pick up due to its "flat" (non-structured) nature.
> Open internet with cheating detection => cheating is widespread, 20-36% of tasks are flagged for the stronger models, with source code lookup accounting for the majority of the violations.
Therefore:
> blocking internet access entirely is the appropriate default for ProgramBench
The fact that your Anthropic coding assistant has a tendency to search on the Internet code to be inserted into your program may count for an additional copyright violation (besides the possibility of reproducing recognizable fragments of its training data).
(I do not agree that copyright, at least in its current form, should be applicable to computer programs, but it is weird that the same companies who try to exploit copyrights against others also insist on the use of coding assistants that are a workaround against copyright laws, which is the main reason why they can increase programming productivity, because they may cut and paste code that you are not allowed to copy yourself.)
Nice work once again from Ofir Press and team; this seems to be an idea that's in the air.
> Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task
Fwiw, this is very different from what we find in MirrorCode:
> Opus 4.6 successfully reimplements almost every program up to gotree’s size in our benchmark.
https://epoch.ai/blog/mirrorcode-preliminary-results
I don't have time right now to dig in to what could explain the difference (I'm working hard on getting the full MirrorCode out as soon as possible). But I suspect that the ProgramBench authors are either under-eliciting the AIs, or their tasks are unfair/impossible given the constraints, or both.
I hope to look more into it after releasing MirrorCode, and write up my conclusions.
Problem with these types of benchmarks is that it’s 100% certain the LLM has been trained on all that code already, so they’re all tainted since you don’t know whether it’s just benchmarking recall vs actual reasoning.
Same with SWE-bench and others.
I would love to try this out. I have a horrible legacy project that is written in angular by a really amateur developer, full of huge blocks of copy pasted code that has minor modifications in each block. I’ve tried before to get an LLM to rewrite it to something more sensible, but I have not succeeded, usually it just ends up breaking everything. Is there a guide or some system to follow? What’s the best way to accomplish a task like this?
Surely the biggest difference is that you guys are mostly testing LLMs on simpler utilities, mostly involving higher-level languages, whereas ProgramBench are all very complex C programs (and much older programs with much more comprehensive test cases).
Eg cal is totally routine. I would expect most sophomores to be able to write a perfectly good cal. In fact the only program you tested which actually has anywhere close to the complexity of SQLite or FFmpeg is is Pkl, and it looks like Opus 4.6 totally failed.
I think your results are consistent. You're just measuring different things. Your benchmarks mostly tests LLMs ability to write technically routine programs of moderate length - yes the bioinformatics package involves specialized domain knowledge, but not specialized Go engineering. ProgramBench is harder.
"Models favor monolithic, single-file implementations that diverge sharply from human-written code."
You say! I might have been just an LLM all along without even knowing it since I too prefer single file implementations.
Back in the old VB5/VB6 days Visual Studio had this mode where it showed the different functions in a file almost as if they were separate files. You could not scroll beyond the functions end but you could easily transition between that mode and global file view. I always found that a nice way of working (but admittedly the world was a lot simpler back then).
Also my preference for fewer but longer files is only there when I write the code myself. For working with AI I think smaller files are beneficial for quicker turn around between human and machine.
I am not surprised but this one sticks out...
> Models favor monolithic, single-file implementations that diverge sharply from human-written code.
Well, all of our code is monolithic with some files close 20K lines of code and we do use coding agents - not for the original code but as of late. I've always had that hunch that splitting everything into tiny files does not improve AI coding agent performance although it feels counterintuitive due to model context constraints.
To me the important parts of a program should be clustered together so the implementation is obvious. Scattering the implementation in various files all over the source tree does not help much building the mental model.
That also closely match how software used to be written in the past too.
> Scattering the implementation in various files all over the source tree
If you treat the source tree seriously, you can communicate a lot with how it is structured
Well you can communicate organisation structure but not logic or intent. The directory is a tree and the Code is a graph.
You can communicate some information by looking at the org chart of a company but it does not really tell you much how it works.
Arguably a coding agent is less concerned about where the files are at then the code itself.
Kinda surprising to me, since i had some trouble with Cursor & Co. once the file went over ~800 lines. It repeatedly failed to edit it, until i split it up into multiple logical components. As it should have been from the beginning...
Though, it was some time ago, so things might have improved?
VSCode basically any model can edit the 20K file without any issues. The coding harness does not read the entire file at once though. It reads chunks of it so the size does not really matter. What matters is how close are the things the agent needs to make the edit.
Yeah, that was my experience with Grok, whenever I gave it a file with over 400 lines it would just fail to comprehend it or be too lazy to write too much at a time. Splitting stuff up into separate files helped.
this is a big frustration for web code what with HTML, CSS, JS, PHP all spread about
https://htmx.org/essays/locality-of-behaviour/ is a good fight back as exemplified in many stacks, eg https://harcstack.org
> Scattering the implementation in various files all over the source tree does not help much building the mental model.
Yeah, that happens where I work and I hate it. A combination of lint rules and AI reviewer prompts complain about long files and long functions. This means something that could be a 300 line self contained function that could be read linearly, gets split up into 6 functions across 6 files.
It's the illusion of "clean code". If you're casually skimming the code, you feel good. But as soon as you go beyond the surface level it becomes annoying.
> Models favor monolithic, single-file implementations that diverge sharply from human-written code.
This isn't the case if models are prompted to actually plan the file architecture beforehand, it's only the case if they're given a dumb monolithic "code this thing" prompt.
It’s unfortunate that they didn’t eval using subagents/orchestration for such a complex set of tasks (from what I can tell), e.g. analyze program to produce initial spec -> code -> review and rinse&repeat with each of those steps being a separate subagent allocated
I would be interested to see if there’s a significant quantifiable difference.
This might actually be the whole value prop of this benchmark. Forget their initial scores, take open models (so we can be sure the base doesn't change), and test different combinations of harness + prompts + strategies + whatever memthing is popular today. See if the scores improve. Repeat.
It's interesting that Figure 4 shows that Sonnet and Opus have a very clear distinct curve from all other models, even from GPT 5.4. Anthropic superiority I guess.
It's funny, because that task is very diverse. Any LLM will use the codebase given as a template(At least in free-tier models)
My software as a contract of behaviors works like a program bench(I even cross tested buildouts) Made an entire corpus layout for multi agent multi platform builds to be compared. Even went ahead and ran 50 contracts for an example. It honestly showed improvable areas, and distinct differences between model code.
{contract_name}/ └── submissions/ └── {date}_{os}_{agent}_{model}_{stack}/ ├── {contract}.osc.md ├── osc.osc.md └── results/ └── {contract}.snapshot.json That's it, compare to the same contract, or find a new contract to use to compare. Lot's of signed/hash pinned files are all you need to reproduce software from nothing, with an LLM.
Programbench is close to that(they have a nice paper/article here. But I don't like the work used. Having software to start with is not a bench of making code but reverse engineering.
github/s1ugh34d/osc
In before "but they did not use my agent swarm"
It’s the annoying thing about AI. If it works, the AI is magic. If it doesn’t work, you’re using it wrong.
It was the same thing with OOP, TDD, agile development, C, C++, Rust, ORMs..
Whenever something impacts a ton of people you will get some who gain a lot from it and some who don't, and they're generally unable to relate to the other side.
Maybe the thing works in some domain and not the other. Maybe the two groups are doing different things. Maybe the context around it is different. Maybe they have a different definition of "better".
I think it helps to keep an open mind and not grow attached to either position, but rather inquire, "well we did X with outcome Y, what did you do instead?"
So, would you change your view if someone else runs this bench w/ a different harness and gets better results?
In science N=1 is statistically insignificant. In business it might mean that you have a product.
RE: monolithic, single-file implementations
We have a lint that caps source code files at 650 LOC and it works really well.
How long until AI is not even writing code but producing machine code?
Think about it, all these compilers, tooling, what a waste!
I imagine a future where chipset makers will provide a model you can just prompt to "act upon that chipset" and voila, "You're absolutely right! Here is your binary."
We won't be developers, we won't be devops, we'll be rollmops! /s
Coding agents can write ASM. But if you mean writing the actual byte-code that will require a very different approach at a very different level of abstraction that LLMs are not designed to do. Keep in mind that all LLMs are trained first on text and then fine-tuned on code.
Good point! Long live ASM! Wasm everything!!1 /jk
Good luck reasoning about the output in any meaningful way then. AI introduces a bug? Well, you're fucked.
Welcome to the future!
My hunch is that it would take years of hundreds of thousands of developers working with machine code, posting stackoverflow questions with machine code, and publishing github repos written on it with documentation. Thats all the free labor LLMs leveraged to use high level langs.
>We won't be developers, we won't be devops, we'll be modelops! /s
I can still see this happening with higher level langs. the thing is the compiler is not replaced in the training data, more likely LLMs will give rise to semideterministic layers on the compilers
I could see nvidia achieving this first with how nice the devex is with CUDA
FWIW I think "LLMs are semideterministic" is something of a red herring. The real difference between LLM codegen and compilers is that compilers output logically the same assembly regardless of the variable names. If you're numerically solving a differential equation the compiler does not care if the floats represent heat through a pipe or dollars through a brokerage. Compilers don't care about semantic meaning, that concern is totally separated.
But even if its putatively implementing the same algorithm, LLMs certainly do not output basically the same finance Python as they would mechanical engineering Python. The style will be a little different. Sometimes the performance/clarity tradeoffs will be different. Sometimes it'll be fairly fancy and object-oriented, other times it'll be more low-level "objects are just dicts."
It's way more than a higher abstraction layer: LLM codegen involves a nontechnical tangling of concerns that doesn't exist with even the hoitiest-toitiest proof-checking compilers. It's a complete sea change. I find it incredibly disconcerting... for the same reason, by the way, that assembly programmers found Fortran and C disconcerting, and continued to reliably find employment for a good 40 years after higher-level languages were invented :) Actually even today. The assembly programmers who got hosed by C tended to be electricians who learned on the job - it's kind of cool to read old manuals from the 70s, carefully (and correctly!) explaining to electricians that a computer program is essentially an ephemeral circuit.
But I think there are specific skills around scientific thinking (learned at a formal college) and engineering carefulness (learned via hard knocks) that aren't going anywhere.
I heard they are already proficient at assembly languages.
They are - probably more proficient than with some high-level languages. I've used it for embedded stuff, including TI sitara PRU assembly, with great results. Frontier models can also easily "learn" directly from the manuals; asm is quite easy for them to pick up due to its "flat" (non-structured) nature.
>Frontier models can also easily "learn" directly from the manuals;
Really? So you just include the manual in the context? Or how does that work?