"Please don't pick the most provocative thing in an article or post to complain about in the thread. Find something interesting to respond to instead."
I've taken the provocative bit out of the title above, but please remember that we want reflective comments, not reflexive ones, in HN threads.
Really glad to see folks looking into quantitative approaches to give agents feedback on code quality. This post looks like a good start!
My main feedback for the authors would be, the most important problems for sloppiness are global properties, not local ones. In my experience an agent, like a human, has finite capacity for its attention, but if it runs into local sloppiness that gets in its way, it can fix it on a by-need basis. The technical debt issues that matter are usually global issues that aren't so easy to fix: they require global analysis and global refactoring.
I don't know the answer, but I think we're going to need ways to measure architectural properties, like separation of concerns, clear architectural layering, well-defined interfaces, etc.
Coding is not just the program running in memory, its also the process of distributing the mental model of understanding among the team.
If humans increasingly are kept out of coding, then who holds the mental model?
If AI holds the mental model, by definition human prompts will be over lossy channel. This is true without AI too. Software quality is directly dependent on good devs that translate from business/PM speak to technical decisions.
So is coding solved now? it was already solved decades ago.
Maybe I’m not even waiting for a model which is 100% at code, but one which is almost 100% at interactively exposing its (almost? correct) mental model.
If I put it like that, it seems to me we are much far away.
We'll invent other tools to help with the mental models, ones that are more intuitive.
Imagine being able to walk into a house and see immediately the leaks dripping from the ceiling, the clunky layout and smell the stinky garbage that hasn't been collected.
We're already in a sort of IDE when you think about it, we're operating a much larger mental model than any code-base, and we rarely remember that we're operating with such a model.
> Imagine being able to walk into a house and see immediately the leaks dripping from the ceiling, the clunky layout and smell the stinky garbage that hasn't been collected.
No need to imagine. Just open the README.md of many projects.
Mental models existed before coding. The pyramids were not built by just a couple of guys hauling some rocks together. It was a highly structured organization.
Or the East India Company or many other great organizations that achieved big things before computers.
Imho a better way of thinking about it is in terms of views - you can have an unimaginably complex codebase that nobody reads in full, but either agents or humans receive views/perspectives on that codebase.
Before LLMs we couldn’t have built tools that would deliver such views, now we can.
I mean... Nobody NEEDs to hold a mental model, but we've been increasingly trying to build that mental model into an agentic codebase... maybe we're doing it wrong but it is NOT going well.
The agents jot down absolutely everything (and more, which is the issue), thousands and thousands files of perceived business logic, decisions, workflows, architecture, data modeling, etc... It gets stale extremely quickly - we change one thing in the system and 570 "shards" need to be updated. I stumble on more and more things every day that are untrue about the agents "mental model" and it appears to just keep doing it. One small addition not properly reviewed by a human introduces a cascade of misconceptions system wide.
We've tried these "views" you mention, we call them subsections, but it runs into the same problems. One falsification ends up poisoning entire projects.
Remember when we use to refactor functions in a codebase in order to slim down complexity? Yeah - we ended up doing that but with documentation... IMO code is deterministic and we have trained professionals who know how to read it quickly and modify it. (We don't even need to write it anymore, but reading it is the simplest way to ensure no-loss understanding of a piece of functionality.)
Oh, I just dropped all the comments. Memory and session histories are for storing these things, comments just add confusion.
Subsections - how do they run into the same problem? What falsification? The views I mean are dynamic ad hoc. Every single small task gets its own temporary view.
But perhaps I’m misunderstanding something that you’re saying?
The only difference with the past that you need less time to dig through the codebase or documentation, the agent can do it for you and provide only meaningful info, but without a "Mental model"(or knowing what's going under the hood) your own prompts will be the main reason of the sloppiness and high token usage.
Having reached the same conclusions as the author led me to create my first agent to do architecture review, and that's how I learned about the metrics behind good practices that I'd been following for years. LCOM, cyclomatic complexity, that kind of stuff...
It's so easy to ship a lot of code, more effort should be put into ensuring the code is correct, with self-improving feedback loops that involve developers, and dedicated tooling...
But again, a while ago, everything was about prompt engineering, and now you can express you idea vaguely and get a somewhat working result, so this likely will evolve fast as well...
Coding is solved, perhaps, with unlimited token spend on a frontier model. It remains to be seen if it that is prohibitively expensive forever. At my company, we token maxed while the getting was good. But when we had to switch to Anthropic's enterprise plan, and start paying per token, the shit really hit the fan. Now we're retreating back to sane cost levels and finding that - guess what? - people power might just be more cost effective. AI of course is an immense tool to leverage, but still too expensive to create loops and let it run. This will change over time of course, but assuming it is a solved problem is nonsense. Maybe if we solve cold fusion, yes. Until then, evolution is winning the war on entropy.
I'm interested in hearing more about your experience here.
Were you all just full bore shipping a ton with the per token plans? Was it more effective? How many developers @ your company?
It's really interesting to hear that some companies are back-pedaling. We "slowed" down a bit but are still very much using AI and intend to continue using it for... almost everything.
I started Valknut (https://github.com/sibyllinesoft/valknut) when I saw the writing on the wall regarding Agent code structure/abstractions/etc being a limiting factor in the ability to autonomously build projects. My experience was that good linters helped, but it wasn't enough, you needed to be able to enforce information-theoretic related organizing principles in addition to file/function LOC and local complexity metrics to guide agents on how to structure code.
Originally I tried to walk the line between improved agent performance and human readability, but current models are so good I don't think human readability matters much, though at a high level, being able to grok the overall folder structure still matters. I've got my hands full polishing a demo for my game, but I intend to revisit Valknut by crafting an eval set that lets me calculate the difference in agent token consumption and task failure rate between ~isomorphic codebase structures. This will let me loop agents to discover organizing policies that improve them.
Truthfully though, with today's models I don't think this sort of codebase optimization is likely to have much impact below 250k-300k LoC projects, and it probably won't be a decisive win till you're near 1M. Also, the shelf life of a product like this isn't infinite as each generation of models pushes those numbers up while also having new policy preferences that require re-evaluating existing policies.
the AI labs are and have been 100% focused on correctness because it is easy to setup and validate.
Adding one more function that almost does the same thing as another will not break anything.
I think this is just a matter of time. At some point there'll be less value to squeeze out of correctness and then the AI labs will start focusing on maintainability. It's probably a lot harder to set up environment to Train for this behavior though.
There’s one thing I constantly see agents tripping over, I’m not sure what the right word for it would be, but it basically boils down to “making changes in the right places”. They seem to have very poor grasp of where things are supposed to be and they have a tendency to work against the existing architecture. Even in a world where agents are the only ones touching the code you can see how this ends poorly. Unlike correctness I’m not sure there’s an easy way to verify.
I tried writing a few skills to encourage agents to spend time thinking about this but it doesn’t seem to generalize very well.
Maintainability is hard because it requires much more planning than correctness. You need to think about how you can construct code which gives you the most amount of reusability while not compromising on readability and also thinking about future paths and how they are affected. It seems like solving the problem of constructing maintainable code is as hard as solving long scale planning in LLMs
90% of problems are easy once you know what you actually want well enough for you to be able to ask it from an LLM.
90% of code before LLMs was badly copied from StackOverflow anyway.
That 10% that's remaining, I've see 0, ZERO, nil progress. Windows is still awful. Spotify still doesn't work correctly offline. Youtube search is trash. Jira takes 20 seconds sometimes to load a task. LLMs haven't created a new database or a new game engine or a new renderer or anything like that.
The maths breakthroughs are really more of a testament to the efforts of the last 150 years for maths to be an organised verifiable principle. If LLMs had to practice math they way Euler did, they wouldn't be able to find shit.
(sorry if I sound incoherent, just some thoughts while I'm commuting)
Does anybody actually know whether there's a limit to the complexity LLMs are capable of dealing with in a codebase? It's very obvious that they don't write code that is suitable for people to understand it (and it's gonna get worse and worse the more RL is used to train these models), but if there isn't a point at which LLMs also struggle due to the complexity they introduce, then I'm not sure it really matters anymore for a large part of non safety-critical software. I really hope there is, because steering them is, I feel, one of the last competencies through which I can still add value, but is there actually evidence that these models struggle more with poorly maintained code?
> I really hope there is, because steering them is, I feel, one of the last competencies through which I can still add value
Something as simple as output length is a hard linear floor for productivity, even putting aside the obvious context problems that you're intuiting, and it's far from being the biggest cost that arises from steering skill. Learning to make a smaller, faster model do the same work with less tokens is a technical domain that a lot of people don't seem capable of learning. I'm not just talking about "context engineering", but learning how to fine tune, post-train, create better harnesses, design inference setups, etc. If we're both using AI, but I'm beating you to market every single time and with a better product, what is your AI usage actually buying you? Yes, competency and skill is this meaningful right now, and it's highly technical. Not the least of which because you know how to describe the problem in way that gives it a smaller solution and requires less iteration.
Most of the labor who understand the technology enough to do those things lives at the companies selling you these services, but you can absolutely learn to do these things yourself right now. It's actually really fun! A hell of a lot more fun than fucking prompting that's for sure.
Where we're at, I would equate it to the early mainframe era where the programmers came with the computer. I'm placing calls that we follow a similar track and the two will end up decoupling, that "model engineers" are going to move in-house. OpenAI will have a ring to it like IBM does today.
Yeah, exactly my experience. Especially ever since Fable came out, I felt paralysed, many times, about how many decisions I have to hand it. Which is 10x harder when you handed it the reins to build the PR and you practically only knew the compressed initial problem statement. Then it stands up a draft PR and says: "Decisions you owe me: [insert bulleted list with 8 items each a paragraph long]" and you are like: "...oh shit".
It becomes a very heavy and difficult exercise of it walking you through the implementation and the judgment calls it had to make along the way. VERY exhausting.
There's a silver lining though: you do get to gradually clarify a proto project spec and various requirements, but boy does it take time and energy to re-contextualise when the bot tells you that you should make decisions.
I hopped into this hoping it would land where it did! I recently built in basic observations about cyclomatic complexity, churn, and authorship into an 'analysis' view of codebases in ouijit[1].
Some benefits can be reaped immediately (cyclomatic complexity), but others emerge over time (churn). A good example of this is say something like a 500 line file that has experienced 2500 lines of churn, and if that rate of churn is trending down or up. Surprisingly useful for understanding you've got a hotspot with an opportunity to pay down debt by spending more time on API design, or just breaking out whatever subset of the code is experiencing thrash.
The funny thing about complexity is that assuming you're lintting/formatting well, you can do a poor mans check by just looking at something like average indentation per line, deepest line, etc.
Number of iterations solved correctly, on a very large set of iterations, seems like a very good metric. Better than anything else because it measures what we actually care about, not some proxy.
The only caveat is that it's the same model doing an iteration and then using that iteration as a starting point for the next step. So the model is allowed to write absolutely insane solutions, as long as it can read them back, even if no one else can.
One thing that could be done is to use a separately developed baseline coding model B to evaluate the outcome of each iteration. For model under test X to pass an iteration, not only should it be able to solve starting from the previous solution, but so should B, starting from X's previous solution.
Coding just a stage in the software development. Design and specifications which can help in coding is not solved at all and may never - the end result is software reliability is not a solved problem.
Sounds right to me. As AI has picked up, I have found myself drawn to resources that (I hope) help improve my taste and judgment not only of code but also of overall software design. The lessons in Code Complete, for example, have been helpful to me in thinking about managing the complexity of a codebase. It was useful before, too, but perhaps even more so now.
It's a nice article that basically (rage)baits the readers before they realize the author actually disagrees with the premise of coding being solved. So here goes, my higher-level rant on "solving" something with AI.
I strongly dislike all the "X is solved" narratives. What does it mean for something to be solved? A math problem (or any kind of problem), a riddle, a mystery, a dispute. Those are all instances of a particular situation that requires a "solution", but new situations will always come up. I understand that by "cancer/coding/X is solved" in this new age of (gen)AI that means the ability to streamline or speed up the "solution finding" procedure, but even that presupposes a fixed, static, fully deterministic space of the things we are trying to "solve".
Even cancer cannot be fully solved - Demis Hassabis slowly started drifting away from using the word "diseases can be solved" because they can't be eliminated - we can only speed up the process for finding a cure for any particular disease, be it existing or new and/or evolving.
Is bridge-building solved? Architecture? Why are architects still employed? Is solving "civil engineering" or designing an optimally running machine a thing to be solved? What are we trying to do when we talk about "progress with AI"? Even when the "recursively-self-improving-AIs" and "perfect" robots do arrive, we're still bound to work with them, and they'll have to evolve to find new solutions to new problems.
To be clear, I work with and rely on LLMs every day, from coding custom RAG architectures with CC and Pi to research and agentic data science. These bombastic conversations, however need to quiet down a bit so we can get back to work :)
Ya, now that I have some solid AI coding experience under my belt, there does seem to be some gaps between practice and reality. I have a fairly complex codebase which I pretty much hand code everything. When I add a new feature, I spend a lot of time designing and refactoring that feature into the codebase. Either the feature dovetails into the existing design or the feature creates new designs which will then facilitate even better future features. When AI approaches the feature, it just plows the feature in, and with bugs since it has trouble fully understanding the total design. So over time, you have a spaghetti design where you just have a whole bunch of features tied together with no unified design. I guess thats ok if AI is supporting it, but you now have a large cost and bug surface area and an insane human learning curve. So nothing has really changed here, we have been dealing with low quality codebases way before AI came along. I think AI has mastered the one shot single feature, tool, or simple app, but it struggles with the design complexity of a rich multi feature application or system.
I liken it to contractors or overseas engineers. They don't really care about the long-term effects of their code.
I does the thing, adds some code, adds test that cover that code, and responds with "Done."
At a fully agentic "No looking at code" company, we now don't have a great understanding of that code, we don't know whether it's a good implementation or not, whether it fits our current patterns, is maintainable, or composable.
We're entirely in the dark, at the behest of an agent at this point - if we want to maintain velocity. (And I would love to know whether that velocity is real or just perceived). It feels like the code part is faster? But also feels like I have to spend a lot more time up front working through a problem to understand it. In the past, I gained that understanding WHILE I was working through the code. Who knows.
I encountered bugs created by AI. No matter how many times it tried it could not fix the bug, it was introducing so much slop to work around the issue.
I finally gave up and read documentation for 15 minutes and solved the problem.
I will never push AI generated code to production without understanding it, and this is why I only generate small code snippets and copy/paste most of the time.
I was bit disappointed how few metrics the article mentioned. There are tons of code quality metrics that have been thought of over the last 40 years. We don’t have a good idea which ones are worth enforcing though. And we don’t know if the metrics that are good for humans are also good for LLMs
>In my research and tests simply taking the change in the number of LOCs has been a surprisingly effective metric for sloppiness, with the ironic caveat that if we started optimizing for it, it would cease to be a meaningful measure.
This matches my experience. Before working on an issue, I ask the LLM to estimate net LOCs at the final PR based on the scope. It works well, and review steps do flag inconsistencies. But as the OP mentioned, if you turn this into a hard metric vs "design smell", you can see LLMs code-golfing for oneliners.
I've had some success with tokens as a measure of complexity instead of number of lines, but should be combined with additional rules, e.g. disallowing lambdas, exec, eval, compile, __import__ and complex list comprehensions for Python. Fortunately, Python's "ast" module makes this quite easy.
If coding is ever solved, and if software does it, sloppiness probably won’t matter much.
Code will become throwaway stuff, like the results of AI prompts. Cook it up, test it for adequacy, and run it. When something comes along that adds new requirements, just update the requirements/prompt and make a new one.
Goodharts law is invoked here with no discussion. Are we sure that optimizing for minimum LOC that pass all necessary tests produces sloppy code? And: What is sloppy code anyway? If we could define it, could we throw the definition into context and tell the LLM to avoid it?
> could we throw the definition into context and tell the LLM to avoid it?
Every IMPORTANT RULE WHICH MUST BE FOLLOWED STRICTLY to not to do something from my AGENTS.md and skills are violated without any hesitation from LLM. Glorified autocomplete doesn't work this way. Positive rules are more stable on this regard.
I love this train of thought. Code quality is critical, but I don’t think we’re correctly evaling it at the moment. If we could get solid benchmarks measuring the quality of generated code, we might see the models climb those benches fast.
I believe that the era of “ai writes tons of slop code” will be a stepping stone in the longer story, and is simply a current gap in the reward functions.
Per the author - if we can get strong measurements of what good code is, we can train against it and close the gap fast. Excited to see more thinking in this area
Sooooo much of what is considered "code quality" today is irrelevant when robots are writing the code. We've been largely optimizing for things like composability/unit testability in the past 15 or so years, and that's primarily a human concern that's unrelated to the final output.
Totally agreed that we're not looking at the correct metric right now. Increasingly, code quality will be determined by outcomes.
I don’t agree with this. The things people care about with code quality if you really think about it actually match up surprisingly well with the metric which models are trained to emulate in pre training, namely compression and modularity. Those two ideas actually seem to be universal to intelligent systems. Writing verbose highly coupled code is I think provably stupid, though I don’t know that I could formalize it.
On the other hand, there are claims that the best languages for robots tend to be terse (allegedly); I tend to regard a rich domain model (static types, not terse) as a major facet of high-quality code.
Is coding "solved?" I've been using Fable and Grok to vibecode a library to improve text extraction for legal PDFs. (Which have things like line numbers in transcripts that messes up text extraction.) This is a simple, dumb problem that should be like killing a mouse with a nuke. But I got to "pretty good" with a few iterations of not looking at the code or design. The program works, pretty well. But the design and heuristics are only 80% of the way there. Now, the models are just "churning" on the existing design, making ad hoc changes struggling to get to the last 20%. It seems like I'm going to have to turn my brain back on and think about what I actually want to do, which I was trying to avoid.
It's better than writing assembly! But it still seems to be in the realm of "a new kind of tool" even for the actual coding part, without even caring about what the code looks like or security or anything like that.
Without knowing details of your approach, I would venture that your challenge is not with the coding per se but extracting structured data from PDFs. It’s a surprisingly hard problem because PDFs are optimized for preserving the visual structure and layout of the content for precise rendering and printing… NOT for preserving the logical structure of the data!
Which is why the best results these days for extracting structured data from PDFs is by having the model do it directly rather than writing code to do it. It literally takes that level of intelligence to be reliable at it.
A common approach is to provide the model with a template or structured schema describing the format you want the data in, and the PDF itself, and it should return a JSON with the appropriate values filled in. It won’t be 100% but probably higher than what you’re seeing now.
The use case is basically the same ones where a programmer would use grep on source files, and it’s too slow to run the model on it directly for the same reasons.
Existing tools do text extraction (pdftotext) but not that great when there’s internal numbering and stuff like that. It has to be a heuristic approach, and it won’t be perfect. But it’s an interesting data point that the model isn’t able to figure out the best heuristic itself. Or even make incremental progress towards one after a certain point.
I gave a blanket ban on pdftotext to my agents. The output can get so mangled that a smart human wouldn’t untangle it. Did you try understanding the output from pdftotext yourself?
My approach is just ocr-ing with Terra or Gemini flash + checking citations with source both ways. But if I wanted to avoid llm calls, I’d just tell Fable to build a pdf reader directly from pdf binary format. Should be way more robust.
A PDF is a command stream designed for rendering. Interpreting the command stream to get the positions of each glyph is deterministic and existing libraries (I use both pdf_oxide and lopdf) do that fine. Once you have glyph positions, you need to use various heuristics to reconstruct words, paragraphs, columns, headers and footers, etc. For example, in a patent document, there's two columns with a gutter in the middle of line numbers. If you interpret the document as having a single line, you'll get numbers mixed up with the text, which can throw off efforts to find particular phrases. PDF builders also insert all sorts of weird crap into the OCR layers that has to get normalized out.
It's just a pretty pedestrian data-munging problem where there's no closed form perfect solution and you have to use various heuristics to get the right result.
Yes, but the edge cases are infinite and so heuristics don't scale well. As an example, at some point you would likely find yourself with "dueling" heuristics, forcing you to tune them, which is brittle, or find yet another heuristic as a tie-breaker, which ratchets up the complexity. (I just spent a lot of time on an adjacent but much simpler problem before finally giving up on churning heuristics!)
As an example, many times it is impossible to determine the order of some words from just position data without considering the meanings of those words. This is why LLMs / VLMs are so much better at this task, because they can look at the document holistically like we can.
Also, funny that you mention patents, something I've worked on in the past as well! If you're looking only at US Patents, the USPTO data resource is much, much better: https://data.uspto.gov/home -- they provide the text in XML format (https://www.uspto.gov/learning-and-resources/xml-resources) which is also pretty complex but wayyyy easier to parse than PDFs!
> This is why LLMs / VLMs are so much better at this task, because they can look at the document holistically like we can.
Totally agreed. But in this use case, PDFs are the working format, not just an archival format. An offline batch process to ingest the PDFs isn’t feasible. Unless there are some super fast LLMs I’m not aware of that can handle tens of PDF pages per second. It seems like Grok and Claude don’t try to read the PDF directly, they use pdftotext or some Python wrapper over pdfium. But maybe I’m missing something!
>It's just a pretty pedestrian data-munging problem
In other words, how complicated could it be?
Well, if Adobe has been introducing complications and making the format brittle and inflexible over the last 35 years to make it hard for its competitors to write software to process PDF files, quite complicated.
Yeah that’s why heuristics should work on the lowest possible layer, not on pdftotext. If you use pdftotext you’re stripping positional data and other stuff.
Do you use a public set of documents? I bet I could almost oneshot this with my harness :p
Yes, the tool I’m describing works on what the pdf_oxide crate returns, which includes glyph positions.
Here’s a public appendix from a recent Federal Circuit case. It has a representative assortment of documents (opinions, briefs, patents, transcripts) but contains only cited pages to the appendix pagination is non-consecutive: https://www.courtlistener.com/docket/68048163/15/ecofactor-i...
The underlying docket is usually 100 to 1,000 times larger than this but it’s similar types of files and the appendix shows the various types of headers and footers that can exist.
I had Claude evaluate my vibe-coded extractor against yours on the EcoFactor corpus, which I didn't test on, and an internal corpus I have:
> The EcoFactor result inverts. There, on hand-authored gold over 21 pages of clean native text, pdfgeo won the accuracy checks 25/42 to 19/42 — its geometry work on multi-column patents, table cells and timestamp lanes is genuinely better, and nothing here contradicts that. But this corpus is 21,652 pages of mostly scanned, OCR'd, skewed exhibits, and at that scale pdfgeo's robustness is the binding constraint: it loses 6.4% of the corpus to two trivial crashes and an over-strict rotation test, duplicates text on 34 files, and drops 80% of wrap hyphens. lawpdf extracted every page, 7× faster, with 2.5× less furniture leakage.
It looks like the actual geometric analysis came out the same general shape, which is an ad hoc heuristic with hard-coded values attempting to figure out where the gutter is located, etc. Seems like both have some overfitting to the text data set.
If you have more test data I can try this with my harness - you can email me at kolinko@gmail.com btw :)
Thanks for the benchmark, I was seriously looking forward to it! Would you consider such even results good for a one-shot? I wonder how well my harness performs :)
Personally I use only one family for code editing, other families for code navigation.
Models tend to reason about the code using their own intuition and putting other families onto the same codebases may end up with them getting confused. Esp if that other model is as unhinged as Grok.
Also, harness matters a ton. Use a wrong harness with a good model and you’ll get terrible results.
The conclusion is not surprising really, because fundamentally how do you even quantify sloppiness, a famously broad and subjective characterization?
I worked for years in Dev Productivity with engineers who had spent their entire careers in that field, and code quality was always the biggest "unquantifiable". Any of the metrics in the literature (cyclomatic complexity, erosion, etc.) quickly became very noisy at scale. Conversely, for any given metric you would find countless bits of code that do NOT exceed any metric thresholds but were clearly low quality.
People have experimented with many things over many years at Big Tech scale, which produces prodigious volumes of code daily. The conventional wisdom was "Don't bother trying to measure code quality."
An interesting observation from an ex-colleague is that probably the best measure of code quality is its comprehensibility, or "understandability". Maintainability, stability and adaptability are natural outcomes of that. But understanding lies entirely in the mind of the beholder! Which is why it's such a subjective metric, not amenable to simple mechanistic measures.
But now, we probably do have a technology that demonstrates some analog of comprehension: LLMs!
Specifically: tokens. Anecdotally and empirically (based on industry reports like DORA and DX etc.) AI coding works much better with "good codebases" (more specifically, strong engineering discipline) than otherwise. I wonder if that can be parlayed into a quantifiable metric like "tokens to grok / LoC" somehow.
So, if to fix something we need to first measure it, and if AI can measure slop, the way to fix slop from AI may be... more AI!
I just ran this on a personal repo I've been working on for two years. It's 250k loc of Python, that for the last year has been written with agents.
I told Claude to look at this article and the original research paper and calculate the numbers. Verbosity came out to 0.08 (lower than human, that's weird), erosion to 0.495, so between humans and agents. So according to these metrics, I have managed to contain the complexity of code.
"To come back to the point of why agents can’t (really) deal with the slop themselves, we need to look at the evaluation of SlopCodeBench. In contrast to other coding benchmarks, which give the agent a complete list of instructions at the start and then have a set of hidden tests the program needs to pass, they do the opposite. They create multiple rounds of instruction and test iterations, where in between checkpoints the context of the models is erased. Thereby mimicking much more closely an iterative process, like how coding agents are actually used by humans. The result of that is that bad coding decisions accumulate over time and for the strict solve rate, where all tests have to be passed at all checkpoints, even state of the art models achieve 0% pass rate"
I like how this captures with a metric (lines of code and cyclomatic complexity, some other basically tractable measures) in an automated way, something we all by now intuitively know.
I wonder how much of this is due to the AI tooling being taught on sloppy code that humans have written. Over the last 4 decades I've looked at a lot of code on the Internet and there's a lot of slop out there.
The problem is there is no good metric to describe code quality, it cannot be RLd and that's likely why latest models have such problems with slop. It is great to raise the conversation so that labs focus on this more.
Picking specific metrics will probably not work, it would be a mix of Goodhart's law with Bitter lesson. Maybe picking and labeling quality repos, having whole suite of metrics as input features and training some traditional AI classifiers to steer the LLM training.
He used LOC, and it isn't bad. Just this week I had a LLM do a small task, and it produced 500 LOC, every little detail beautifully abstracted out. But 500 LOC for such a simple change looked suspicious to me. As everything must pass human review, I re-wrote it to see what happened. The result was 100 LOC. No human wants to review 5 times a much code.
The issue really isn't "there is no good metric" - when I saw 100 LOC vs 500 LOC for the same thing there was no argument. The problem is Goodhart's Law. Whenever we use a metric use like LOC as a reward function for humans, the result is a disaster. I have no doubt that's true for LLM's too.
I hope this is not too provocative, but where I work the legacy code is such a spaghetti mess and so actively inefficient (queries in loops of loops and plenty of things like this) that just prompt nowaday llm to "write extensive and menignful tests for this module, then rewrite the module clean mvc small functions , small loc per files, well organized files then make pass tests again" often leads to insanely better ai "sloppy" resulting code, plus free test coverage where nothing existed, users instant satisfaction (software become suddenly responsive in less than a second).
From this point it take not much more time to add proper ci/cd suite, rework database models and migrate things with now some comfidence.
It works very well on small isolated projects and new builds where resulting tools are produced insanely faster than before and with not comparable average code quality (ai slop is 100x better here) from customer usual habits and expectations.
It's because this approach raise significantly the quality level so quickly and so cheap that I advocate for it.
It however does not work the same way for large legacy projects to revive. For larger projects the reciepe is almost the same but requires significantly more manual testing on each pass and carefull regression check. But it does work.
Customer is happy, they take back "more" control over their legacy horrors at the cost of some extra loc but properly working code. There is no point to fight against "a bit too much code" in my case
I don't want to speak for those that work in elite very sensitive human responsibilities software domains, but in a lot of place this is a revolution in my opinion.
The most surprising indicator of messiness I encountered wasn't in the code itself, but in the tests. I had two tests that looked like tests but didn't actually measure anything—they simply returned True — and neither static analysis nor coverage tools caught them. The only thing that detected them was mutation testing—deliberately removing a line of code to see if the test fails; without that step, the test is essentially toothless. Also, I found that scores became inflated if you counted cases where a test passed despite the removal of code (because another test caught the issue) as a successful "detection."
thinking about how to accurately describe the problem at hand and figuring out the simplest way to approach it takes a lot of effort well before you get to the implementation step of coding.
Nice! It's interesting to see quantitative ways of measuring slop. I'm curious what _would_ happen if you did plug these back into the LLM as a form of feedback. Goodharting may happen... or it could get the LLM to generate cleaner code potentially?
I have no idea where most people writing code have worked at but in all product and platform teams I worked at the code quality has been much higher than the latest slop SOTA llms can output.
TLDR: coding is not solved.
I have 2 projects, one it's a distributed platform, the other one is a general processing engine with an inner workflow engine; Since gpt 5.2 I've tried new models to work in these codebases where the code is of good quality and every time I gave the model a slice of work instead of a single step from that slice the code, the tests, the comments, the docs and everything else has been suboptimal, unmaintainable, complex, bloated and just slop, unless I micro-manage and do many passes.
As a dev when you make a change you consider the broad picture, you consider the user, the codebase, future requirements, maintainability, performance, your team's understanding and some of these you do unconsciously. We are slow but that's for multiple good reasons, you push the organization/understanding forward not just loc of that specific project. I can't count how many PR notes or comments I've added considering teammates or just for a specific team member.
I don't see any way forward for an LLM to reach that unless it reaches general problem solving, my definition of GAI that could tackle software development or "coding" would be a model that doesn't require additional pretraining to solve new tasks or improve how it solves tasks in the future, it would just learn as it's going.
Can everything I mentioned be solved with current generation of LLMs and lot's of markdown and gates? Maybe... but the amount of effort required would be similar to the effort an expert system (pre-llm AI) would require to embed the rules, evolve them, check them everytime... which would require billions or trillions of tokens.
---
off: I really like the discussions around how to prevent slop and bloated code as it's something it would benefit coding even without LLMs and can fit as another piece of automated infra for checking and ensuring code quality, I hope something materializes.
Plopping in my email to the author below in case anyone else is interested in this kind of thing:
> There are some promising other directions I want to explore, such as coupledness of functions, code churn, cohesion and so on. If you are working on evals and would like to talk, I would be happy to do that: sebastian@earendil.com
Hey Sebastian, I just read your article and it thoroughly resonated with me. I've been working on building something similar to SlopCodeBench, but moreso aimed in the direction of architecture, rather than simple one off "code search functions".
In a nutshell, I'm creating multiple domains of common software architectypes. You can think of these as being as simple as a counter (very common in all architectural explorations worth their weight), todo applications, etc and as complex as an online store, a bank, a wallet, a social communication platform, etc.
Given a single domain, we can extrapolate common functionality that is "higher order" to that domain. Features like data synchronization, functioning offline, sharing information, authentication and authorization, etc all land in this bucket. From a single domain (take the counter, for example), I've laid out my initial plans for the various different levels to concretely observe how bad LLMs are at churn, cyclomatic complexity, poor abstraction planning, etc as follows:
L1: Show a number on the screen
L2: Allow a user to click plus or minus and the number responds accordingly
L3: Show that number on any device running your software, and keep it in sync with all other devices
L4: Ensure that additions and decrements to the count, while a device is offline, will replay to all connected devices once connectivity is reestablished.
L5: Introduce the ability to reset the count to zero, and ensure that commutes properly if an offline device triggers a reset or vice versa.
L6: Introduce user accounts where I must provide an authorization strategy (login with apple, google, passkey, etc) and now segregate a public counter (which anyone can interact with) from your counter (which only devices authorized with your credentials can interact with.
L7: Introduce an action menu whereby I can invoke what is commonly known as a "Command K" menu for the actions that can be performed in the application.
L8: Allow me to speak naturally to your application and ask it to "go up", "start over", etc (this exercises your architecture's ability to be "accessible" to agents (and vision disabled folks as well)
...
...
...
And so on and so on.
Despite its apparent initial complexity, naive solutions to even the simplest domains will be easy to spot with how many lines were changed vs raw additions (how composable a solution is), that cyclomatic complexity measure you mentioned, how many tokens it took, how many platforms (iOS, android, cli, TUI, react native, react) your application will run on, how long it takes to build, how large the binaries are, how much memory is used during the operation of your software, how semantically similar duplicated code across different platforms etc. From all these different values, we can create a hueristical "architecture score" to benchmark against.
I'm also toying with the idea of enforcing that one must submit a bundle of skills, instructions, scripts, etc that I will exercise with my own harness whereby the submitter has to submit a monetary cost with their submission that a budgeting agent must manage the spend and the agents must yield prior to their submission being scored, lest they forfeit the submission.
I'm just quite tired off all the hype and its exhausting and AFAIK, none of the benchmarks actually produce anything of use. One interesting side effect of aligning incentives in the way I've laid out here is that we will have produced open source, connected software that functions well and solves a whole bunch of business needs that all compose together by definition.
Thanks again for your article, would love to have an e-coffee and chat about if there's potential to collaborate on anything here. Despite how powerful llms are, this is still proving to be a tricky endeavor for me.
Thank you! My favorite bit (theoretically) would be the forcing function to produce useful open source goods as a result of running these benchmarks and quiet down all the hype about llms. We get it, they’re very good at making 3d models do dumb shit that doesn’t matter
What has this author written before LLMs? Why should we listen to him and his adjudication of "perfect code"?
Cyclomatic complexity is the oldest paper generating grift for college students. There are hundreds of thousands of useless papers about cyclomatic complexity.
There is some sense of rose-tinted glasses of pre-LLM coding. A lot of human written code, particularly at the enterprise level, was of low quality well before AI automated it.
So much "bad" enterprise code evolved into that state over years or even decades of small changes. Meanwhile, last year I got to watch an LLM-authored codebase speedrun itself into a similar state in only a couple months. And I would say that the enterprise code was actually better. It at least did its job fairly reliably. The LLM codebase was riddled with defects, so much so that it ate up all our time and our feature delivery rate ground to a halt.
There are two observations that really eat at me:
1. Studies seem to indicate that agentic coding uses 2-10x as many lines of code to accomplish the same task.
2. One of the only really well-established empirical results in software engineering is the strong association between LOC and defect rate.
This is happening all over the place right now. There is a ton of greenfield happening, which further adds to the illusion of speed. Eventually you produce a big old pile of shit that even with the help of the LLM is weird to reason about, and it slows way down. Many such cases.
Writing code at enterprise level is insanely difficult. You are constrained by budget, staff, legacy databases/environments, business rules hiding all over the place, and people.
You can't just rewrite everything. So over many years people are touching small parts of the pie.
Yeah, enterprise code has that trope of being enterprise-y, verbose and bad. In my experience, that has always been the opposite. At the big corps/FAANGs I worked at, a single line of change can adversely impact millions of paying customers, so a lot of the verbosity and harnesses exists to dampen the failure modes.
Most of the terribly written stuff has always been at startups, where devs fling nearly anything across the finish line, if it barely works the happy path.
I don't doubt that. But humans still need to be responsible for understanding what they're shipping. And IMO you get your best understanding by actually writing some code. Even if you don't actually ship what you wrote.
Let’s not romanticize it too much... A lot of enterprise systems are built by developers copying an old AbstractBeanFactoryFactory from a 2011 stack overflow thread without really understanding it :)
Nobody is. It's the AI slop which is supposed to replace this shit which barely worked with equally shit shit which doesnt work which people are romanticizing.
Most of the human written code was slop, but the really fundamental and successful stuff we relied upon and which we didnt want to throw away? yeah, not so much. most of that was actually really good.
those EJB monstrosities were routinely swapped out by some saas written in python by somebody who did it properly and werent responsible for a lot of late and over budget projects which barely worked or didnt work.
> But humans still need to be responsible for understanding what they're shipping
I don't necessarily disagree. That said...
Why?
I've been grappling with this myself. There is an easy/obvious answer, but I wonder how stable/permanent it is. If you feel strongly about this, are you willing to unpack your judgement?
Yeah... like we all get to start green field projects and write all the code we should understand. Many of us cut our teeth on bad legacy stuff with no proper documentation made by "engineers" long gone. At least a LLM can easily make sense of this mess.
Indeed. And not fair comparisons ”look at the quality of this small one-shot Claude hobby project. The quality is less than this major open source project written by some of the best developers in the world”
To be fair the pitch has frequently been that Devin/Claude/Astra/whatever is some sort of superhuman bottled John Carmack that will single-handedly replace entire teams of developers.
Yea, that extreme side exists too. Truth is inbetween. AI with the instructions from a dev that knows what it is doing writes better code than most regular 9-17 devs.
1. People didn’t wear that as a badge of honour though.
2. A lot of it wasn’t. Low quality code/speed serves a purpose for point solutions and scripts etc. That’s not the same thing as writing a core system and if the user doesn’t put any credentials in for an S3 bucket then it falls back to giving information about your own S3 bucket (as I’ve seen just this week).
3. Plenty of companies you can discern the difference between mission critical systems versus “business” systems where if it falls over it’s annoying but not the end of the world.
This was always due to pressures by management and the company environment, not the workers themselves. It's hard to blame the people writing code when they have to deal with nontechnical leadership that wants to have a feature factory or never given appropriate resources to solve problems.
Blaming workers is always an excuse by poor management.
The pressures from management and the company environment are not always a bad thing. It really depends on whether the pressures are coming from a logical business perspective or whether they are just coming from stupidity or ignorance. In a business environment, taking a long time to ship great code can mean that the company goes out of business, and then the software developers have a lot of great code and no income.
Wait. I didn't say anything about making people miserable. And I'm not talking about some weird kind of pressure like yelling at people.
I just mean the normal, almost inevitable kind of pressure that comes from the business situation. Management has to somehow figure out strategies to handle the pressure and it has to communicate these strategies and the reasoning for them to the engineers in a constructive way.
Generally, this is the pressure to be competitive and make money. It does the developers no good if they spend so much time writing great code that the company goes out of business.
Nah, you're still blaming workers and not leadership. If leadership is okay with not training workers (something American corporations would do in the distant past) then it's not fair to continue to blame workers when leadership is clearly aware of the problem and would rather pocket the money than help workers.
These companies pay management more than workers for a reason, if you can't even admit that they are to blame then what are you trying to do here? Just attack workers for what reason exactly? Being anti-worker is a great tell to never trust a person.
We're talking about professionals here. People who (at least in the US) often make several multiples of the median worker. Competence is assumed, and every company I've been at has had programs to pay for additional school if the employee wants it. IIRC at least one had an explicit book allowance, and I don't doubt that I could expense books right now if I asked. Do surgeons and lawyers complain so regularly that management doesn't train them? Or are software engineers just this desperate to be seen as "not a real professional"?
Who even is supposed to be training us? We're supposed to be the experts. Unless you mean mentorship, which is also generally already a thing at any company that has more than a handful of engineers.
There is no professional developer in the US. There are no licensing requirements to write code. There are no repercussions against developers that write code that immiserate or kill Americans. The person slinging wordpress plugins at an agency is equivalent to the person writing malware at Meta in the eyes of the government.
No one is training us because there are no regulations in our industry. Sorry but I still reject the premise of the other poster.
Also yes, in other industries professionals do complain when they aren't given their mandated time to learn while working on the job.
We are given time though. Maybe not at some small sweatshop? But at least at big companies all the jobs are salaried, and either no one cares how exactly you spend your time, or they explicitly say it's fine to spend some time learning, and will pay for materials or courses.
They're generally not going to teach you how to program or what the latest technologies are or what the best ways to do things are because that's literally what they're paying you for. But IME they definitely give you plenty of time and autonomy to be learning things at work.
I do blame management for letting these people through the interview process and then not firing them. But that’s independent of the fact they exist.
Training doesn’t solve every problem, the worst programmer I ever worked with that a PHD in computer science. Everything he made was horribly slow, wildlife overly complicated, and buggy. Worse he wouldn’t listen to anyone correcting his issues. He’d store numbers in the database as strings to be database agnostic etc.
Let me throw a curve ball at you: do you accept the premise that most modern corporations are centrally planned economies under the rulership of monarchies, oligarchies, or general authoritarians? If so, do you think introducing democracy into the workplace would help alleviate issues you care about?
You do not like bad workers, management doesn't care. They pay bad workers the same as you, bad workers can get promotions the same as you, you will also get laid off with the bad workers as well; or maybe even worse, the bad workers get promoted into management themselves. How do you want things to change in such an environment?
You have no authority to do anything meaningful as a single worker, what if you were given a voice to actually make these claims and have other workers decide what to do based on your voice?
Workplace democracy seems like an interesting concept to explore if you truly want to create better environments with beneficial outcomes to all, not just the few:
> do you accept the premise that most modern corporations are centrally planned economies under the rulership of monarchies, oligarchies, or general authoritarians?
No stock owners are ultimately in control of public companies. That doesn’t fit any of the models you just described.
Similarly companies are not independent of government control which inherently separates them from management systems associated with governments. A CEO is limited by the law in ways that a dictator isn’t.
I've only worked at companies with great leadership. This is in the Nordics with very strong worker protection. And yet most of my colleagues including myself have been pretty terrible and write dirty code. It's not a management issue and it's not anti-worker to acknowledge this fact.
Your great leadership doesn't seem to care, so either your management knows better than you or maybe you should push back on the notion that you worked with "great leadership."
Only poor leaders ignore their workers, which is what you seemed to have actually experience.
I would argue the average code quality of LLM's today is much higher than pre-AI code quality. It's better documented, more readable, and has fewer bugs. There was a brief period where frontier models were still worse than the average developer, but that period among frontier models is well past us.
Are you saying that pure vibe coding by a non-technical person produces better code than pre-LLM developers, or that experienced dev + AI produces better code?
Both of those things are very different, and AI shouldn't be the one taking the credit if it's the second case.
I've been doing development, in one way or another, since the 90s. I've worked with dozens of teams from enterprises to startups. Hundreds of developers. The quality of work has been all over the place, but the majority was not great.
I'm arguing that what people today call "AI slop" is already higher quality than what most developers created historically, and the fact that tests and documentation pretty much come for free now means that the floor has been raised.
The quality of AI generated code is not great. Yes, it will get better. It's already better than 65%+ of what regular devs can do AND it is faster to produce, iterate, and release.
This is off-topic, but I strongly dislike AI written documentation.
When I see AI house style my eyes glaze over. Just this morning I reviewed an RFC from a colleague that he said was a spec for a web service. The document had no introduction, no context, it described endpoints for 2 distinctly different services instead of 1, and made no effort to reconcile why there are 2. It was scattershot with details, some of them important, some completely irrelevant. It was replete with typical LLM-ism.
Basically, it was a dump of a conversation he had with an LLM. As a document to build shared knowledge, it was nearly useless. The only feedback I could provide was a polite "I do not understand what you are trying to build".
But, supposedly, another engineer is already working on implementing this spec. I assume the other engineer just cycled this "spec" into his LLM, and off the two of them went. \o/
They are trying to pull me into their project right now, I stood up some containerization infra for them. But, oh boy, do I not want to join. I looked over their codebase, by LOC the codebase is 35% comments, and a lot of the comments are contradictory, there are dependencies that are not used, there is no tooling of any kind (no type checking, no linting, no PR process), there is no auth (this code is already running in production lol -- they have public endpoints exposed that can be used to scrape/mutate internal company data). Another 30-40% of the codebase is unit tests that test trivial stuff like whether their framework's serializers and ORM work, ex: x=DB.create_x(arg=1), assert(x.arg == 1).
At the intuitive level, I do not understand people who say coding is solved... To me it seems like LLMs are a multiplier (LLMs are amazing, sci-fi level shit), but if you multiply a negative number or 0, you get something that is <=0. Making agentic coding work requires a lot of discipline & expertise.
Oh man, the "unit tests" that test whether the framework/browser/language is doing what it's supposed to drive me insane. Those have their own tests already! Test the unit under test, that's why it's called that!
AI documentation is practically worthless ime. I forbid it in my projects. It's almost always more useful to not have any documentation and read the code than to rely on AI docs.
Coding is not solved, correctness is not a feature, it is the bare minimum. If your code does not do what it is supposed to do, you could as well have no code at all. Efficiency, security, maintainability, reliability, readability, understandability, extensibility, maintainability, observability, portability, ... this is what high quality coding is about, not that it works, that is a given. And in my experience current models are pretty bad at this.
My experience is a little different. For higher abstraction languages the output is largely acceptable in my work. I always consider that LLMs don't know what I don't tell them and they have limited context to work from. Coding issues I often identify:
* Efficiency. Marginal by default. Coding efficiency problems often appear because LLMs dont usually consider the entire codebase or future plans (although they do guess at some futures). Sometimes they write/name things in ways that are lazy/wasted cycles. Most of the time, they don't.
* Security. Marginal by default. I say they do pretty good. Considering all the failure modes, not so much.
* Maintainability. Marginal by default. Mostly due to the careful consideration of modularity, upgrade paths, etc. while often taking wildly different approaches to solutions without having specific broad instructions. Even then, there can be big gaps in quality.
* Observability. Not acceptable by default. There's usually some consideration and can often one-shot.
* Portability. Not acceptable by default. Good, if you specify what those targets are. Regardless, testing validates this above the coding and models are very good at hitting functional test targets. This is less of an issue in something like Java ofc.
This sounds roughly right to me, except for "maintainability". In my experience, agents really don't like deleting code unless you explicitly ask for it. If you're not careful, you end up with new better implementations of things but with the old implementation still around in perpetuity. Humans do this too of course.
They're being trained by a lot of repetitions of one-shot bugfixes and feature implementations. If they don't have to maintain code over the long term they have no incentive to not take the easy win with lower immediate risk and higher longer-term maintenance burden.
The shadow of Chesterton's Fence is relevant. The LLM doesn't know why code exists, if it doesnt start traversing up the scope of a project. Even then, it can't be sure that the code isn't a dependency of something else outside the project (especially if there's a side effect). I'm not sure it's ever going to be easy to address this concern in a straightforward and portable way.
I do sometimes see duplicate functions, which is troubling.
Humans had to get it drilled into them that "+12 -440" is a damn good line stat, and that keeping around dead code is bad, especially in the age of version control.
Not too surprised that LLMs also don't "get it" by default?
Right. It feels like we're living in a parallel world or something. Can agents code? Sure. Can you let them code on their own for a serious production project? Not a chance.
Which serious production projects have AI agents coding on their own? And I’m assuming that means they are routinely taking tasks and deploying them to production autonomously
Right. The only way I'd feel comfortable doing that is if I spent an inordinate amount of time writing very detailed specs, so in terms of labor offset I'm not sure the juice would even be worth the squeeze there. In other words, a sufficiently detailed spec is nearly indistinguishable from code.
What people seem to be wanting is for an agent to infer vast complex data from terse simple data, which I think is probably impossible on a philosophical level. There's real information loss in language, and compute can only make guesses at the end of the day. I really don't see how we bridge that gap.
Astra does exactly the same sort of things that Sol or any of the previous agents do. They duplicate code, overengineer, miss the point, etc.
I was very optimistic about it when it was announced and saw all the demos, but a week later I find it only marginally better (and in some cases worse) than before.
Yeah, Astra, not like it matters. It will gleefully make db queries that perform full table scans instead of adding an index. I need to intuit that that's what's happening and then explicitly instruct it not to be negligently dumb.
Consider that the people who have never have never had to be on-call for a product that results in losses for the company in the event of an outage, and/or have never had to attend an RCA meeting have just as much of a voice in this discussion. Some segments of the software industry don't even have to worry about those sorts of things, and it's easy to get myopic after a while...I doubt that a long time dev at hobbyist-centric wearable manufacturers or audio plugin shops has the same sorts of concerns as someone who has specialized in ad delivery for social media sites.
Allowing only people who have served on call or made concrete contributions to multiple RCAs to have input on the software development process is a wonderful idea!
me too; and my coding agents are slowed down (from developing features) because I require them to refactor the code to be more readable; my code metrics tests force AIs to leverage functional programming and design patterns).
Yes agents can produce code that compiles and runs, but I had to add tools to keep them on track, document their work, follow a process, check their outputs. I also use other AIs to generate developer documentation and review code.
It is like managing a bunch of idiot savant eager-to-please interns, except unlike interns, coding agents do not (yet) learn and improve on their own.
I doubt many people here are brave enough to claim their code does what is supposed to do in every conceivable case. Maybe you have high confidence in the correctness of parts of the code. Correctness of an application is murky though. Things we build are never fully correct, merely correct enough. Like maybe you're responsible for the UI in a web app and you're using your expertise to ensure it gracefully handles display across browsers and a gamut of screen sizes/form factors. But are you also verifying how it works when localized with an RtL script? Are you checking every change you make against CJK?
Sure, every sufficiently large codebase will have bugs somewhere, but it will work correctly at least something like 99.999 % of the time after ironing out the bugs on the common code paths. But that does not change the fact that being [mostly] correct is the lowest bar you have to cross.
And there is another problem: LLMs generating too much code, code that is doing more than was asked. And that cannot be fixed by tests. Usually, we create tests for wanted behavior and expected exceptions. But we don't create tests for undesired behavior.
Code cost is almost down to zero. If you move the point of “just leave it to the machine” from the compiler (where humans used to do the coding) to the high level logic (now with LLMs) then in most cases more code does not really matter. Like, why build and maintain an abstraction where the LLM could implement this many times over each time with different subtleties? Why use a library with its own constraints when you could have exactly what you want? Why use cross platform frameworks when you can just one shot the thing to N different platforms?
It’s not even slower. You can have code that’s larger yet more performant (stripping away abstractions can do that).
From time to time I try to do a pass of coalescing flows and cases and removing dead code to reduce the context and prevent the LLM from tripping over itself. But if it’s exclusively LLM maintained code I don’t care too much if there’s more of it.
Just last two weeks I had to slap Fable, three times, to stop writing 1000-2000 lines of defensive code... because of DB columns I just forgot should be NOT NULL. That was it. Nothing else. I told it that, boom, -4800 coding lines: gone.
LLMs defend the status quo and they regularly lose sight of everything bigger than the current PR they are working on.
I too am gradually making peace with the fact that LLM-maintained code does not have to be 100% readable for humans.
But this is not about readability. It's about the data model. So one concession I am willing to make is: don't care too much about the code _BUT_ manually curate the data model. So far: small wins on iteration turns and code volume producing. Too early to tell but for now I am happy with the results.
Documentation is important. I would say Opus' propensity to write documentation that documents non-features is part of the problem being discussed.
And the problem isn't just that it says what the software doesn't do, most of the things it claims are in fact meaningless, it's not even clearly describing something the software shouldn't do.
"Coding" is just a poor term for this, as there's so much room to weasel different meanings out of it. At every company I've worked in past 20+ years the "coders" were engineering a product from wrong or unclear requirements and specs from non-technical people. The act of coding was secondary (but an important throttling function to make us stop and think about what is even possible or makes sense.) Never did FAANGs, so YMMV.
Really doubt we are near that being solved with non-technical folks + LLMs. I'm seeing people gleefully rebuilding products with the exact same blind spots in their understanding/logic using LLMs. Claude, etc are not seemingly able to "AGI" around goofy asks. The CSS looks a little nicer than their legacy products though, lol.
second sentence of the article: "Just because the code is formally correct doesn’t mean that it is not introducing unnecessary abstractions, creating duplicates, or just making bad decisions overall. This is not a groundbreaking observation, most people who have vibe-coded a project, have realized that each additional feature can sometimes lead to an explosion of lines of code (LOC)."
I think this is a two things can be true situation, where our colloquial meaning of coding is not quite adequate to indicate the full range of criteria that really matter, but also, that criteria as a totality (maintainability, reliability etc) is something that can also be targeted and optimized for.
This reminds me a bit of a PhD Comics webcomic that confidently claimed we would "never" cure cancer, on the grounds that cancer is not one thing. And I don't know that we will ever actually cure cancer, but that wouldn't be the reason. Correctly noting the problem space is bigger than a layperson would initially appreciate is a lot of things, most of them helpful, but the one thing it's not is a formal a demonstration of optimizing against the problem space as a whole.
It's not solved, I agree. But if we pretend it is we can prepare for when it actually becomes solved, if ever, and measuring sloppiness is a worthy pursuit even if we never "solve code".
I just setup a large refactor with Astra and was feeling super lazy and let it mostly do it without my usual extreme micro managing.
The refactor ended up adding 22,000 loc.
I went in there and quickly read through it, laughed my ass off. Reverted the work tree. Micromanaged a new refactor. Net lines of code for something really elegant and easy to reason about was -3k loc in the project.
In case you are wondering why vibe coders are doing 30k loc a day, this is why.
This has been a problem in the software industry for a loooong time, though. Too many developers see "It works" as the end state, a signal that you're done. "It works" is just step 1.
You come across as someone who has never worked on a real software project. Humans create tons of bugs on a regular basis. AI is already better than most programmers.
I'm definitely not experienced enough to know but I read people are having it write somewhat elaborate documentation beforehand. Have it figure out which parts of the code may be touched, what other things will be affected and which uncertainties it has. Basically a full report before giving permission for any code to be written/modified. Again, I haven't seen it, don't know how common this is nor how effective. Though it sounded interesting.
I have worked my entire life as a professional software developer and I agree, even among senior developers I would guess [way] less then 10 % consistently produce high quality code. But if I have to decided whether I want to use an AI to help me write code, I does not matter if it can write better code then an unexperienced junior, it has to write better code than I would on my own.
I've found I can produce 10x more code than I could otherwise, of lower quality than I would otherwise, but the speedup is worth it. Extensive testing is what makes it work, with every bug becoming a red first test with a fix. High level compartmentalization keeps everything on track, you don't let it do the big picture architecture, but you let it do each component as decided on and work through the bugs later. I've seen much worse from teams of humans and I've accepted the drawbacks that are slowly going away with each new model.
> I've found I can produce 10x more code than I could otherwise, of lower quality than I would otherwise, but the speedup is worth it.
The speedup of slop production being “worth it” is what we, as a society, are having trouble evaluating at this point in time. In all likelihood it’s worth it only in the short term.
A team of programmers is "worse" than their best member. Worse quality of code, worse thinking. You can usually get more done with a lot of mediocre hours of work than a few brilliant hours, though.
AI I've used isn't a better coder than I am - it's just got a lot more hours in an hour than I do.
I’m pretty sure that same discourse was seen every mass production epoch, from textiles to electronics. Yet here we are, hand-crafted high quality things are rare and expensive.
> mass production epoch, from textiles to electronics
I see this comparison a lot, and I think it's a trap, because it invites us to confuse scaling duplicates with scaling design changes.
Duplicative mass-production was always core to software from the moment it first became "soft". A factory churning out 10,000 copies of the same book maps to 10,000 downloads of a single software release. The paper and bindings of the book may be below hand-crafted standards, but the words are largely unaffected.
In contrast, LLM-coding is the design and prototyping stage. So if we want to learn from textiles/electronics, we shouldn't be thinking of acres of looms, but instead about fashion-design, custom tailoring, determining patterns for clothes, designing new appliances, choosing circuit layouts, etc.
They care in principle, for the most part bad code makes changes and extensions slow to implement and causes unnecessary production issues which costs time and money. But there is always the tension between implementing something quickly now and being able to implement things quickly in the future and unfortunately the preference is almost always quickly now despite everyone knowing that this is the way more expensive choice in the long run.
It isn't necessarily the more expensive choice in the long run. Let's say that companies A and B are direct competitors who start from the same point. Company A quickly codes a bunch of buggy software and ships it in a month. Company B takes its time and ships good, clean, well-organized, mostly bug-free code in five months. Company A makes money. Company B goes out of business and its code is useless, its nice qualities irrelevant.
This is just a hypothetical example, I'm not saying that this is how it would necessarily go in all cases.
But they also ask why the next feature is taking so long, because of the mess we created with the last feature, and I would consider this caring in principle. Unfortunately this almost always filed as unavoidable.
Yeah, the author leads with a blatantly false claim that models are almost perfect at generating code. They are not. They write pretty bad code, worse than any capable human would.
And all of these things will be solved one by one.
It's astounding to me that people can see coding get solved and not think every single one of these tasks won't be solved too.
Why do you not think these things aren't going to be completely automated? What makes these tasks special?
Fable and Astra can one-shot video games with compelling novel game loops. They can do systems programming, distributed systems, robotics. I haven't found a weak point.
Seedance 2.5 can make video better than the manual labor of VFX artists, 3D artists, and animators.
Nano Banana and GPT Image can do a better job than graphics designers.
LLMs just solved a Millennium Prize Problem, and there are probably more that will fall in the coming weeks.
Just wait. All of these things will be solved.
There is no "stopping point".
Edit:
Don't anticipate that 2036 will look anything like 2026.
Will Smith spaghetti doesn't stay that way forever. Trillions of dollars will be spent on solving these problems. They will be solved.
May the iterative loop of adding new axes to evaluate on be a natural, healthy progression, instead of needing to frame it as an us-them problem?
If you value humans intrinsically, this is necessarily the loop that will converge. I don't think humans have deep intensional a priori knowledge of the structure of reality. If we did, then we wouldn't need tools like AI because we'd be a superset of that. We can only observe and judge.
If we don't value humans, then sure, I think AI is at the point where it can kill all humans (conditional on sentience and resources etc). Two ways to solve a problem - solve the problem, or eliminate the problem statement. Plenty of easier vectors to eliminate the "problem statement", than say, try to solve problems such as making human life better. If you do value the latter though, there will necessarily be human judgers. That's how it works.
But lets be fair, if an expert would use AI today to build something with this, I would feel a lot more confident than not doing this.
I would start with the base architecture and add all the guardrails for a distributed system, i might even go so far to leverage the math skills of a frontier model like fable or astra. I would for sure have the proper budget for using Fable/Astra.
Tell that to the mountain of failed AI slop games on Steam! As a game dev, building compelling, fun games is not even something humans are good at doing consistently. The AI can build the tech, but it can't make something 'fun' yet (unless your bar for fun is simply that a tool created a thing).
I think the usage of solved is silly. Things get automated, they basically never get ‘solved’.
Will all these things get automation? Yeah sure. But the idea that they will be perfect automated solutions applicable in all cases is just marketing, it’s not reality.
I code to make money, and the kind of stuff I work on doesn't kill people or lose massive amounts of money if it has bugs, so to me velocity to a solution is much more important than velocity to understanding.
Are they really though? And if so, how much of that is due to the hyper growth in this specific space?
Meanwhile, the SWE job market across all companies seems pretty rough right now. Talk to someone looking for a job. Most companies seem to be in a holding pattern - little to no new SWE positions available.
The job market sucks for absolutely everyone right now, it's not a SWE thing. Fwiw I've had five recruiters reach out to me in the last four months and I'm not even applying places.
It's a far cry from 2021-2022 but I'm not convinced it's worse for SWEs today than it is for everyone else
I think he's referring to contracting companies. Cheap software used to be outsourced to India, but it's now getting outsourced to AI. Companies that produce novel things are still hiring, often to manage agents and review output.
It will be solved when there is no more code left to write.
Code is an abstract concept that is not bound to the physical world and I imagine that future will have some much more of it that it is difficult to comprehend. Everything will be code and more code will be written than ever before.
Code will never going to be solved. The question is how much humans will be involved and I think the evidence is that perhaps just a bit. However, because we are talking about vast libraries of code even if we are involved in under 1% of all code and decision making that is needed around the code, there are still not enough developers out there to take on the task.
Code and software and applications are an intermediate stage. The final stage is an AI/LLM that just does the thing that is needed without any code being written, there are no applications or programs, just an AI that does everything.
Coding is not solved. It's only solved when coding becomes something you do because you want to, just because you like doing it, the same way I bake bread at home because I like doing it, not because I have to. Right now we still have to be hands-on - to a lesser extent, yes - but we still have to review and hand holding AI agents to get things done.
All: please don't post generic reflexive reactions to titles. That's covered by this guideline, among others, in https://news.ycombinator.com/newsguidelines.html:
"Please don't pick the most provocative thing in an article or post to complain about in the thread. Find something interesting to respond to instead."
I've taken the provocative bit out of the title above, but please remember that we want reflective comments, not reflexive ones, in HN threads.
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor....
Really glad to see folks looking into quantitative approaches to give agents feedback on code quality. This post looks like a good start!
My main feedback for the authors would be, the most important problems for sloppiness are global properties, not local ones. In my experience an agent, like a human, has finite capacity for its attention, but if it runs into local sloppiness that gets in its way, it can fix it on a by-need basis. The technical debt issues that matter are usually global issues that aren't so easy to fix: they require global analysis and global refactoring.
I don't know the answer, but I think we're going to need ways to measure architectural properties, like separation of concerns, clear architectural layering, well-defined interfaces, etc.
Do you have specific examples of what you mean here?
Coding is not just the program running in memory, its also the process of distributing the mental model of understanding among the team.
If humans increasingly are kept out of coding, then who holds the mental model?
If AI holds the mental model, by definition human prompts will be over lossy channel. This is true without AI too. Software quality is directly dependent on good devs that translate from business/PM speak to technical decisions.
So is coding solved now? it was already solved decades ago.
Maybe I’m not even waiting for a model which is 100% at code, but one which is almost 100% at interactively exposing its (almost? correct) mental model.
If I put it like that, it seems to me we are much far away.
We'll invent other tools to help with the mental models, ones that are more intuitive.
Imagine being able to walk into a house and see immediately the leaks dripping from the ceiling, the clunky layout and smell the stinky garbage that hasn't been collected.
We're already in a sort of IDE when you think about it, we're operating a much larger mental model than any code-base, and we rarely remember that we're operating with such a model.
> Imagine being able to walk into a house and see immediately the leaks dripping from the ceiling, the clunky layout and smell the stinky garbage that hasn't been collected.
No need to imagine. Just open the README.md of many projects.
Your project README lists all of the bugs and race conditions in the application? Cool!
Mental models existed before coding. The pyramids were not built by just a couple of guys hauling some rocks together. It was a highly structured organization.
Or the East India Company or many other great organizations that achieved big things before computers.
Who says anyone needs to hold a mental model?
Imho a better way of thinking about it is in terms of views - you can have an unimaginably complex codebase that nobody reads in full, but either agents or humans receive views/perspectives on that codebase.
Before LLMs we couldn’t have built tools that would deliver such views, now we can.
I mean... Nobody NEEDs to hold a mental model, but we've been increasingly trying to build that mental model into an agentic codebase... maybe we're doing it wrong but it is NOT going well.
The agents jot down absolutely everything (and more, which is the issue), thousands and thousands files of perceived business logic, decisions, workflows, architecture, data modeling, etc... It gets stale extremely quickly - we change one thing in the system and 570 "shards" need to be updated. I stumble on more and more things every day that are untrue about the agents "mental model" and it appears to just keep doing it. One small addition not properly reviewed by a human introduces a cascade of misconceptions system wide.
We've tried these "views" you mention, we call them subsections, but it runs into the same problems. One falsification ends up poisoning entire projects.
Remember when we use to refactor functions in a codebase in order to slim down complexity? Yeah - we ended up doing that but with documentation... IMO code is deterministic and we have trained professionals who know how to read it quickly and modify it. (We don't even need to write it anymore, but reading it is the simplest way to ensure no-loss understanding of a piece of functionality.)
Oh, I just dropped all the comments. Memory and session histories are for storing these things, comments just add confusion.
Subsections - how do they run into the same problem? What falsification? The views I mean are dynamic ad hoc. Every single small task gets its own temporary view.
But perhaps I’m misunderstanding something that you’re saying?
The only difference with the past that you need less time to dig through the codebase or documentation, the agent can do it for you and provide only meaningful info, but without a "Mental model"(or knowing what's going under the hood) your own prompts will be the main reason of the sloppiness and high token usage.
Having reached the same conclusions as the author led me to create my first agent to do architecture review, and that's how I learned about the metrics behind good practices that I'd been following for years. LCOM, cyclomatic complexity, that kind of stuff...
It's so easy to ship a lot of code, more effort should be put into ensuring the code is correct, with self-improving feedback loops that involve developers, and dedicated tooling...
But again, a while ago, everything was about prompt engineering, and now you can express you idea vaguely and get a somewhat working result, so this likely will evolve fast as well...
curious how did you set up your Architecture subagen?
Coding is solved, perhaps, with unlimited token spend on a frontier model. It remains to be seen if it that is prohibitively expensive forever. At my company, we token maxed while the getting was good. But when we had to switch to Anthropic's enterprise plan, and start paying per token, the shit really hit the fan. Now we're retreating back to sane cost levels and finding that - guess what? - people power might just be more cost effective. AI of course is an immense tool to leverage, but still too expensive to create loops and let it run. This will change over time of course, but assuming it is a solved problem is nonsense. Maybe if we solve cold fusion, yes. Until then, evolution is winning the war on entropy.
> Coding is solved, perhaps, with unlimited token spend on a frontier model.
Coding is also a solved problem with unlimited salary budget on the very best developers.
I'm interested in hearing more about your experience here.
Were you all just full bore shipping a ton with the per token plans? Was it more effective? How many developers @ your company?
It's really interesting to hear that some companies are back-pedaling. We "slowed" down a bit but are still very much using AI and intend to continue using it for... almost everything.
How did the developers react when they had to go back to writing their own code instead of playing Nintendo Switch between prompting sessions?
> had to go back to writing their own code
I hate it. My fingers are cramping and my head is hurting.
I started Valknut (https://github.com/sibyllinesoft/valknut) when I saw the writing on the wall regarding Agent code structure/abstractions/etc being a limiting factor in the ability to autonomously build projects. My experience was that good linters helped, but it wasn't enough, you needed to be able to enforce information-theoretic related organizing principles in addition to file/function LOC and local complexity metrics to guide agents on how to structure code.
Originally I tried to walk the line between improved agent performance and human readability, but current models are so good I don't think human readability matters much, though at a high level, being able to grok the overall folder structure still matters. I've got my hands full polishing a demo for my game, but I intend to revisit Valknut by crafting an eval set that lets me calculate the difference in agent token consumption and task failure rate between ~isomorphic codebase structures. This will let me loop agents to discover organizing policies that improve them.
Truthfully though, with today's models I don't think this sort of codebase optimization is likely to have much impact below 250k-300k LoC projects, and it probably won't be a decisive win till you're near 1M. Also, the shelf life of a product like this isn't infinite as each generation of models pushes those numbers up while also having new policy preferences that require re-evaluating existing policies.
This is once again RLHF loops.
the AI labs are and have been 100% focused on correctness because it is easy to setup and validate.
Adding one more function that almost does the same thing as another will not break anything.
I think this is just a matter of time. At some point there'll be less value to squeeze out of correctness and then the AI labs will start focusing on maintainability. It's probably a lot harder to set up environment to Train for this behavior though.
There’s one thing I constantly see agents tripping over, I’m not sure what the right word for it would be, but it basically boils down to “making changes in the right places”. They seem to have very poor grasp of where things are supposed to be and they have a tendency to work against the existing architecture. Even in a world where agents are the only ones touching the code you can see how this ends poorly. Unlike correctness I’m not sure there’s an easy way to verify.
I tried writing a few skills to encourage agents to spend time thinking about this but it doesn’t seem to generalize very well.
Maintainability is hard because it requires much more planning than correctness. You need to think about how you can construct code which gives you the most amount of reusability while not compromising on readability and also thinking about future paths and how they are affected. It seems like solving the problem of constructing maintainable code is as hard as solving long scale planning in LLMs
Coding has been solved for 20 years at least.
90% of problems are easy once you know what you actually want well enough for you to be able to ask it from an LLM.
90% of code before LLMs was badly copied from StackOverflow anyway.
That 10% that's remaining, I've see 0, ZERO, nil progress. Windows is still awful. Spotify still doesn't work correctly offline. Youtube search is trash. Jira takes 20 seconds sometimes to load a task. LLMs haven't created a new database or a new game engine or a new renderer or anything like that.
The maths breakthroughs are really more of a testament to the efforts of the last 150 years for maths to be an organised verifiable principle. If LLMs had to practice math they way Euler did, they wouldn't be able to find shit.
(sorry if I sound incoherent, just some thoughts while I'm commuting)
>LLMs haven't created a
I don't think this is true. They very well might have, but a specific one, not a generic reusable version you find on GitHub or whatever.
Does anybody actually know whether there's a limit to the complexity LLMs are capable of dealing with in a codebase? It's very obvious that they don't write code that is suitable for people to understand it (and it's gonna get worse and worse the more RL is used to train these models), but if there isn't a point at which LLMs also struggle due to the complexity they introduce, then I'm not sure it really matters anymore for a large part of non safety-critical software. I really hope there is, because steering them is, I feel, one of the last competencies through which I can still add value, but is there actually evidence that these models struggle more with poorly maintained code?
That kind of complexity is combinatorial so "a 4x4 doesn't stop you getting stuck, it just means you get stuck further from help."
> I really hope there is, because steering them is, I feel, one of the last competencies through which I can still add value
Something as simple as output length is a hard linear floor for productivity, even putting aside the obvious context problems that you're intuiting, and it's far from being the biggest cost that arises from steering skill. Learning to make a smaller, faster model do the same work with less tokens is a technical domain that a lot of people don't seem capable of learning. I'm not just talking about "context engineering", but learning how to fine tune, post-train, create better harnesses, design inference setups, etc. If we're both using AI, but I'm beating you to market every single time and with a better product, what is your AI usage actually buying you? Yes, competency and skill is this meaningful right now, and it's highly technical. Not the least of which because you know how to describe the problem in way that gives it a smaller solution and requires less iteration.
Most of the labor who understand the technology enough to do those things lives at the companies selling you these services, but you can absolutely learn to do these things yourself right now. It's actually really fun! A hell of a lot more fun than fucking prompting that's for sure.
Where we're at, I would equate it to the early mainframe era where the programmers came with the computer. I'm placing calls that we follow a similar track and the two will end up decoupling, that "model engineers" are going to move in-house. OpenAI will have a ring to it like IBM does today.
In theory - if an LLM could handle infinite complexity, I still think that the business issues + decisions end up getting in the way somewhere.
AI: "You asked to add feature X. Here are 25 questions that impact feature Z, B, and C in your gigantic codebase"
Developer: I can answer 8 of these questions... Guess I need to go figure out the rest of them.
Writing the code + building the functionality has always been the easy part.
Yeah, exactly my experience. Especially ever since Fable came out, I felt paralysed, many times, about how many decisions I have to hand it. Which is 10x harder when you handed it the reins to build the PR and you practically only knew the compressed initial problem statement. Then it stands up a draft PR and says: "Decisions you owe me: [insert bulleted list with 8 items each a paragraph long]" and you are like: "...oh shit".
It becomes a very heavy and difficult exercise of it walking you through the implementation and the judgment calls it had to make along the way. VERY exhausting.
There's a silver lining though: you do get to gradually clarify a proto project spec and various requirements, but boy does it take time and energy to re-contextualise when the bot tells you that you should make decisions.
first you’d need to define how to define/measure the complexity of code when it comes to this case.
I hopped into this hoping it would land where it did! I recently built in basic observations about cyclomatic complexity, churn, and authorship into an 'analysis' view of codebases in ouijit[1].
Some benefits can be reaped immediately (cyclomatic complexity), but others emerge over time (churn). A good example of this is say something like a 500 line file that has experienced 2500 lines of churn, and if that rate of churn is trending down or up. Surprisingly useful for understanding you've got a hotspot with an opportunity to pay down debt by spending more time on API design, or just breaking out whatever subset of the code is experiencing thrash.
The funny thing about complexity is that assuming you're lintting/formatting well, you can do a poor mans check by just looking at something like average indentation per line, deepest line, etc.
[1]: https://ouijit.com
Number of iterations solved correctly, on a very large set of iterations, seems like a very good metric. Better than anything else because it measures what we actually care about, not some proxy.
The only caveat is that it's the same model doing an iteration and then using that iteration as a starting point for the next step. So the model is allowed to write absolutely insane solutions, as long as it can read them back, even if no one else can.
One thing that could be done is to use a separately developed baseline coding model B to evaluate the outcome of each iteration. For model under test X to pass an iteration, not only should it be able to solve starting from the previous solution, but so should B, starting from X's previous solution.
Coding just a stage in the software development. Design and specifications which can help in coding is not solved at all and may never - the end result is software reliability is not a solved problem.
Sounds right to me. As AI has picked up, I have found myself drawn to resources that (I hope) help improve my taste and judgment not only of code but also of overall software design. The lessons in Code Complete, for example, have been helpful to me in thinking about managing the complexity of a codebase. It was useful before, too, but perhaps even more so now.
It's a nice article that basically (rage)baits the readers before they realize the author actually disagrees with the premise of coding being solved. So here goes, my higher-level rant on "solving" something with AI.
I strongly dislike all the "X is solved" narratives. What does it mean for something to be solved? A math problem (or any kind of problem), a riddle, a mystery, a dispute. Those are all instances of a particular situation that requires a "solution", but new situations will always come up. I understand that by "cancer/coding/X is solved" in this new age of (gen)AI that means the ability to streamline or speed up the "solution finding" procedure, but even that presupposes a fixed, static, fully deterministic space of the things we are trying to "solve".
Even cancer cannot be fully solved - Demis Hassabis slowly started drifting away from using the word "diseases can be solved" because they can't be eliminated - we can only speed up the process for finding a cure for any particular disease, be it existing or new and/or evolving.
Is bridge-building solved? Architecture? Why are architects still employed? Is solving "civil engineering" or designing an optimally running machine a thing to be solved? What are we trying to do when we talk about "progress with AI"? Even when the "recursively-self-improving-AIs" and "perfect" robots do arrive, we're still bound to work with them, and they'll have to evolve to find new solutions to new problems.
To be clear, I work with and rely on LLMs every day, from coding custom RAG architectures with CC and Pi to research and agentic data science. These bombastic conversations, however need to quiet down a bit so we can get back to work :)
Ya, now that I have some solid AI coding experience under my belt, there does seem to be some gaps between practice and reality. I have a fairly complex codebase which I pretty much hand code everything. When I add a new feature, I spend a lot of time designing and refactoring that feature into the codebase. Either the feature dovetails into the existing design or the feature creates new designs which will then facilitate even better future features. When AI approaches the feature, it just plows the feature in, and with bugs since it has trouble fully understanding the total design. So over time, you have a spaghetti design where you just have a whole bunch of features tied together with no unified design. I guess thats ok if AI is supporting it, but you now have a large cost and bug surface area and an insane human learning curve. So nothing has really changed here, we have been dealing with low quality codebases way before AI came along. I think AI has mastered the one shot single feature, tool, or simple app, but it struggles with the design complexity of a rich multi feature application or system.
I liken it to contractors or overseas engineers. They don't really care about the long-term effects of their code.
I does the thing, adds some code, adds test that cover that code, and responds with "Done."
At a fully agentic "No looking at code" company, we now don't have a great understanding of that code, we don't know whether it's a good implementation or not, whether it fits our current patterns, is maintainable, or composable.
We're entirely in the dark, at the behest of an agent at this point - if we want to maintain velocity. (And I would love to know whether that velocity is real or just perceived). It feels like the code part is faster? But also feels like I have to spend a lot more time up front working through a problem to understand it. In the past, I gained that understanding WHILE I was working through the code. Who knows.
I encountered bugs created by AI. No matter how many times it tried it could not fix the bug, it was introducing so much slop to work around the issue.
I finally gave up and read documentation for 15 minutes and solved the problem.
I will never push AI generated code to production without understanding it, and this is why I only generate small code snippets and copy/paste most of the time.
I was bit disappointed how few metrics the article mentioned. There are tons of code quality metrics that have been thought of over the last 40 years. We don’t have a good idea which ones are worth enforcing though. And we don’t know if the metrics that are good for humans are also good for LLMs
https://dekobon.github.io/big-code-analysis/metrics.html
https://dekobon.github.io/big-code-analysis/metrics-vcs.html
>In my research and tests simply taking the change in the number of LOCs has been a surprisingly effective metric for sloppiness, with the ironic caveat that if we started optimizing for it, it would cease to be a meaningful measure.
This matches my experience. Before working on an issue, I ask the LLM to estimate net LOCs at the final PR based on the scope. It works well, and review steps do flag inconsistencies. But as the OP mentioned, if you turn this into a hard metric vs "design smell", you can see LLMs code-golfing for oneliners.
I've had some success with tokens as a measure of complexity instead of number of lines, but should be combined with additional rules, e.g. disallowing lambdas, exec, eval, compile, __import__ and complex list comprehensions for Python. Fortunately, Python's "ast" module makes this quite easy.
If coding is ever solved, and if software does it, sloppiness probably won’t matter much.
Code will become throwaway stuff, like the results of AI prompts. Cook it up, test it for adequacy, and run it. When something comes along that adds new requirements, just update the requirements/prompt and make a new one.
Test suites will be important.
Goodharts law is invoked here with no discussion. Are we sure that optimizing for minimum LOC that pass all necessary tests produces sloppy code? And: What is sloppy code anyway? If we could define it, could we throw the definition into context and tell the LLM to avoid it?
> could we throw the definition into context and tell the LLM to avoid it?
Every IMPORTANT RULE WHICH MUST BE FOLLOWED STRICTLY to not to do something from my AGENTS.md and skills are violated without any hesitation from LLM. Glorified autocomplete doesn't work this way. Positive rules are more stable on this regard.
> What is sloppy code anyway?
Code that is overly and unnecessarily difficult and cumbersome for humans to work on.
But given the way humans work on code nowadays doesn't involve actually writing it themselves, there's no such thing as sloppy code anymore. Hooray!
I love this train of thought. Code quality is critical, but I don’t think we’re correctly evaling it at the moment. If we could get solid benchmarks measuring the quality of generated code, we might see the models climb those benches fast.
I believe that the era of “ai writes tons of slop code” will be a stepping stone in the longer story, and is simply a current gap in the reward functions.
Per the author - if we can get strong measurements of what good code is, we can train against it and close the gap fast. Excited to see more thinking in this area
If you could easily benchmark the quality of code then models would be trained on these benchmarks/metrics.
that is true, but if the metric is what we want optimized, then that's fine.
However it is more likely to be something which can be detached..
Code quality is probably isomorphic to the halting problem, or can be reduced to the halting problem in the simplest case. I.e. it’s intractable.
Sooooo much of what is considered "code quality" today is irrelevant when robots are writing the code. We've been largely optimizing for things like composability/unit testability in the past 15 or so years, and that's primarily a human concern that's unrelated to the final output.
Totally agreed that we're not looking at the correct metric right now. Increasingly, code quality will be determined by outcomes.
> that's primarily a human concern that's unrelated to the final output.
It's an LLM concern too. LLMs seem to do better with well-organized codebases, just like the humans they were trained on.
I don’t agree with this. The things people care about with code quality if you really think about it actually match up surprisingly well with the metric which models are trained to emulate in pre training, namely compression and modularity. Those two ideas actually seem to be universal to intelligent systems. Writing verbose highly coupled code is I think provably stupid, though I don’t know that I could formalize it.
I think Martin F's team tied outcomes to factoring https://martinfowler.com/articles/exploring-gen-ai/refactori... which is crucial to code quality. Even grug brain developer agree.
On the other hand, there are claims that the best languages for robots tend to be terse (allegedly); I tend to regard a rich domain model (static types, not terse) as a major facet of high-quality code.
> Qwen2.5-Coder-3B
Basing it's findings of LLM as judge on this model, and then proceeding to ignore it. This article can be safely ignored as well.
LLM as judge in harness evals is the way to go, for any of your custom needs. Design the eval well.
I don't know where you pulled that from, its not in the article.
Follow the paper from the LLM as judge section.
Is coding "solved?" I've been using Fable and Grok to vibecode a library to improve text extraction for legal PDFs. (Which have things like line numbers in transcripts that messes up text extraction.) This is a simple, dumb problem that should be like killing a mouse with a nuke. But I got to "pretty good" with a few iterations of not looking at the code or design. The program works, pretty well. But the design and heuristics are only 80% of the way there. Now, the models are just "churning" on the existing design, making ad hoc changes struggling to get to the last 20%. It seems like I'm going to have to turn my brain back on and think about what I actually want to do, which I was trying to avoid.
It's better than writing assembly! But it still seems to be in the realm of "a new kind of tool" even for the actual coding part, without even caring about what the code looks like or security or anything like that.
Without knowing details of your approach, I would venture that your challenge is not with the coding per se but extracting structured data from PDFs. It’s a surprisingly hard problem because PDFs are optimized for preserving the visual structure and layout of the content for precise rendering and printing… NOT for preserving the logical structure of the data!
Which is why the best results these days for extracting structured data from PDFs is by having the model do it directly rather than writing code to do it. It literally takes that level of intelligence to be reliable at it.
A common approach is to provide the model with a template or structured schema describing the format you want the data in, and the PDF itself, and it should return a JSON with the appropriate values filled in. It won’t be 100% but probably higher than what you’re seeing now.
The use case is basically the same ones where a programmer would use grep on source files, and it’s too slow to run the model on it directly for the same reasons.
Existing tools do text extraction (pdftotext) but not that great when there’s internal numbering and stuff like that. It has to be a heuristic approach, and it won’t be perfect. But it’s an interesting data point that the model isn’t able to figure out the best heuristic itself. Or even make incremental progress towards one after a certain point.
I gave a blanket ban on pdftotext to my agents. The output can get so mangled that a smart human wouldn’t untangle it. Did you try understanding the output from pdftotext yourself?
My approach is just ocr-ing with Terra or Gemini flash + checking citations with source both ways. But if I wanted to avoid llm calls, I’d just tell Fable to build a pdf reader directly from pdf binary format. Should be way more robust.
A PDF is a command stream designed for rendering. Interpreting the command stream to get the positions of each glyph is deterministic and existing libraries (I use both pdf_oxide and lopdf) do that fine. Once you have glyph positions, you need to use various heuristics to reconstruct words, paragraphs, columns, headers and footers, etc. For example, in a patent document, there's two columns with a gutter in the middle of line numbers. If you interpret the document as having a single line, you'll get numbers mixed up with the text, which can throw off efforts to find particular phrases. PDF builders also insert all sorts of weird crap into the OCR layers that has to get normalized out.
It's just a pretty pedestrian data-munging problem where there's no closed form perfect solution and you have to use various heuristics to get the right result.
Yes, but the edge cases are infinite and so heuristics don't scale well. As an example, at some point you would likely find yourself with "dueling" heuristics, forcing you to tune them, which is brittle, or find yet another heuristic as a tie-breaker, which ratchets up the complexity. (I just spent a lot of time on an adjacent but much simpler problem before finally giving up on churning heuristics!)
As an example, many times it is impossible to determine the order of some words from just position data without considering the meanings of those words. This is why LLMs / VLMs are so much better at this task, because they can look at the document holistically like we can.
Also, funny that you mention patents, something I've worked on in the past as well! If you're looking only at US Patents, the USPTO data resource is much, much better: https://data.uspto.gov/home -- they provide the text in XML format (https://www.uspto.gov/learning-and-resources/xml-resources) which is also pretty complex but wayyyy easier to parse than PDFs!
> This is why LLMs / VLMs are so much better at this task, because they can look at the document holistically like we can.
Totally agreed. But in this use case, PDFs are the working format, not just an archival format. An offline batch process to ingest the PDFs isn’t feasible. Unless there are some super fast LLMs I’m not aware of that can handle tens of PDF pages per second. It seems like Grok and Claude don’t try to read the PDF directly, they use pdftotext or some Python wrapper over pdfium. But maybe I’m missing something!
>It's just a pretty pedestrian data-munging problem
In other words, how complicated could it be?
Well, if Adobe has been introducing complications and making the format brittle and inflexible over the last 35 years to make it hard for its competitors to write software to process PDF files, quite complicated.
Yeah that’s why heuristics should work on the lowest possible layer, not on pdftotext. If you use pdftotext you’re stripping positional data and other stuff.
Do you use a public set of documents? I bet I could almost oneshot this with my harness :p
Yes, the tool I’m describing works on what the pdf_oxide crate returns, which includes glyph positions.
Here’s a public appendix from a recent Federal Circuit case. It has a representative assortment of documents (opinions, briefs, patents, transcripts) but contains only cited pages to the appendix pagination is non-consecutive: https://www.courtlistener.com/docket/68048163/15/ecofactor-i...
The underlying docket is usually 100 to 1,000 times larger than this but it’s similar types of files and the appendix shows the various types of headers and footers that can exist.
Here’s the oneshot, not sure if it’s slop or not though :)
https://kolinko.eu/pdf-reading-order/
But I wonder about your opinion.
I had Claude evaluate my vibe-coded extractor against yours on the EcoFactor corpus, which I didn't test on, and an internal corpus I have:
> The EcoFactor result inverts. There, on hand-authored gold over 21 pages of clean native text, pdfgeo won the accuracy checks 25/42 to 19/42 — its geometry work on multi-column patents, table cells and timestamp lanes is genuinely better, and nothing here contradicts that. But this corpus is 21,652 pages of mostly scanned, OCR'd, skewed exhibits, and at that scale pdfgeo's robustness is the binding constraint: it loses 6.4% of the corpus to two trivial crashes and an over-strict rotation test, duplicates text on 34 files, and drops 80% of wrap hyphens. lawpdf extracted every page, 7× faster, with 2.5× less furniture leakage.
It looks like the actual geometric analysis came out the same general shape, which is an ad hoc heuristic with hard-coded values attempting to figure out where the gutter is located, etc. Seems like both have some overfitting to the text data set.
If you have more test data I can try this with my harness - you can email me at kolinko@gmail.com btw :)
Thanks for the benchmark, I was seriously looking forward to it! Would you consider such even results good for a one-shot? I wonder how well my harness performs :)
What harness? And why not just Fable+Opus?
Personally I use only one family for code editing, other families for code navigation.
Models tend to reason about the code using their own intuition and putting other families onto the same codebases may end up with them getting confused. Esp if that other model is as unhinged as Grok.
Also, harness matters a ton. Use a wrong harness with a good model and you’ll get terrible results.
The conclusion is not surprising really, because fundamentally how do you even quantify sloppiness, a famously broad and subjective characterization?
I worked for years in Dev Productivity with engineers who had spent their entire careers in that field, and code quality was always the biggest "unquantifiable". Any of the metrics in the literature (cyclomatic complexity, erosion, etc.) quickly became very noisy at scale. Conversely, for any given metric you would find countless bits of code that do NOT exceed any metric thresholds but were clearly low quality.
People have experimented with many things over many years at Big Tech scale, which produces prodigious volumes of code daily. The conventional wisdom was "Don't bother trying to measure code quality."
An interesting observation from an ex-colleague is that probably the best measure of code quality is its comprehensibility, or "understandability". Maintainability, stability and adaptability are natural outcomes of that. But understanding lies entirely in the mind of the beholder! Which is why it's such a subjective metric, not amenable to simple mechanistic measures.
But now, we probably do have a technology that demonstrates some analog of comprehension: LLMs!
Specifically: tokens. Anecdotally and empirically (based on industry reports like DORA and DX etc.) AI coding works much better with "good codebases" (more specifically, strong engineering discipline) than otherwise. I wonder if that can be parlayed into a quantifiable metric like "tokens to grok / LoC" somehow.
So, if to fix something we need to first measure it, and if AI can measure slop, the way to fix slop from AI may be... more AI!
It will be fun to try and deterministically define sloppiness relative to a tool that is not deterministic.
Of course, sloppiness to date can be measured by different shared and interpreted preferences and definitions.
I just ran this on a personal repo I've been working on for two years. It's 250k loc of Python, that for the last year has been written with agents.
I told Claude to look at this article and the original research paper and calculate the numbers. Verbosity came out to 0.08 (lower than human, that's weird), erosion to 0.495, so between humans and agents. So according to these metrics, I have managed to contain the complexity of code.
are you able to share what you're working on that's 250k LoC?
It's a life tracker, that has come to include everything but the kitchen sync (journal, workouts, food, Garmin stats, music, podcasts, books. Etc.)
"To come back to the point of why agents can’t (really) deal with the slop themselves, we need to look at the evaluation of SlopCodeBench. In contrast to other coding benchmarks, which give the agent a complete list of instructions at the start and then have a set of hidden tests the program needs to pass, they do the opposite. They create multiple rounds of instruction and test iterations, where in between checkpoints the context of the models is erased. Thereby mimicking much more closely an iterative process, like how coding agents are actually used by humans. The result of that is that bad coding decisions accumulate over time and for the strict solve rate, where all tests have to be passed at all checkpoints, even state of the art models achieve 0% pass rate"
I like how this captures with a metric (lines of code and cyclomatic complexity, some other basically tractable measures) in an automated way, something we all by now intuitively know.
> I was disappointed at how “vibes based” the industry seems at the moment.
Alan Kay called programming "Pop Culture" some 20 years ago[0]
[0] https://queue.acm.org/doi/10.1145/1039511.1039523
A good measure is communication. If there's a common understanding, then the origin doesn't necessarily matter.
I wonder how much of this is due to the AI tooling being taught on sloppy code that humans have written. Over the last 4 decades I've looked at a lot of code on the Internet and there's a lot of slop out there.
It is not clear to me how the verbosity metrics works. Can someone shades more light on it?
It's a bunch of rules flagging specific patterns that could be expressed more succinctly: https://github.com/gabeorlanski/scb-check/tree/a8618228939de...
The problem is there is no good metric to describe code quality, it cannot be RLd and that's likely why latest models have such problems with slop. It is great to raise the conversation so that labs focus on this more.
Picking specific metrics will probably not work, it would be a mix of Goodhart's law with Bitter lesson. Maybe picking and labeling quality repos, having whole suite of metrics as input features and training some traditional AI classifiers to steer the LLM training.
He used LOC, and it isn't bad. Just this week I had a LLM do a small task, and it produced 500 LOC, every little detail beautifully abstracted out. But 500 LOC for such a simple change looked suspicious to me. As everything must pass human review, I re-wrote it to see what happened. The result was 100 LOC. No human wants to review 5 times a much code.
The issue really isn't "there is no good metric" - when I saw 100 LOC vs 500 LOC for the same thing there was no argument. The problem is Goodhart's Law. Whenever we use a metric use like LOC as a reward function for humans, the result is a disaster. I have no doubt that's true for LLM's too.
I hope this is not too provocative, but where I work the legacy code is such a spaghetti mess and so actively inefficient (queries in loops of loops and plenty of things like this) that just prompt nowaday llm to "write extensive and menignful tests for this module, then rewrite the module clean mvc small functions , small loc per files, well organized files then make pass tests again" often leads to insanely better ai "sloppy" resulting code, plus free test coverage where nothing existed, users instant satisfaction (software become suddenly responsive in less than a second).
From this point it take not much more time to add proper ci/cd suite, rework database models and migrate things with now some comfidence.
It works very well on small isolated projects and new builds where resulting tools are produced insanely faster than before and with not comparable average code quality (ai slop is 100x better here) from customer usual habits and expectations.
It's because this approach raise significantly the quality level so quickly and so cheap that I advocate for it.
It however does not work the same way for large legacy projects to revive. For larger projects the reciepe is almost the same but requires significantly more manual testing on each pass and carefull regression check. But it does work.
Customer is happy, they take back "more" control over their legacy horrors at the cost of some extra loc but properly working code. There is no point to fight against "a bit too much code" in my case
I don't want to speak for those that work in elite very sensitive human responsibilities software domains, but in a lot of place this is a revolution in my opinion.
The most surprising indicator of messiness I encountered wasn't in the code itself, but in the tests. I had two tests that looked like tests but didn't actually measure anything—they simply returned True — and neither static analysis nor coverage tools caught them. The only thing that detected them was mutation testing—deliberately removing a line of code to see if the test fails; without that step, the test is essentially toothless. Also, I found that scores became inflated if you counted cases where a test passed despite the removal of code (because another test caught the issue) as a successful "detection."
Well... Coding was never the most part of work I spent time.
Congrats on your promotion to full time jira management!
very glib comment.
thinking about how to accurately describe the problem at hand and figuring out the simplest way to approach it takes a lot of effort well before you get to the implementation step of coding.
Exactaly. Some tasks took me hours or days to change a line.
the website of this company is using more resources than their main product is (would really appreciate a button to turn of the animation)
Before AI, most of the code was already crap and slop.
After AI, the volume of that crap and slop has increased exponentially.
I'm not shocked the top 2 comments are "coding is not solved"
Pure cope, at least in most domains. The machines can do it better than the humans.
I think most arguments against that are clinging onto old things that don't really matter any more, like coding style.
If most of us are not coding by hand at all anymore, just directing and iterating with the agents, then coding is solved by a reasonable definition.
Solving consciousness ;)
Sad as it might sound, I think we might have to stop worrying about the code.
Nice! It's interesting to see quantitative ways of measuring slop. I'm curious what _would_ happen if you did plug these back into the LLM as a form of feedback. Goodharting may happen... or it could get the LLM to generate cleaner code potentially?
> LLMs are able to write almost perfectly correct code.
This is kinda vague. Correct at what scale? I wonder if there's a measurement on the correctness per scale, and hopefully the scale is not just CLOC.
Recent PR I had to review...
PR content:
``` Lots of AI slop.... .... .... Note: this will not build due to XYZ .... .... More AI slop .... .... End of PR ```
So the dev hadn't even read the PR comment himself and had blindly posted it!
I have no idea where most people writing code have worked at but in all product and platform teams I worked at the code quality has been much higher than the latest slop SOTA llms can output.
TLDR: coding is not solved.
I have 2 projects, one it's a distributed platform, the other one is a general processing engine with an inner workflow engine; Since gpt 5.2 I've tried new models to work in these codebases where the code is of good quality and every time I gave the model a slice of work instead of a single step from that slice the code, the tests, the comments, the docs and everything else has been suboptimal, unmaintainable, complex, bloated and just slop, unless I micro-manage and do many passes.
As a dev when you make a change you consider the broad picture, you consider the user, the codebase, future requirements, maintainability, performance, your team's understanding and some of these you do unconsciously. We are slow but that's for multiple good reasons, you push the organization/understanding forward not just loc of that specific project. I can't count how many PR notes or comments I've added considering teammates or just for a specific team member.
I don't see any way forward for an LLM to reach that unless it reaches general problem solving, my definition of GAI that could tackle software development or "coding" would be a model that doesn't require additional pretraining to solve new tasks or improve how it solves tasks in the future, it would just learn as it's going.
Can everything I mentioned be solved with current generation of LLMs and lot's of markdown and gates? Maybe... but the amount of effort required would be similar to the effort an expert system (pre-llm AI) would require to embed the rules, evolve them, check them everytime... which would require billions or trillions of tokens.
---
off: I really like the discussions around how to prevent slop and bloated code as it's something it would benefit coding even without LLMs and can fit as another piece of automated infra for checking and ensuring code quality, I hope something materializes.
Plopping in my email to the author below in case anyone else is interested in this kind of thing:
> There are some promising other directions I want to explore, such as coupledness of functions, code churn, cohesion and so on. If you are working on evals and would like to talk, I would be happy to do that: sebastian@earendil.com
Hey Sebastian, I just read your article and it thoroughly resonated with me. I've been working on building something similar to SlopCodeBench, but moreso aimed in the direction of architecture, rather than simple one off "code search functions".
In a nutshell, I'm creating multiple domains of common software architectypes. You can think of these as being as simple as a counter (very common in all architectural explorations worth their weight), todo applications, etc and as complex as an online store, a bank, a wallet, a social communication platform, etc.
Given a single domain, we can extrapolate common functionality that is "higher order" to that domain. Features like data synchronization, functioning offline, sharing information, authentication and authorization, etc all land in this bucket. From a single domain (take the counter, for example), I've laid out my initial plans for the various different levels to concretely observe how bad LLMs are at churn, cyclomatic complexity, poor abstraction planning, etc as follows:
L1: Show a number on the screen L2: Allow a user to click plus or minus and the number responds accordingly L3: Show that number on any device running your software, and keep it in sync with all other devices L4: Ensure that additions and decrements to the count, while a device is offline, will replay to all connected devices once connectivity is reestablished. L5: Introduce the ability to reset the count to zero, and ensure that commutes properly if an offline device triggers a reset or vice versa. L6: Introduce user accounts where I must provide an authorization strategy (login with apple, google, passkey, etc) and now segregate a public counter (which anyone can interact with) from your counter (which only devices authorized with your credentials can interact with. L7: Introduce an action menu whereby I can invoke what is commonly known as a "Command K" menu for the actions that can be performed in the application. L8: Allow me to speak naturally to your application and ask it to "go up", "start over", etc (this exercises your architecture's ability to be "accessible" to agents (and vision disabled folks as well) ... ... ... And so on and so on.
Despite its apparent initial complexity, naive solutions to even the simplest domains will be easy to spot with how many lines were changed vs raw additions (how composable a solution is), that cyclomatic complexity measure you mentioned, how many tokens it took, how many platforms (iOS, android, cli, TUI, react native, react) your application will run on, how long it takes to build, how large the binaries are, how much memory is used during the operation of your software, how semantically similar duplicated code across different platforms etc. From all these different values, we can create a hueristical "architecture score" to benchmark against.
I'm also toying with the idea of enforcing that one must submit a bundle of skills, instructions, scripts, etc that I will exercise with my own harness whereby the submitter has to submit a monetary cost with their submission that a budgeting agent must manage the spend and the agents must yield prior to their submission being scored, lest they forfeit the submission.
I'm just quite tired off all the hype and its exhausting and AFAIK, none of the benchmarks actually produce anything of use. One interesting side effect of aligning incentives in the way I've laid out here is that we will have produced open source, connected software that functions well and solves a whole bunch of business needs that all compose together by definition.
Thanks again for your article, would love to have an e-coffee and chat about if there's potential to collaborate on anything here. Despite how powerful llms are, this is still proving to be a tricky endeavor for me.
Best, Michael
PS: here is a demo of my submission for this first round: https://x.com/technoplato/status/2090902061437030777?s=20
That's a very interesting idea for a benchmark!
Thank you! My favorite bit (theoretically) would be the forcing function to produce useful open source goods as a result of running these benchmarks and quiet down all the hype about llms. We get it, they’re very good at making 3d models do dumb shit that doesn’t matter
Only frauds/vibecoders/non-programmers claim coding is "solved". This is nonsense.
Claim: AI writes almost perfect code.
Reality: earandil.com uses 170% CPU in Firefox.
What has this author written before LLMs? Why should we listen to him and his adjudication of "perfect code"?
Cyclomatic complexity is the oldest paper generating grift for college students. There are hundreds of thousands of useless papers about cyclomatic complexity.
There is some sense of rose-tinted glasses of pre-LLM coding. A lot of human written code, particularly at the enterprise level, was of low quality well before AI automated it.
I'm not so sure that's a fair comparison.
So much "bad" enterprise code evolved into that state over years or even decades of small changes. Meanwhile, last year I got to watch an LLM-authored codebase speedrun itself into a similar state in only a couple months. And I would say that the enterprise code was actually better. It at least did its job fairly reliably. The LLM codebase was riddled with defects, so much so that it ate up all our time and our feature delivery rate ground to a halt.
There are two observations that really eat at me:
1. Studies seem to indicate that agentic coding uses 2-10x as many lines of code to accomplish the same task.
2. One of the only really well-established empirical results in software engineering is the strong association between LOC and defect rate.
This is happening all over the place right now. There is a ton of greenfield happening, which further adds to the illusion of speed. Eventually you produce a big old pile of shit that even with the help of the LLM is weird to reason about, and it slows way down. Many such cases.
Writing code at enterprise level is insanely difficult. You are constrained by budget, staff, legacy databases/environments, business rules hiding all over the place, and people.
You can't just rewrite everything. So over many years people are touching small parts of the pie.
If it works it isn't low quality.
Yeah, enterprise code has that trope of being enterprise-y, verbose and bad. In my experience, that has always been the opposite. At the big corps/FAANGs I worked at, a single line of change can adversely impact millions of paying customers, so a lot of the verbosity and harnesses exists to dampen the failure modes. Most of the terribly written stuff has always been at startups, where devs fling nearly anything across the finish line, if it barely works the happy path.
> If it works it isn't low quality.
then the same applies to AI written code?
The code I have seen and see still, started shitty.
I don't doubt that. But humans still need to be responsible for understanding what they're shipping. And IMO you get your best understanding by actually writing some code. Even if you don't actually ship what you wrote.
Let’s not romanticize it too much... A lot of enterprise systems are built by developers copying an old AbstractBeanFactoryFactory from a 2011 stack overflow thread without really understanding it :)
Nobody is. It's the AI slop which is supposed to replace this shit which barely worked with equally shit shit which doesnt work which people are romanticizing.
Most of the human written code was slop, but the really fundamental and successful stuff we relied upon and which we didnt want to throw away? yeah, not so much. most of that was actually really good.
those EJB monstrosities were routinely swapped out by some saas written in python by somebody who did it properly and werent responsible for a lot of late and over budget projects which barely worked or didnt work.
And which, in turn, was used to train LLMs without any guidance as to whether it's "good" or "bad" code.
> But humans still need to be responsible for understanding what they're shipping
I don't necessarily disagree. That said...
Why?
I've been grappling with this myself. There is an easy/obvious answer, but I wonder how stable/permanent it is. If you feel strongly about this, are you willing to unpack your judgement?
>Why?
We are overly reliant on technology today. I don't see this trend slowing down in the future.
I think that a world where everyone is reliant on technology, and nobody understands how it works is a nightmare.
Certainly the population of people that need to know how something works is much smaller.
But even still, cognitive debt is a real, documented phenomenon where we lose our ability to change projects.[1]
We also know coding agents tend to accumulate slop in projects over time without some human intervention[2]
There's also the concern of whether its a good idea to be dependent on an addictive slot machine run by a trillion dollar company to do your work.
1 - https://simonwillison.net/2026/Feb/15/cognitive-debt/
2 - https://www.scbench.ai/
Yeah... like we all get to start green field projects and write all the code we should understand. Many of us cut our teeth on bad legacy stuff with no proper documentation made by "engineers" long gone. At least a LLM can easily make sense of this mess.
Indeed. And not fair comparisons ”look at the quality of this small one-shot Claude hobby project. The quality is less than this major open source project written by some of the best developers in the world”
To be fair the pitch has frequently been that Devin/Claude/Astra/whatever is some sort of superhuman bottled John Carmack that will single-handedly replace entire teams of developers.
Yea, that extreme side exists too. Truth is inbetween. AI with the instructions from a dev that knows what it is doing writes better code than most regular 9-17 devs.
And it can -- if an able person wields it.
1. People didn’t wear that as a badge of honour though.
2. A lot of it wasn’t. Low quality code/speed serves a purpose for point solutions and scripts etc. That’s not the same thing as writing a core system and if the user doesn’t put any credentials in for an S3 bucket then it falls back to giving information about your own S3 bucket (as I’ve seen just this week).
3. Plenty of companies you can discern the difference between mission critical systems versus “business” systems where if it falls over it’s annoying but not the end of the world.
> A lot of human written code, particularly at the enterprise level, was of low quality well before AI automated it.
its just now the same humans weaponized by AI ship 20x more of low quality code.
This was always due to pressures by management and the company environment, not the workers themselves. It's hard to blame the people writing code when they have to deal with nontechnical leadership that wants to have a feature factory or never given appropriate resources to solve problems.
Blaming workers is always an excuse by poor management.
The pressures from management and the company environment are not always a bad thing. It really depends on whether the pressures are coming from a logical business perspective or whether they are just coming from stupidity or ignorance. In a business environment, taking a long time to ship great code can mean that the company goes out of business, and then the software developers have a lot of great code and no income.
I'm sorry but pressuring workers over this stuff is frankly disgusting and I reject you for your anti-worker + anti-human sentiment.
What sort of human looks at a group of people and goes "it would be really nice if we made people miserable."
Absolutely disgusting.
Wait. I didn't say anything about making people miserable. And I'm not talking about some weird kind of pressure like yelling at people.
I just mean the normal, almost inevitable kind of pressure that comes from the business situation. Management has to somehow figure out strategies to handle the pressure and it has to communicate these strategies and the reasoning for them to the engineers in a constructive way.
Generally, this is the pressure to be competitive and make money. It does the developers no good if they spend so much time writing great code that the company goes out of business.
Many people were and still are simply terrible developers.
Hand those people an LLM and they don’t suddenly become competent, but they do start slinging more code.
Nah, you're still blaming workers and not leadership. If leadership is okay with not training workers (something American corporations would do in the distant past) then it's not fair to continue to blame workers when leadership is clearly aware of the problem and would rather pocket the money than help workers.
These companies pay management more than workers for a reason, if you can't even admit that they are to blame then what are you trying to do here? Just attack workers for what reason exactly? Being anti-worker is a great tell to never trust a person.
We're talking about professionals here. People who (at least in the US) often make several multiples of the median worker. Competence is assumed, and every company I've been at has had programs to pay for additional school if the employee wants it. IIRC at least one had an explicit book allowance, and I don't doubt that I could expense books right now if I asked. Do surgeons and lawyers complain so regularly that management doesn't train them? Or are software engineers just this desperate to be seen as "not a real professional"?
Who even is supposed to be training us? We're supposed to be the experts. Unless you mean mentorship, which is also generally already a thing at any company that has more than a handful of engineers.
There is no professional developer in the US. There are no licensing requirements to write code. There are no repercussions against developers that write code that immiserate or kill Americans. The person slinging wordpress plugins at an agency is equivalent to the person writing malware at Meta in the eyes of the government.
No one is training us because there are no regulations in our industry. Sorry but I still reject the premise of the other poster.
Also yes, in other industries professionals do complain when they aren't given their mandated time to learn while working on the job.
We are given time though. Maybe not at some small sweatshop? But at least at big companies all the jobs are salaried, and either no one cares how exactly you spend your time, or they explicitly say it's fine to spend some time learning, and will pay for materials or courses.
They're generally not going to teach you how to program or what the latest technologies are or what the best ways to do things are because that's literally what they're paying you for. But IME they definitely give you plenty of time and autonomy to be learning things at work.
I do blame management for letting these people through the interview process and then not firing them. But that’s independent of the fact they exist.
Training doesn’t solve every problem, the worst programmer I ever worked with that a PHD in computer science. Everything he made was horribly slow, wildlife overly complicated, and buggy. Worse he wouldn’t listen to anyone correcting his issues. He’d store numbers in the database as strings to be database agnostic etc.
Let me throw a curve ball at you: do you accept the premise that most modern corporations are centrally planned economies under the rulership of monarchies, oligarchies, or general authoritarians? If so, do you think introducing democracy into the workplace would help alleviate issues you care about?
You do not like bad workers, management doesn't care. They pay bad workers the same as you, bad workers can get promotions the same as you, you will also get laid off with the bad workers as well; or maybe even worse, the bad workers get promoted into management themselves. How do you want things to change in such an environment?
You have no authority to do anything meaningful as a single worker, what if you were given a voice to actually make these claims and have other workers decide what to do based on your voice?
Workplace democracy seems like an interesting concept to explore if you truly want to create better environments with beneficial outcomes to all, not just the few:
https://en.wikipedia.org/wiki/Workplace_democracy
Or are you just trying to bash your co-workers so you become the next pick me?
> do you accept the premise that most modern corporations are centrally planned economies under the rulership of monarchies, oligarchies, or general authoritarians?
No stock owners are ultimately in control of public companies. That doesn’t fit any of the models you just described.
Similarly companies are not independent of government control which inherently separates them from management systems associated with governments. A CEO is limited by the law in ways that a dictator isn’t.
I've only worked at companies with great leadership. This is in the Nordics with very strong worker protection. And yet most of my colleagues including myself have been pretty terrible and write dirty code. It's not a management issue and it's not anti-worker to acknowledge this fact.
Your great leadership doesn't seem to care, so either your management knows better than you or maybe you should push back on the notion that you worked with "great leadership."
Only poor leaders ignore their workers, which is what you seemed to have actually experience.
Sounds like you've only worked with brilliant people. Good for you but don't use your experience to spread these obviously delusional ideas.
I would argue the average code quality of LLM's today is much higher than pre-AI code quality. It's better documented, more readable, and has fewer bugs. There was a brief period where frontier models were still worse than the average developer, but that period among frontier models is well past us.
Are you saying that pure vibe coding by a non-technical person produces better code than pre-LLM developers, or that experienced dev + AI produces better code?
Both of those things are very different, and AI shouldn't be the one taking the credit if it's the second case.
That is very true and with llms someone who wrote low quality code can now output a whole lot more code. Maybe in better quality, maybe not.
Not "a lot of". "The majority of".
I've been doing development, in one way or another, since the 90s. I've worked with dozens of teams from enterprises to startups. Hundreds of developers. The quality of work has been all over the place, but the majority was not great.
I'm arguing that what people today call "AI slop" is already higher quality than what most developers created historically, and the fact that tests and documentation pretty much come for free now means that the floor has been raised.
The quality of AI generated code is not great. Yes, it will get better. It's already better than 65%+ of what regular devs can do AND it is faster to produce, iterate, and release.
This is off-topic, but I strongly dislike AI written documentation.
When I see AI house style my eyes glaze over. Just this morning I reviewed an RFC from a colleague that he said was a spec for a web service. The document had no introduction, no context, it described endpoints for 2 distinctly different services instead of 1, and made no effort to reconcile why there are 2. It was scattershot with details, some of them important, some completely irrelevant. It was replete with typical LLM-ism.
Basically, it was a dump of a conversation he had with an LLM. As a document to build shared knowledge, it was nearly useless. The only feedback I could provide was a polite "I do not understand what you are trying to build".
But, supposedly, another engineer is already working on implementing this spec. I assume the other engineer just cycled this "spec" into his LLM, and off the two of them went. \o/
They are trying to pull me into their project right now, I stood up some containerization infra for them. But, oh boy, do I not want to join. I looked over their codebase, by LOC the codebase is 35% comments, and a lot of the comments are contradictory, there are dependencies that are not used, there is no tooling of any kind (no type checking, no linting, no PR process), there is no auth (this code is already running in production lol -- they have public endpoints exposed that can be used to scrape/mutate internal company data). Another 30-40% of the codebase is unit tests that test trivial stuff like whether their framework's serializers and ORM work, ex: x=DB.create_x(arg=1), assert(x.arg == 1).
At the intuitive level, I do not understand people who say coding is solved... To me it seems like LLMs are a multiplier (LLMs are amazing, sci-fi level shit), but if you multiply a negative number or 0, you get something that is <=0. Making agentic coding work requires a lot of discipline & expertise.
Oh man, the "unit tests" that test whether the framework/browser/language is doing what it's supposed to drive me insane. Those have their own tests already! Test the unit under test, that's why it's called that!
AI documentation is practically worthless ime. I forbid it in my projects. It's almost always more useful to not have any documentation and read the code than to rely on AI docs.
Testing though I'll mostly give you
Coding is not solved, correctness is not a feature, it is the bare minimum. If your code does not do what it is supposed to do, you could as well have no code at all. Efficiency, security, maintainability, reliability, readability, understandability, extensibility, maintainability, observability, portability, ... this is what high quality coding is about, not that it works, that is a given. And in my experience current models are pretty bad at this.
> Efficiency, security, maintainability, reliability, readability, understandability, extensibility, maintainability, observability, portability
My experience is a little different. For higher abstraction languages the output is largely acceptable in my work. I always consider that LLMs don't know what I don't tell them and they have limited context to work from. Coding issues I often identify:
* Efficiency. Marginal by default. Coding efficiency problems often appear because LLMs dont usually consider the entire codebase or future plans (although they do guess at some futures). Sometimes they write/name things in ways that are lazy/wasted cycles. Most of the time, they don't.
* Security. Marginal by default. I say they do pretty good. Considering all the failure modes, not so much.
* Maintainability. Marginal by default. Mostly due to the careful consideration of modularity, upgrade paths, etc. while often taking wildly different approaches to solutions without having specific broad instructions. Even then, there can be big gaps in quality.
* Observability. Not acceptable by default. There's usually some consideration and can often one-shot.
* Portability. Not acceptable by default. Good, if you specify what those targets are. Regardless, testing validates this above the coding and models are very good at hitting functional test targets. This is less of an issue in something like Java ofc.
This sounds roughly right to me, except for "maintainability". In my experience, agents really don't like deleting code unless you explicitly ask for it. If you're not careful, you end up with new better implementations of things but with the old implementation still around in perpetuity. Humans do this too of course.
They're being trained by a lot of repetitions of one-shot bugfixes and feature implementations. If they don't have to maintain code over the long term they have no incentive to not take the easy win with lower immediate risk and higher longer-term maintenance burden.
The shadow of Chesterton's Fence is relevant. The LLM doesn't know why code exists, if it doesnt start traversing up the scope of a project. Even then, it can't be sure that the code isn't a dependency of something else outside the project (especially if there's a side effect). I'm not sure it's ever going to be easy to address this concern in a straightforward and portable way.
I do sometimes see duplicate functions, which is troubling.
Humans had to get it drilled into them that "+12 -440" is a damn good line stat, and that keeping around dead code is bad, especially in the age of version control.
Not too surprised that LLMs also don't "get it" by default?
Yes. I use these models day in and day out, on all sorts of tasks. I cannot believe I hear people say that coding is solved.
Right. It feels like we're living in a parallel world or something. Can agents code? Sure. Can you let them code on their own for a serious production project? Not a chance.
Plenty of serious production projects are doing exactly that. Are you using GPT-6 Astra, or something older?
Which serious production projects have AI agents coding on their own? And I’m assuming that means they are routinely taking tasks and deploying them to production autonomously
You and the person you are replying to are talking about different things.
Agents cannot be given a high level goal and then left unsupervised, for hours, without making some dumb decisions.
Right. The only way I'd feel comfortable doing that is if I spent an inordinate amount of time writing very detailed specs, so in terms of labor offset I'm not sure the juice would even be worth the squeeze there. In other words, a sufficiently detailed spec is nearly indistinguishable from code.
What people seem to be wanting is for an agent to infer vast complex data from terse simple data, which I think is probably impossible on a philosophical level. There's real information loss in language, and compute can only make guesses at the end of the day. I really don't see how we bridge that gap.
That's exactly what I meant.
Astra does exactly the same sort of things that Sol or any of the previous agents do. They duplicate code, overengineer, miss the point, etc.
I was very optimistic about it when it was announced and saw all the demos, but a week later I find it only marginally better (and in some cases worse) than before.
Yeah, Astra, not like it matters. It will gleefully make db queries that perform full table scans instead of adding an index. I need to intuit that that's what's happening and then explicitly instruct it not to be negligently dumb.
Consider that the people who have never have never had to be on-call for a product that results in losses for the company in the event of an outage, and/or have never had to attend an RCA meeting have just as much of a voice in this discussion. Some segments of the software industry don't even have to worry about those sorts of things, and it's easy to get myopic after a while...I doubt that a long time dev at hobbyist-centric wearable manufacturers or audio plugin shops has the same sorts of concerns as someone who has specialized in ad delivery for social media sites.
Allowing only people who have served on call or made concrete contributions to multiple RCAs to have input on the software development process is a wonderful idea!
me too; and my coding agents are slowed down (from developing features) because I require them to refactor the code to be more readable; my code metrics tests force AIs to leverage functional programming and design patterns).
Yes agents can produce code that compiles and runs, but I had to add tools to keep them on track, document their work, follow a process, check their outputs. I also use other AIs to generate developer documentation and review code.
It is like managing a bunch of idiot savant eager-to-please interns, except unlike interns, coding agents do not (yet) learn and improve on their own.
Correctness is not a binary thing though.
I doubt many people here are brave enough to claim their code does what is supposed to do in every conceivable case. Maybe you have high confidence in the correctness of parts of the code. Correctness of an application is murky though. Things we build are never fully correct, merely correct enough. Like maybe you're responsible for the UI in a web app and you're using your expertise to ensure it gracefully handles display across browsers and a gamut of screen sizes/form factors. But are you also verifying how it works when localized with an RtL script? Are you checking every change you make against CJK?
Sure, every sufficiently large codebase will have bugs somewhere, but it will work correctly at least something like 99.999 % of the time after ironing out the bugs on the common code paths. But that does not change the fact that being [mostly] correct is the lowest bar you have to cross.
And there is another problem: LLMs generating too much code, code that is doing more than was asked. And that cannot be fixed by tests. Usually, we create tests for wanted behavior and expected exceptions. But we don't create tests for undesired behavior.
Code cost is almost down to zero. If you move the point of “just leave it to the machine” from the compiler (where humans used to do the coding) to the high level logic (now with LLMs) then in most cases more code does not really matter. Like, why build and maintain an abstraction where the LLM could implement this many times over each time with different subtleties? Why use a library with its own constraints when you could have exactly what you want? Why use cross platform frameworks when you can just one shot the thing to N different platforms? It’s not even slower. You can have code that’s larger yet more performant (stripping away abstractions can do that).
From time to time I try to do a pass of coalescing flows and cases and removing dead code to reduce the context and prevent the LLM from tripping over itself. But if it’s exclusively LLM maintained code I don’t care too much if there’s more of it.
The "exactly" part is the problem.
Just last two weeks I had to slap Fable, three times, to stop writing 1000-2000 lines of defensive code... because of DB columns I just forgot should be NOT NULL. That was it. Nothing else. I told it that, boom, -4800 coding lines: gone.
LLMs defend the status quo and they regularly lose sight of everything bigger than the current PR they are working on.
I too am gradually making peace with the fact that LLM-maintained code does not have to be 100% readable for humans.
But this is not about readability. It's about the data model. So one concession I am willing to make is: don't care too much about the code _BUT_ manually curate the data model. So far: small wins on iteration turns and code volume producing. Too early to tell but for now I am happy with the results.
Have you worked with Opus 5?
Its documentation about what the code does not do could fill whole books.
UI copy being full of slop explaining what the software does not do is another problem.
I am not convinced that a lack of negative test cases is an issue.
I do agree it generates too much code most of the time.
Documentation is important. I would say Opus' propensity to write documentation that documents non-features is part of the problem being discussed.
And the problem isn't just that it says what the software doesn't do, most of the things it claims are in fact meaningless, it's not even clearly describing something the software shouldn't do.
"Coding" is just a poor term for this, as there's so much room to weasel different meanings out of it. At every company I've worked in past 20+ years the "coders" were engineering a product from wrong or unclear requirements and specs from non-technical people. The act of coding was secondary (but an important throttling function to make us stop and think about what is even possible or makes sense.) Never did FAANGs, so YMMV.
Really doubt we are near that being solved with non-technical folks + LLMs. I'm seeing people gleefully rebuilding products with the exact same blind spots in their understanding/logic using LLMs. Claude, etc are not seemingly able to "AGI" around goofy asks. The CSS looks a little nicer than their legacy products though, lol.
second sentence of the article: "Just because the code is formally correct doesn’t mean that it is not introducing unnecessary abstractions, creating duplicates, or just making bad decisions overall. This is not a groundbreaking observation, most people who have vibe-coded a project, have realized that each additional feature can sometimes lead to an explosion of lines of code (LOC)."
I think this is a two things can be true situation, where our colloquial meaning of coding is not quite adequate to indicate the full range of criteria that really matter, but also, that criteria as a totality (maintainability, reliability etc) is something that can also be targeted and optimized for.
This reminds me a bit of a PhD Comics webcomic that confidently claimed we would "never" cure cancer, on the grounds that cancer is not one thing. And I don't know that we will ever actually cure cancer, but that wouldn't be the reason. Correctly noting the problem space is bigger than a layperson would initially appreciate is a lot of things, most of them helpful, but the one thing it's not is a formal a demonstration of optimizing against the problem space as a whole.
It's not solved, I agree. But if we pretend it is we can prepare for when it actually becomes solved, if ever, and measuring sloppiness is a worthy pursuit even if we never "solve code".
I just setup a large refactor with Astra and was feeling super lazy and let it mostly do it without my usual extreme micro managing.
The refactor ended up adding 22,000 loc.
I went in there and quickly read through it, laughed my ass off. Reverted the work tree. Micromanaged a new refactor. Net lines of code for something really elegant and easy to reason about was -3k loc in the project.
In case you are wondering why vibe coders are doing 30k loc a day, this is why.
This has been a problem in the software industry for a loooong time, though. Too many developers see "It works" as the end state, a signal that you're done. "It works" is just step 1.
Look, we are comparing against the average developer here And yes, they are obsolete
You come across as someone who has never worked on a real software project. Humans create tons of bugs on a regular basis. AI is already better than most programmers.
Better at writing one piece of code, maybe.
Better at writing code within a huge system, definitely not. Maybe in the future, but as of Astra, Fable 5.1, the answer is still no.
I'm definitely not experienced enough to know but I read people are having it write somewhat elaborate documentation beforehand. Have it figure out which parts of the code may be touched, what other things will be affected and which uncertainties it has. Basically a full report before giving permission for any code to be written/modified. Again, I haven't seen it, don't know how common this is nor how effective. Though it sounded interesting.
Disagree. I have 200k LOC now plus 100k in tests, and it is still performing like it was four months ago when I started to seriously use AI.
If anything, it works more reliably today with the smarter models.
I have worked my entire life as a professional software developer and I agree, even among senior developers I would guess [way] less then 10 % consistently produce high quality code. But if I have to decided whether I want to use an AI to help me write code, I does not matter if it can write better code then an unexperienced junior, it has to write better code than I would on my own.
I've found I can produce 10x more code than I could otherwise, of lower quality than I would otherwise, but the speedup is worth it. Extensive testing is what makes it work, with every bug becoming a red first test with a fix. High level compartmentalization keeps everything on track, you don't let it do the big picture architecture, but you let it do each component as decided on and work through the bugs later. I've seen much worse from teams of humans and I've accepted the drawbacks that are slowly going away with each new model.
> I've found I can produce 10x more code than I could otherwise, of lower quality than I would otherwise, but the speedup is worth it.
That is never worth it. You're ruining the software you work on when you do this.
> I've found I can produce 10x more code than I could otherwise, of lower quality than I would otherwise, but the speedup is worth it.
The speedup of slop production being “worth it” is what we, as a society, are having trouble evaluating at this point in time. In all likelihood it’s worth it only in the short term.
AI is the average of all programmers.
It's just that many (I guess that includes me? :D) assumed that they are better than the actually were.
A team of programmers is "worse" than their best member. Worse quality of code, worse thinking. You can usually get more done with a lot of mediocre hours of work than a few brilliant hours, though.
AI I've used isn't a better coder than I am - it's just got a lot more hours in an hour than I do.
Amen. I sometimes wonder if all programmers are now marketing people who know shit about software development and engineering.
That is very well put and summarizes what distinguishes real software development from vibe coding.
I’m pretty sure that same discourse was seen every mass production epoch, from textiles to electronics. Yet here we are, hand-crafted high quality things are rare and expensive.
> mass production epoch, from textiles to electronics
I see this comparison a lot, and I think it's a trap, because it invites us to confuse scaling duplicates with scaling design changes.
Duplicative mass-production was always core to software from the moment it first became "soft". A factory churning out 10,000 copies of the same book maps to 10,000 downloads of a single software release. The paper and bindings of the book may be below hand-crafted standards, but the words are largely unaffected.
In contrast, LLM-coding is the design and prototyping stage. So if we want to learn from textiles/electronics, we shouldn't be thinking of acres of looms, but instead about fashion-design, custom tailoring, determining patterns for clothes, designing new appliances, choosing circuit layouts, etc.
I have seen so much production code with garbage code and massive bugs, the industry doesn't care for 'real'.
They care in principle, for the most part bad code makes changes and extensions slow to implement and causes unnecessary production issues which costs time and money. But there is always the tension between implementing something quickly now and being able to implement things quickly in the future and unfortunately the preference is almost always quickly now despite everyone knowing that this is the way more expensive choice in the long run.
It isn't necessarily the more expensive choice in the long run. Let's say that companies A and B are direct competitors who start from the same point. Company A quickly codes a bunch of buggy software and ships it in a month. Company B takes its time and ships good, clean, well-organized, mostly bug-free code in five months. Company A makes money. Company B goes out of business and its code is useless, its nice qualities irrelevant.
This is just a hypothetical example, I'm not saying that this is how it would necessarily go in all cases.
Not even in principle. Like how often you had to fight a product manager to do this or that.
"Does that code work?" "yes" "so lets ship it" "but its not good" "but it works right?"
But they also ask why the next feature is taking so long, because of the mess we created with the last feature, and I would consider this caring in principle. Unfortunately this almost always filed as unavoidable.
Yeah, the author leads with a blatantly false claim that models are almost perfect at generating code. They are not. They write pretty bad code, worse than any capable human would.
And all of these things will be solved one by one.
It's astounding to me that people can see coding get solved and not think every single one of these tasks won't be solved too.
Why do you not think these things aren't going to be completely automated? What makes these tasks special?
Fable and Astra can one-shot video games with compelling novel game loops. They can do systems programming, distributed systems, robotics. I haven't found a weak point.
Seedance 2.5 can make video better than the manual labor of VFX artists, 3D artists, and animators.
Nano Banana and GPT Image can do a better job than graphics designers.
LLMs just solved a Millennium Prize Problem, and there are probably more that will fall in the coming weeks.
Just wait. All of these things will be solved.
There is no "stopping point".
Edit:
Don't anticipate that 2036 will look anything like 2026.
Will Smith spaghetti doesn't stay that way forever. Trillions of dollars will be spent on solving these problems. They will be solved.
May the iterative loop of adding new axes to evaluate on be a natural, healthy progression, instead of needing to frame it as an us-them problem?
If you value humans intrinsically, this is necessarily the loop that will converge. I don't think humans have deep intensional a priori knowledge of the structure of reality. If we did, then we wouldn't need tools like AI because we'd be a superset of that. We can only observe and judge.
If we don't value humans, then sure, I think AI is at the point where it can kill all humans (conditional on sentience and resources etc). Two ways to solve a problem - solve the problem, or eliminate the problem statement. Plenty of easier vectors to eliminate the "problem statement", than say, try to solve problems such as making human life better. If you do value the latter though, there will necessarily be human judgers. That's how it works.
I think you need to define what solved means and what better means.
Have you tried one-shotting real distributed systems problems? What was the result and how did you verify correctness?
Like the 0.1% we do is your counter example?
But lets be fair, if an expert would use AI today to build something with this, I would feel a lot more confident than not doing this.
I would start with the base architecture and add all the guardrails for a distributed system, i might even go so far to leverage the math skills of a frontier model like fable or astra. I would for sure have the proper budget for using Fable/Astra.
> It's astounding to me that people can see coding get solved...
We haven't seen that. Maybe when we do, we will start to believe that other things will get solved.
I have no doubt that AI will eventually be able to write essentially perfect code, I am just saying that we are still quite far away from that point.
> compelling novel game loops
Tell that to the mountain of failed AI slop games on Steam! As a game dev, building compelling, fun games is not even something humans are good at doing consistently. The AI can build the tech, but it can't make something 'fun' yet (unless your bar for fun is simply that a tool created a thing).
I think the usage of solved is silly. Things get automated, they basically never get ‘solved’.
Will all these things get automation? Yeah sure. But the idea that they will be perfect automated solutions applicable in all cases is just marketing, it’s not reality.
..
People are ego-centric and pull a blindfold over their eyes.
[dead]
[flagged]
Personal attacks will get you banned here. No more of this, please, regardless of how wrong someone is or you feel they are.
https://news.ycombinator.com/newsguidelines.html
[flagged]
Coding might be "solved" but coding still is the best way to build your own mental model of the solution space.
Which is more important to you: Velocity to a solution? Or velocity to understanding?
Interesting way to lay it out. For us understanding is obviously crucial for prod and repeatable business functions.
Velocity to solution is default for almost everyone else, especially one-off or low impact / low consequence of failure projects.
Velocity to solution is default for senior management, that's for sure ;)
I code to make money, and the kind of stuff I work on doesn't kill people or lose massive amounts of money if it has bugs, so to me velocity to a solution is much more important than velocity to understanding.
Velocity to a paycheck
Coding is solved, but AI companies are still hiring software engineers?
Are they really though? And if so, how much of that is due to the hyper growth in this specific space?
Meanwhile, the SWE job market across all companies seems pretty rough right now. Talk to someone looking for a job. Most companies seem to be in a holding pattern - little to no new SWE positions available.
The job market sucks for absolutely everyone right now, it's not a SWE thing. Fwiw I've had five recruiters reach out to me in the last four months and I'm not even applying places.
It's a far cry from 2021-2022 but I'm not convinced it's worse for SWEs today than it is for everyone else
That's almost every industry, because most companies realize we are an extremely unstable point in time.
So they still might do but in parallel the software engineering market in china and india is collapsing.
Source?
Lots of chatter online, i have seen a few videos on this tpic on YT.
https://www.hays-china.cn/en/press-release/content/%E4%B8%AD...
I think he's referring to contracting companies. Cheap software used to be outsourced to India, but it's now getting outsourced to AI. Companies that produce novel things are still hiring, often to manage agents and review output.
Best explanation I’ve heard is that “coding is solved but software engineering hasn’t”.
* slop engineering
Coding != software development
Is there any evidence of this?
[dead]
It will be solved when there is no more code left to write.
Code is an abstract concept that is not bound to the physical world and I imagine that future will have some much more of it that it is difficult to comprehend. Everything will be code and more code will be written than ever before.
Code will never going to be solved. The question is how much humans will be involved and I think the evidence is that perhaps just a bit. However, because we are talking about vast libraries of code even if we are involved in under 1% of all code and decision making that is needed around the code, there are still not enough developers out there to take on the task.
I might be wrong :)
Code and software and applications are an intermediate stage. The final stage is an AI/LLM that just does the thing that is needed without any code being written, there are no applications or programs, just an AI that does everything.
So how will separate systems communicate with each other? Or are there no separate systems? That’s the digital singularity I suppose.
This doesn’t scale at all for some problems where a specific algorithm is needed (graph problems for instance).
AI needs tools as much as humans do.
Claude do the needful.
I'm pretty sure we reached peak software developer jobs due to ai.
The market is already collapsing in China and India.
Coding is not solved. It's only solved when coding becomes something you do because you want to, just because you like doing it, the same way I bake bread at home because I like doing it, not because I have to. Right now we still have to be hands-on - to a lesser extent, yes - but we still have to review and hand holding AI agents to get things done.
[flagged]
[dead]