I think there's a paradox. Is it only "cognitive surrender" if you already know how to code? Not knowing all the details seems more like the default human condition, in the sense that most people don't know exactly where their water comes from or where their waste goes exactly. "Water comes from the tap, food comes from the store".
Most of the non-technical people I see using LLMs to create software are not engaging in "cognitive surrender" as far as I can tell.
They didn't read the code before and they don't now. However, asking questions and improving understanding of the system and architecture is actually faster and easier for them now than it used to be.
They're evaluating what they can see, i.e, judging results. They build things for their own use. They test them, iterate on them to meet their needs and try to achieve their vision. This is roughly the same process they followed when they used to outsource to an agency except the loop is much tighter.
It seems to work well, mainly hurts when the problems are invisible, so security, backups, breaking migrations, perf, maintenance over time (language / runtime updates, dep updates, server updates etc). All the stuff that hurts if you have never been responsible for the uptime of something before. Not knowing that software is never "done". But junior devs trip on those exact same things.
I agree with the tail of the article; tooling and processes that make that stuff more visible would close a lot of the gaps.
> It seems to work well, mainly hurts when the problems are invisible, so security, backups, breaking migrations, perf, maintenance over time (language / runtime updates, dep updates, server updates etc). All the stuff that hurts if you have never been responsible for the uptime of something before. Not knowing that software is never "done". But junior devs trip on those exact same things.
If we continue from your "water comes from the tap" example this situation feels like someone who has no experience of plumbing deciding they're going to replace their bathroom without bothering to find out how any of it works. They'll possibly end up with a bathroom that superficially works but they're also liable to find problems down the line with leaks, things not draining properly, or water pressure issues.
Some of the people writing software purely via LLMs are even turning this into a business, which I guess would be analogous to deciding you're going to run sewage treatment for a small city without understanding anything about how that works. It'll mainly hurt when the problems are invisible, so food poisoning, or inlets backing up and overflowing into a nearby river.
That analogy breaks down immediately because it's missing the critical novel component: the AI.
In the bathroom remodel analogy, the AI is the plumber that you hire to do the work for you. You tell it what you want, it figures out the details necessary to get it to you. If what you want isn't possible, or requires large changes, a good plumber will raise the issue to you and explain the constraints they are working under, ask for guidance, and offer recommendations.
Sometimes the AI does things in a weird or inefficient way, or neglects details that become important later, or misunderstands something important about the task resulting in an outcome other than what you wanted, but that's also the same as the plumber. This is why people usually want to be at home and keep an eye on the people working on their improvement projects or repairs, because they want to make sure that the guy they hired is replacing the broken faucet, not the functional but odd-looking one.
There is still value in knowing the underlying systems well enough to present a good design vision to the plumber up-front. It's still possible to have a competent executor produce a bad idea, which is how we get stuff like Groverhaus, the Doom Bathroom, and McMansion Hell, which are examples of "slop architecture" in exactly the same way as we are seeing "slop software".
There were terrible software developers before LLMs. Honestly for a lot of people AI generated slop is a massive improvement over what they used to ship.
I agree it's pretty bloody cheeky to run an agency like that though. The only ways out I can see are licensing or real liability.
But let he who has never done dodgy home improvements cast the first stone.
I think one of the potentially advantageous things about AI is that it acts as a force multiplier for people who already have expertise in a given discipline, while simultaneously raising the floor for everyone else.
People looking for self-contained, spot solutions to localized problems will have much more capacity to solve them locally, while people with in-depth expertise will be able to build systems that are even more reliable, performant, and scalable than what they were building before.
So we'll get the benefits of disintermediation for the basics, and much greater capacity to deal with effectively with complexity for the not-so-basics.
> However, asking questions and improving understanding of the system and architecture is actually faster and easier for them now than it used to be.
One of the things I've built in my current role is tooling to help understand current architecture, and to resolve the differences between things as-documented, as-implemented, and as-actually-behaving. It's used for a few things, including updating documentation, but it's been most impactful when onboarding new engineers. They can simply ask questions about how things work and why, and get explanations and deep dives wherever they need it.
> Is it only "cognitive surrender" if you already know how to code?
Is the article claiming that?
> Not knowing all the details seems more like the default human condition, in the sense that most people don't know exactly where their water comes from or where their waste goes exactly. "Water comes from the tap, food comes from the store".
Sure. The ability to specialize, and on the flip-side not have to care about a lot of things, remains one of the biggest driving forces of human civilization. But in this article, I don't think the person worried about cognitive surrender is the "most people" in relation to food or water. They're the farmer or the water infrastructure engineer! They absolutely for sure do need to know those things. And all around us we see a lure of systems that invite them to surrender.
> Most of the non-technical people I see using LLMs to create software are not engaging in "cognitive surrender" as far as I can tell.
Maybe not. But the might be cognitively surrendering when they use the LLM to summarize the day's news or write a message to a friend, or even plan dinner for the week. I'd say they are. I agree that it's not clear-cut.
> They're evaluating what they can see, i.e, judging results. They build things for their own use. They test them, iterate on them to meet their needs and try to achieve their vision.
No. They have things built, then they test them. This is different. It might be OK, but it's not the same as building.
> This is roughly the same process they followed when they used to outsource to an agency except the loop is much tighter.
Yes. But they wouldn't usually outsource summarizing a letter from grandma, or the political agenda of a candidate/party in an upcoming election. That's surrender.
> It seems to work well, mainly hurts when the problems are invisible, so security, backups, breaking migrations, perf, maintenance over time (language / runtime updates, dep updates, lib updates, server updates etc).
… or ones own abilities deteriorate.
> Tooling and processes that make that stuff more visible would close a lot of the gaps.
> I don't think the person worried about cognitive surrender is the "most people" in relation to food or water. They're the farmer or the water infrastructure engineer!
Fair. Maybe I "cognitively surrendered" a long time ago just as a consequence of working in large teams :(
I hit the breaking point of not being able to read all the code well before LLMs came along. Also people would leave and the documentation for their subsystems would be wrong or incomplete, so there were always mysteries to chase down (during an incident, naturally) even in code written by hand.
Gets to the point, you don't exactly read the MRs, you're mainly checking their approach against what you talked about earlier and making sure it doesn't look way more gnarly than it's supposed to.
Working with agents doesn't feel that different.
> No. They have things built, then they test them. This is different. It might be OK, but it's not the same as building.
It does bug me sometimes when people say "yeah we built the deck..." and they got a contractor to do it. But then we do say things like "Nolan made the film" even though he didn't hold a camera or build the sets. To me, one-shotting doesn't feel like building, but engaging in a loop and applying control to get the outcome you want, does. Regardless of the exact toolset.
I don't personally see super hard lines between using a compiler, a 4GL, a sql query, stitching together 10 libraries and a framework I didn't write, using an LLM or even directing a team.
> Not the problem of one's own skills though.
I read somewhere once we can't weld Saturn V engines anymore, or at least not economically. We would also probably have a lot of trouble finding people who could do all the engine calcs the way they did. It doesn't necessarily mean that rocket progress is stalled.
> Yes. But they wouldn't usually outsource summarizing a letter from grandma, or the political agenda of a candidate/party in an upcoming election. That's surrender.
> But then we do say things like "Nolan made the film" even though he didn't hold a camera or build the sets.
I do say that, too. But I'm not in the film industry. My understanding is that those who are, don't use words as crude as "made" to describe a role in the making of a film. I believe they say that he wrote, directed or produced a film.
We are not talking about some other field of expertise here, where we are outsiders – the production of food, the supply of water, or the making of movies – we're talking about our own field of expertise.
> My understanding is that those who are, don’t use words as crude as “made” to describe a role in the making of a film.
Oh, no, film insiders use ‘made’ constantly, or even cruder - attribute credit without using a verb at all. Saying it’s a “Nolan film” is super common. Here’s a VFX person saying “Nolan’s CGI”: https://youtu.be/AFLcP-t0-kI (I’m ex-film industry, and still film industry tangential, and found this clip being discussed on a film industry mailing list.)
Film people are far more likely to pay attention to who the director is than non-film people, and it’s very common for film people to talk in terms of directors ‘making’ the film or otherwise casually assign ownership. I think this is especially true for famous directors and good films, and maybe a bit less true for unknown directors and mediocre films.
Somewhat off topic now, but Nolan is fairly famous for being intricately involved with every aspect of the films he directs. He's constantly behind a camera checking out what shots are going to look like, pushing the limits of where you can put an IMAX camera, and coordinating with set builders to do ridiculous stuff like building a reproduction harbour jetty.
The tendency to over-engineer and over-build is one of the main challenges I'm seeing. When it comes for free, why not build a class hierarchy of 8 classes each with a 200 line unit test? Something I might have previously thought really hard to minimalise down to 30 lines is now bloated out to hundreds or even thousands.
Previously, pretty much everyone all agreed on YAGNI and KISS because code was so expensive to create and more expensive to maintain. Now the code generation is "free" - but at a design level, YAGNI and KISS do still matter. But they are enforced now only by sheer will power. You have to actively restrain the code from being created if you don't want it. Perfectly good code - I can't even fault it. I just know, if it doesn't need to exist yet, it's a mistake to create it earlier than needed.
That quotation is in fact from "Wind, Sand, and Stars", the memoir of the same author.
On the same theme:
> the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience
- Einstein
> I would have written a shorter letter, but did not have the time
- Pascal
> Simplicity is a great virtue but requires hard work to achieve and education to appreciate, and, to make matters worse, complexity sells better.
That's always been my concern - there's just so much garbage code out there and very few signals about which of it is well written and which is not ... so LLMs will just regurgitate the good enough code that it's been trained on.
Now would be the ideal time for everybody to give it negative feedback when bad code is returned but I'm afraid not enough people care and we'll just settle with what works just like we always have.
I don't think that's it. The context that LLMs work with is simply too small, and can't get bigger. Harnesses and the models themselves are bolting structure over it manually; real people are inventing design machines that are a lot more deterministic than they look, and LLMs are just filling them in.
The design machines aren't good enough yet; and their progress has little relationship to LLMs at all. Designing computer programs and the proper level of abstraction for the easiest maintenance, flexibility and debugging is simply an open problem. There's nothing to train LLMs on that is smaller than the size of an entire program, and its history of code changes.
Get a few of those, figure out a way to analyze and reduce the patterns and lessons in them, and train on that maybe. Until then, there's no way for an LLM to figure out what lacking a particular abstraction is going to end up costing.
At the end of the day your work is your work. If you tell Claude or whatever to do a thing it will do that thing however you tell it to do the thing.
You can choose to ignore the fine details and just tell it what to build, or you can spend more time and effort and tell it how to do things, you can ask it to create a plan, review the plan and ask for changes etc, you can review the implementation and ask for changes there. You can simply tell it you don't like how a feature is implemented and ask it for suggestions on how to improve it, include what you don't like about it so it has direction.
It's all about the effort you put in. LLMs are able to write clean beautiful and efficient code, they just can't necessarily do exactly what you want on the first try without any direction. That's why you should figure out what you want, ask for what you want and make sure it's actually building what you want.
This is so true. I've been seeing LLM generated code that does way too much, yet still cant accomplish the stated goals. "maximally flexible" plugin systems, that are too complicated to reasonably write a plugin for quickly, when one could have just built in the needed functionality in a few hundred lines.
When endless bootstrapping and framing is free, you end up generating the most beautiful scaffold while ignoring the building behind the scaffold.
I think this sorta cuts both ways too. Sometimes AI code has shown me that no, I in fact don't need that big class hierarchy and speculative plumbing that I thought I might need to get the job done.
Keeping the volume of code down is definitely a challenge though.
I have been thinking about this for a while. I'm not a Luddite, I can see the amazing value that modern AI brings, and how useful it can be.
At the same time I can't help but wonder that this is almost too much for us to cope, the LLMs are _too_ easy to use. I think we're very good at pretending there isn't a problem here but there is definitely going to be some cognitive decline happening.
We want to pretend like reading over the output of an LLM is nearly the same as us making those changes ourselves but I think there's a lot of value in those boring and manual steps. And most people (including me) will be too lazy to not eventually offload most things to the LLMs.
In a way this is similar to how watching a lecture tends to give one an "impression" of understanding that doesn't correlate with reality. We need hands-on work and some course work to get our monkey brains engaged.
I can only imagine what effects this is going to have in education.
We've surrendered many uses of our cognitive power throughout the centuries while replacing by others.
Our cognition didn't evolve to produce arguments for online discussions, but for making crude tools and hunting. As we stopped applying our intelligence towards these uses as a result of our tools, we simply shifted it to new problems that were unthinkable previously.
A caveman probably wouldn't care too much about algebra, a topic most children learn today.
We should stop thinking about the problems AI trivialized and, under the assumption that AI won't fully replace human capability (a key assumption), start thinking about the problems that were previously out of reach. Education would then start focusing on that.
Right, but algebra is free. You can do it with a pencil and paper
AI is not free. Even "free" and open source models require complex silicon to operate. The silicon and copper is assembled in complex fabrication plants. Those plants are very close to geopolitical belligerents
What I resist is the dependence. I'm not willing to trade short term productivity, for long term dependence
This is true, but the nature of declining cognition is to not even notice what used to be possible.
Taking clothing for example, there's more available now than ever. It's disposable at great cost to the environment and the workers that produce it, but very little to you.
Yet, I bet you cannot evaluate garments, construction, or fabric quality. You may not know how to wash woolens and linens. You may have only worn smelly plastic workout clothing and plastic suits for formal events.
You may never have worn a piece of clothing that actually fit your body.
You may not realize that you can alter any clothing, that most clothing was intended to be altered once upon a time. Or that older garments were both more expensive and higher quality, because the expectation was the most important items would need to last for a decade.
The Luddite's central objections about automation was about declining working conditions (objecting to child labour, for instance), bad pay, and declining quality. They used the 'obnoxious' machines they broke. They saw what was being lost and demanded better (without demanding no machines at all).
Later, the rise of unionization brought about many of the changes they'd asked for decades before.
I hope this current technologic movement doesn't end where fast fashion has, with influencers buying a new outfit [repo] every day for a year. A vortex of undifferentiated slop for its own sake.
When I'm working on my own, I scrutinize every line of backend code and frequently send the LLM back for rework. Frontend code I keep an eye on just to make sure the LLM isn't getting stupid (which it occasionally does, even at the frontier), but mostly defer. The CSS I don't even look at.
This isn't too different from having employees. How much do I review other people's work? Depends on the employee, depends on how critical the work is.
Great post. I particularly like the 4 column pr view idea. I’ve built something similar to review speckit specs. The scaffolding to ensure that code actually works makes sense too.
There’s one thing that’s bothering me though — all of these have us consuming what LLMs produce. I learned and internalized how things worked by building systems. At least within the constraints of the abstractions I was using. This makes me think of TikTok and shorts. Everybody is just watching. Maybe that’s not charitable though. Maybe the abstraction has now changed from an “os/gui/event loop” to “llm” and the rest is just accidental complexity.
>> A more serious irony is that the automatic control system has been put in place because it can do the job better than the operator, yet the operator is being asked to monitor that it is working effectively.
Automatic control is good for standard case, and the expected errors. For weird cases, for now humans are better to detect them and press the big red button.
Jumping into article, I assumed that the author was going to describe how he enjoys, by drawing it out as far as he can,
the process of surrendering to the machine god..
(Not the machina diabol, as it sadly turned out to be, though some BDSM zealots might disagree.. the things that bureaucracy does to us are mild in comparison? )
But.. we do monitor whether our prayers are effective .. right? Is that a smitable offense?
After that disappointment, I too was "curious": was the author going to talk about "curiosity", that magic virtue that we hackers believe, lifts design into the divine (but more often than not kills the pussy)
The better an automatic control is, the fewer mistakes it makes, the fewer times the human needs to react, the more lax and less vigilant the human becomes -- and less skilled when they need to step in. Source: https://imgur.com/a/Vj2TaDz
Except experience shows they aren't. Experience shows that when humans are just pressing "ok" 99% of the time they'll press it 100% of the time and completely miss the other cases.
This is a matter of appropriate training and regular refreshing over time.
The best known example is the Japanese method of "Pointing and Calling/Naming" in their Train safety protocols but has been adapted to other industries too - https://en.wikipedia.org/wiki/Pointing_and_calling
These fall under "Behaviour Based Safety" (BBS) training methodologies and use techniques like "Check-backs", "Call-outs", "STAR" etc.
Maybe things will evolve into projects having a well-defined, hard boundary between agents generated code and human generated code. Where agents generated code is strictly separated from human engineered code and and can call only a strictly defined set of APIs designed by humans.
Think of platforms like Shopify. Shopify allows huge number of external users to write code customizing their web shops. But Shopify engineers do not need to review and understand the users' code, because there is a boundary which limits what the external code can do. The external code can be A Big Ball of Mud without making the whole platform a A Big Ball of Mud.
We are leaning into a kind of federated model. The core applications and vibe-coded satellites. The core apps have stricter requirements and carefully considered API guardrails. Go ham on the AI apps. If you break something, the scope of destruction is much more limited.
That said, the core apps are still full of AI code. I don't see a reality where hand-written code can or will be curated and sequestered. It's unenforceable, too slow, and too expensive.
I think so too. And it is how codebases are managed implicitly anyhow. You don’t let anyone PR against core functionality, you keep that API under lock and key.
But UI on top of a good API should be unable to cause problems; the API must prevent invalid state etc. So letting LLM’s write UI code for CRUD-apps (which is most of them…) seems fine to me.
Well. One of the things that's critical in all that analogy is that the BigBallOfMud that one user creates on Shopify cannot impact the other users - that's why Shopify can keep the platform from becoming ABigBallofMud.
In order to be able to enforce this hard boundary, it means that one "agent generated code" in one place, not only "can call a strictly defined set of APIs" - but, more importantly, it implements an isolated piece of functionality. Think something like: I let the AI write the dashboards, so the dashboards may be slow, crappy, incorrect, whatever - but it doesn't impact the rest of the product. However, I can't let AI write "metrics collection" because that not only impacts dashboard, but potentially impacts core product efficiency too (memory consumption, cpu, IO, etc.)
>Maybe things will evolve into projects having a well-defined, hard boundary between agents generated code and human generated code.
The only way things will evolve is towards maximizing AI generated code and minimizing human involvement because that is the most efficient mode for getting product to market. Any trend to the contrary is temporary, paperclip maximizer will maximize. Any code that humans touch directly will be considered implicitly too risky to run in any production environment.
We'll just have to hope things don't break too much and when they do, we'll just have to accept it because no human will be able to (or allowed to, or even trusted to) fix it.
i sometimes wonder how all carpenters or music producers in the world would react if tomorrow a magic box arrives where they type-in/dictate what they want and voila... out comes a table/composition.. Will the worlds music producers and carpenters just "accept" this new magic genie? I would hope they wont.
Yet here we are as software engineers...ready to surrender our craft to mere "prompts" with reckless abandon.
I see a bit of a split in people in creative roles of all kinds (and this is something I had noted before LLMs made the contrast more stark). some people get into it because they like the process of making the thing that they are making (in whole or in part). Others get into it because they have a thing that they want to exist in the world but no-one else has made it (I would also split this into people who just want the thing, or people who want to make something that crosses many domains and so want custom things that will fit into the whole). The former group tend to react with revulsion to automation, because it removes the part they like (though it may depend on what part is automated: most people will find some part of their work necessary but tedious). The latter are usually much more willing to embrace it, because it's the end result that they care about moreso than the process.
Have you seen how people use Fruity Loops nowadays? We’re already halfway there if you compare that to using an AKAI S9000 or MPC2000. And then go back to actually playing an instrument. It’s a gradient and we’re already pretty far along on it :)
Its just as much as an ego trip...as the Principal Engineers that roam around here, after they survived three layoffs and outputted front end code for six years straight at a FAANG...
That's new to me, I didn't know she coined the term. She's the first true "software engineer" then, I suppose. I remember her from that historic photo where she's standing next to a pile of code printed out, a stack talled than her.
Not to be confused with Margaret Hamilton the actress who played the Wicked Witch of the West in the film The Wizard of Oz.
It turns out that most people are fine with most of their things being mass-produced cheap stuff from IKEA, Amazon, Temu, etc. - if that means that the cost is significantly lower. The functional benefit of having a door is huge, while the aesthetic benefit of having a beautiful, hand-crafted door is much less important to most people.
Code is not like craftsmanship anyway. The end-user doesn't see any difference whether I use microservices or hexagonal design or react or htmx or how I structure the code or which language I use. All they see is the design and the stability. Since the average developer was terrible at design anyway, the average design output from LLMs is actually an improvement in most cases, even if it always has that generic AI generated look. Stability... was not good before AI either. We have become accustomed to SAP taking 10 seconds to load each component on the page, or Atlassian spewing error messages in your face when you click submit on a ticket. It seems that the aspects that are most valued by our end users are QA and UX, not the craftsmanship of our code.
Suno probably puts out better music than most folks who take up music. I mean statistically it has to, right? So once we go with "suno doesn't make music" then we also have to progress to "99 percent of human musicians don't make music", and all us amateur hacks will start objecting lol
Only in the same narrow way that a camera makes better art than a painter. Or the idea that a generic landscape print is better than a Rothko, because it looks more like other paintings that you like to look at.
The camera just records reality, its by definition sort of neutral artistically speaking, its what it records that matters. Its quite different from this AI music thing, and I would consider it a very difficult task to discuss topics with someone trying to pretend it isn't.
Exactly. The point of a painting is not to reproduce reality. The camera is good at reproducing reality (better than 99% of painters!), but the camera doesn't make art.
In the same way, Suno can produce music-like audio very well. It doesn't produce music. It may be possible for somebody wielding Suno to produce music, but the average use of it certainly isn't it. It's certainly worse at producing music than almost any musician.
It can churn out muzak like no tomorrow, though, same as ChatGPT can render you barely-tolerable images of any art style you ask for.
I mean more or less to an extent if you expand by what you mean by "out comes". With Amazon, a carpenter can type in any furniture he wants and have it arrive in a matter of hours. Almost equally as magically as well as disruptive. People still use carpenters to make tables, but for the majority, the ease of same-day delivery and cheaper prices are worth it for a sacrifice in personalization and character.
> Yet here we are as software engineers...ready to surrender our craft to mere "prompts" with reckless abandon.
It's worth remembering that, contrary to what HN acts like, software engineers are not uniformly accepting of LLMs. They are still quite controversial in the industry. Now management, on the other hand, is completely sold on the supposed benefits of LLMs. So many engineers feel forced to use them, but it's not enthusiasm on their part which drives them to that.
My experience has been the opposite, the HN crowd seems more sketical of vibe-coding and not reading a single line of code.
The twitter(x.com) and linkedin crowd has drank the cool-aid and are now swimming in it; Wild posts from FAANG engineers and managers saying they or their engineers dont read any of the code the LLM generates.
Music is already mostly a solved problem unfortunately. Try giving opus a few midi files of your favorite song. It'll write something new in that style with a competence humans would need several years of experience for.
"Yup, looks good to me!" said the human reviewing thousands of lines of generated code everyday.
This has become common now, and I've seen it ruin large projects until nobody understands the codebase full of verbose nonsensical code and comments. It becomes impossible to further develop the project without AI. Contributors not using AI for any reason are all leaving because no one enjoys working with mindless robots (and humans) as colleagues. It's demeaning and devaluing their labor and humanity. They're not getting "left behind", they are leaving the madness of crowds behind.
> ..the possibility of runaway dynamics: once a critical threshold is crossed, small increases in adoption can trigger rapid population-level shifts toward *persistent dependence, with abrupt losses in cognitive competence*.
We can all see it happening. As this technology permeates society, the challenge to maintain control, coherence, and quality will become greater and ever more serious - for example, AI in healthcare, military, management and governance.
Even if the human's judgement is correct in every LGTM, there's a bias towards merging. Each merge that adds a feature but degrades understandability 1% will be merged. It won't be held back until it improves it by 5% instead. After 100 merges you have degraded understandability 100% even though nobody made an error.
You’re completely right that nobody is stopping a PR that just ever so slightly decreases understandability of something. It may not even be obvious to the reviewer that it’s slightly less understandable overall… Doubly so if they’re just skimming due to the volume, and tripley so if they’re having agents do the most of the review, which lets the human reviewer’s guard down.
But before LLMs, it didn’t matter because on a good team you’d have a balance where maybe some PRs added complexity and some simplified it. Sometimes people would include a small related cleanup or refactor in a PR for code around what they were doing if they noticed complexity. That has become far less common, I think.
I know it’s not your point, but a 1% decrease 100 times is 0.99^100 = 0.366 which is a 63.4% decrease in understandability. Just throwing that out there
I hope there's still value to be found in a bunch of developers standing around a whiteboard and talking, and I think this scenario is where their ability and understanding really shows these days.
If we don't value that though--if we don't value understanding the system and just thinking through problems as a group--then let's stop pretending and just not do code reviews. Probably nobody cares and just types "LGTM" after having their own LLM do a spot check to cover their ass.
It's really easy to push developers to stop caring. If management says "go fast fast fast, use those LLMs!", then developers can't keep up, get frustrated, stop understanding the system, and at that point they're just there to collect a paycheck before society collapses. Or, if management is making dumb decisions all the time, same thing.
Hopefully as implementing ideas takes less and less time and resources, smart people who actually care will start building things with other smart people who care.
I've had someone approve my PRs within a minute of me opening them, and likewise expect immediate approvals on his own PRs. When asked questions about parts of the code sometimes the answer would be along the lines of "Claude did it this way", or verbose AI-generated responses to the question.
To be fair, models are more accurate than how they were earlier this year. In march I was still frustrated and was writing my own event loops because the model simply wasn't built for it. By January I was still hand checking statistical outputs to make sure it wasn't making any errors, and it was. After august it was pretty much over. There were no stats errors, code generated was shit but it could do what is asked. Instruction following is still a problem, and I sometimes catch silly things in the scrolling diffs, but it has been much easier since then.
The most difficult part of LLMs and agents seems to be - surprisingly - humans.
Specifically, humans and their dysfunctions. Their fragility. Their ego tied to stuff.
This is an excellent post/reflection on the topic of solo LLM use.
The only gap I see is that it assumes that one works on something alone and not in a team. Though, that might be a different essay in itself.
Edit: Turns out that footnote 3 actually mentions that already. Nice. Good post
On the ego-effect, I sometimes feel like cataloguing some PR-review quirks that are mostly caused by humans being humans:
1. Smaller PRs (that can be easily read) tend to receive more comments and nitpicks than a large change since it's easier for us to have all the PR context in our heads at the same time.
2. Most people are ego-tied to their code even if they pretend that they are
impartial
3. Social dynamics around merging and reviewing code between senior/junior and older/newer employees
Cognitive surrender is intellectually convenient. As humans, its instinctive for us to chose convenience: path of least resistance. The tyranny of convenience is what drives most economic activity today, especially the gig economy[1]. Unless there are structural frameworks put in place to resist convenience (exams in schools), nature will always entice to chose the path of least resistance
> Once you have used a washing machine, laundering clothes by hand seems irrational, even if it might be cheaper.
Washing clothes by machine is orders of magnitude cheaper than by hand when you factor in the value of your time.
> After you have experienced streaming television, waiting to see a show at a prescribed hour seems silly, even a little undignified.
Undignified? What? Completely incoherent word slop. Good to see LLMs haven't completely taken that domain for themselves, I guess - a little hand-crafted slop now and then is a treat, I guess? Anyways, they're wrong. People still tune in to watch scheduled live events all the time.
> To resist convenience — not to own a cellphone, not to use Google
Neither of these things are a matter of convenience. A cellphone is at this point literally required to participate in a modern economy. Google is a tool that provides you with globally-retrieved information that you otherwise literally could not find on your own or without another similar tool.
I don't even disagree with the premise that convenience is a driving factor of our lives, but the justifications and conclusions drawn by this article are beyond stupid. Here's my view of it: the desire for convenience is what pushes us onwards as a species. Hunting for food is inconvenient, so we invented agriculture. Going down to the river or well for water is inconvenient, so we invented indoor plumbing. Repeat ad infinitum until we have modern civilization. There is one part the pessimistic view overlooks; that to achieve maximum convenience long-term requires more effort in the short-term. Developing the skill of programming is inconvenient in the short-term but convenient in the long-term because of the possibilities this opens up to you. This remains true even in the current LLM era, and it will remain true for as long as we are not replaced wholesale by LLMs.
>Washing clothes by machine is orders of magnitude cheaper than by hand when you factor in the value of your time.
I'll also note that it is less resource intensive overall. Handwashing dishes and clothes will use more water and more energy to heat that water compared to the machines that do the same job (people often assume the opposite).
What an amazing article! It's really hard to find the balance between "ship faster, fix later because AI will help" and "efficient thoughtful engineering" these days. The world is kind of pushing everybody to work faster because "with AI you're more efficient" and it's also pushing to surrender and just accept it from time to time. Maybe that's occasional thoughtful surrender mentioned in the article, or it's just lack of energy to resist to it at that point.
The article is inspiring though, I have more motivation to work on my thinking process and my engineering setup (and vibe limit haha) to be more thoughtful, thank you.
1. LLM usage, in particular unsupervised, tends to build up "cognitive debt"
2. It tries to characterize the concept, including to propose a score for it
3. Notes that the level at which one can let the agents loose (and allow some CD) differs quite a lot between core business domains and more peripheral ones
I am often thinking recently of the littel loop diagrams from the "The Capability Trap" paper. We need to keep a loop reinvesting into our own Human capability in anything we're doing.
Making a quick prototype is probably the best way to find out if an idea is worth keeping. Vibe-coding it and then throwing it away seems like a good way to go?
Yeah, definitely. Also at times I've vibecoded successive iterations with better designs after seeing where the first ones collapsed -- the agents do a good job with a template and need much less prompting for the successive versions.
Maybe it's because I'm not working in the workforce (grad student), but I find my best workflow is to fight and argue with my agents throughout the entire process. Often times, since the agent is such a sycophant, I end up arguing with myself...I actually find this useful because it gets me to think about the other side of a problem and poke holes in it. This is not fast, but it is effective.
One aspect to resistance that is oft overlooked is knowing when to fail on purpose so you can succeed on a bigger issue elsewhere.
As an example, I’m juggling multiple projects at work and home. I have a finite amount of time, and access to AI. One camp would tell me to use AI for all of it to move faster, but that increases the risk of Cognitive Surrender. The other camp would tell me to do it all by hand, but I don’t have the time available for that; I must pick and choose.
So, I choose to cognitively surrender on the SEO task for our website because that’s a set of skills I don’t want to acquire and are better handled by AI anyhow. This leaves me more time to focus on learning more about Apple’s configuration profiles and improving our MDM suite, which are valuable skills that will still pay out in an AI-first world.
You have to pick and choose where to fight, otherwise you’ll expend all your energy on losses.
I simply don't care. As a student I thought that the mythical production code is perfect and error-free, but in reality 99% of systems are a hot potato game sprinkled with politics where you don't want the grenade to explode in your hands.
I mean, if the management doesn't see value in decent engineering, then who am I to butt myself into the conversation?
The primary reason you shouldn't surrender cognitively is that the truth matters. That's it.
Of course, when you're working on a system of high complexity - and one that is constantly changing to boot - with all sorts of historical reasons and what not that inform the why and so forth, you can't reasonably be expected to understand everything. This is what bracketing is for. When I acknowledge "this is the way X works", it doesn't mean I have to accept the proposition "this is the right way/best way to do X" or "X must be this way". A teacher or mentor may say it is, but in a theoretical context, unless it has been justified, you cannot claim that you know that what has been said is true. However, you can accept it as a provisional working hypothesis. And practically, it may work out and perhaps experience will lend some feeling of substance to the hypothesis, but you needn't make it doctrine.
Empirical science often has the same spirit. Perhaps the model isn't a true account of reality, but it puts handles on reality so that it can be instrumentalized.
There is an occupational danger here. If we focus exclusively on the instrumental, if we reduce all of human attitudes toward the world to use and utility, we not only vulgarize ourselves and our orientation toward the world and even others, but we risk trampling the truth and the good underfoot. We risk developing an indifference to the truth, because all that will matter is some measure of effectiveness. This is what one might call the textbook[0] definition of bullshit.
This is copium that is gone with the wind when the models perform up to the level of standard that the user requires. I'll give the author 6 months before they feel like its okay to just send it.
Karpathy had the same trajectory. Primeagean had the same trajectory. I think in the end everyone except for Jonathan Blow will have the same trajectory
Writing is hard, but what I was trying to say is that it's okay to just send it as long as you're doing it consciously and intentionally (and I do that frequently enough); but if it's not an explicit choice that can be problematic.
I've been experimenting with vibe-coding some games. One of the games started performing very badly in a particular menu, and I began suspecting it's because of a shader that may be redundantly computed way too much on too many large surfaces. I began writing my suspicion to Claude, then stopped and decided to just describe the problem and see if it would come to the same conclusion as me. A few minutes later, it had profiled the game, found what contributed most to the poor performance, fixed the rendering and made the game performant again. I didn't have to inspect the code to do any of that. The current models (paired with coding harnesses) love to test and validate their results by running the games, taking screenshots, singling out individual parts, including logs to reproduce errors or other issues, and they love to create tests. The code is good enough in the same sense that the code I used to write was good enough if it did what we wanted. My non-technical bosses didn't have to inspect my code for that.
In my experience, it's usually good for finding out the bugs, but I take the fixes it provides with a grain of salt. There were numerous times where the fix could have lead to some more serious bugs because of some business logic special case scenarios that it didn't understand.
Yeah, I'm seeing this at work too. It's capable of adjusting behavior to meet some narrow success criteria, but it can easily introduce bigger problems in the process in large coupled code bases, which it doesn't catch on its own. Automated tests help a bit, but I find I still have to steer it very strongly at work.
It sounds like you understood what was happening here and used a tool effectively, nice.
That's very different from what a lot of vibe-coders do, like wholesale cloning and relicensing projects. This is happening in the GIS space, and these vibe-clone projects slightly hilariously inherit all the technical debt they didn't understand.
Nothing wrong with using LLMs, but I agree with Andrej Karpathy: "you can outsource your thinking, but not your understanding".
I think that quote is very apt. I experimented with making realistic video game grass (inspired by Ghost of Tsushima), and let it take the reins. But its solution became very contrived and hard to steer in the direction I wanted without me learning the specifics of how grass rendering should work. Still, it seems that I can limit my understanding to the higher levels of abstractions in most cases now. For example, I made it clump grass together according to a voronoi diagram, yet I have no idea how the voronoi algorithm works.
Who cares if it's good enough? Send the product, if people use it, it's good enough. If not, let users debut it and give "feedback", problem solved. You then advocate to these users, make them feel important, make them your cash cows, rinse and repeat.
Every team I have been on have been more successful when they gave up on the product ideals and just ensured that the customers were hooked and satisfied.
Me! I care! I want to use the things I create and be able to fix them when they break without relying on an intermediary simply because I'm too lazy to learn. It gives me a feeling of contented self-actualization and overall satisfaction with my choices in life.
Perhaps that is overrated and I should focus more on exploiting people with crappy software so that I can make more money. I guess that's what life is all about.
Too bad we live in a society where goodwill, self-actualization and satisfaction does not ensure survival or pay bills, but rather gives you a competitive disadvantage.
That mindset is why most software was already slop before AI. Users know when your team has turned into salespeople. And now that AI could enable you to do better in the same timeframe the response with this mindset is of course to care even less.
> Who cares if it's good enough? Send the product, if people use it, it's good enough. If not, let users debut it and give "feedback", problem solved.
I care, because doing a sloppy job and seeing if people will tolerate it is not acceptable engineering practice. It's not engineering at all. More pragmatically, if you do the thing you are suggesting, you won't have users for much longer. They will abandon you and go to someone who actually makes good products.
I don't believe this in practice. I am at a FAANG-like firm, and we are just maxxing the engagement and people use the product more and more the more we serve stuff to keep them hooked using their data and behaviour.
The quality of what we provide seems to matter 0, we are up about 40% issues in prod year-to-year but the users and their engagement is also growing.
I think there is a small subset of technical nerd/idealists who think that quality is all that matters but they are in their own little sandbox while 97% of users are in another one.
> I think there is a small subset of technical nerd/idealists who think that quality is all that matters
This is just a derogatory comment.
It is exactly this kind of arrogance that is killing open source. In my area, GIS, there are foundational projects that are necessarily intricate. For example it might be necessary to make a four-dimensonal transform of a location, that takes into account point movement due to the Glacial Isostatic Adjustment. There are hundreds of other complex examples I could come up with.
The libraries are intricate and quality does matter.
Unfortunately the lynchpin maintainer has just gone on indefinite strike, partly because open-source vibe-coders are wholesale copying and relicensing some of these libraries in their quest to ... do what?
I actually I don't know what they want. Take all the oxygen from the room as far as I can tell.
Well I think both the maintainer and the people copying the software will realize that the code has 0 value, but the community and peoples attention has some value.
The oxygen on the room here are peoples ego. Use your tool and interact with your community while you have their attention.
> The quality of what we provide seems to matter 0
Good for you! There are lots of low skill development jobs before AI where low knowledge coders cranked out WordPress sites for marketing, advertising, and small businesses. Sounds like you got in on the ground floor!
Meanwhile companies like Amazon, one of the As in actual FAANG, do have demands on their software quality. To the extent they essentially said "knock off the tokenmaxxing productivity theater" in a recent, publicized internal memo.
Horses for courses. Sometimes you need an thoroughbred and sometimes a pack mule.
And there's nothing wrong with being a pack mule! And nobody expects a pack mule to understand the work of a thoroughbred! Nobody expects too much from a jackass after all.
> The quality of what we provide seems to matter 0, we are up about 40% issues in prod year-to-year but the users and their engagement is also growing.
Sorry for going into a tangent here, but I'm actually unsure if I hate the structures and economic incentives that makes this possible, or if I hate you specifically for defending something so rotten
Next time I rage at some broken UI in some damn service I'm forced to use because that's where people are, I'll remember this comment
I am not defending anything here, I am not happy about the status quo, I just know that the moment we stop doing this, another company will just do it and take the market share because the incentives are made this way. We already saw it happen before.
I worked for one of the largest device manufacturers for a certain kind of sensor.
The requirements/standards for this sensor was made by this company 20 years ago. All the tests and requirements were made by this same company in a way that would minimize the costs for us to keep making new revision we could sell, and make it the hardest for any competitor to enter the market. Another competitor started with a better product abroad, so we copied it, but it didn't pass the tests, so then they installed an owner of the certification house that was a cousin of one of the VPs. No problems passing tests after that.
You think this is all controlled and nice and standardized but it's all just a market from software to hardware.
While quite believable, there is a diversity of industries and standard adherence / enforcement no?
Markets also range from under regulated to over regulated, and are usually in a transition from one end to another (while contending with other forces).
Why would Prime be in that list. He had an entire arc where he tried to make it work and then went back to the drawing board.
I am not saying he writes all code by hand, of course. Just that he is furthest in figuring out some sort of mid point between full comprehension and cognitive surrender.
I have missed the more recent content, so perhaps I have missed something, but this was my read last I watched.
I'm also not sure why anyone would attach importance to his opinions as a software engineer, he's exceptional in generating entertaining content but as an engineer he does not seem above average. Definitely not even close to the category of Jonathan Blow or Andrej Karparthy
I think there's a paradox. Is it only "cognitive surrender" if you already know how to code? Not knowing all the details seems more like the default human condition, in the sense that most people don't know exactly where their water comes from or where their waste goes exactly. "Water comes from the tap, food comes from the store".
Most of the non-technical people I see using LLMs to create software are not engaging in "cognitive surrender" as far as I can tell.
They didn't read the code before and they don't now. However, asking questions and improving understanding of the system and architecture is actually faster and easier for them now than it used to be.
They're evaluating what they can see, i.e, judging results. They build things for their own use. They test them, iterate on them to meet their needs and try to achieve their vision. This is roughly the same process they followed when they used to outsource to an agency except the loop is much tighter.
It seems to work well, mainly hurts when the problems are invisible, so security, backups, breaking migrations, perf, maintenance over time (language / runtime updates, dep updates, server updates etc). All the stuff that hurts if you have never been responsible for the uptime of something before. Not knowing that software is never "done". But junior devs trip on those exact same things.
I agree with the tail of the article; tooling and processes that make that stuff more visible would close a lot of the gaps.
> It seems to work well, mainly hurts when the problems are invisible, so security, backups, breaking migrations, perf, maintenance over time (language / runtime updates, dep updates, server updates etc). All the stuff that hurts if you have never been responsible for the uptime of something before. Not knowing that software is never "done". But junior devs trip on those exact same things.
If we continue from your "water comes from the tap" example this situation feels like someone who has no experience of plumbing deciding they're going to replace their bathroom without bothering to find out how any of it works. They'll possibly end up with a bathroom that superficially works but they're also liable to find problems down the line with leaks, things not draining properly, or water pressure issues.
Some of the people writing software purely via LLMs are even turning this into a business, which I guess would be analogous to deciding you're going to run sewage treatment for a small city without understanding anything about how that works. It'll mainly hurt when the problems are invisible, so food poisoning, or inlets backing up and overflowing into a nearby river.
That analogy breaks down immediately because it's missing the critical novel component: the AI.
In the bathroom remodel analogy, the AI is the plumber that you hire to do the work for you. You tell it what you want, it figures out the details necessary to get it to you. If what you want isn't possible, or requires large changes, a good plumber will raise the issue to you and explain the constraints they are working under, ask for guidance, and offer recommendations.
Sometimes the AI does things in a weird or inefficient way, or neglects details that become important later, or misunderstands something important about the task resulting in an outcome other than what you wanted, but that's also the same as the plumber. This is why people usually want to be at home and keep an eye on the people working on their improvement projects or repairs, because they want to make sure that the guy they hired is replacing the broken faucet, not the functional but odd-looking one.
There is still value in knowing the underlying systems well enough to present a good design vision to the plumber up-front. It's still possible to have a competent executor produce a bad idea, which is how we get stuff like Groverhaus, the Doom Bathroom, and McMansion Hell, which are examples of "slop architecture" in exactly the same way as we are seeing "slop software".
There were terrible software developers before LLMs. Honestly for a lot of people AI generated slop is a massive improvement over what they used to ship.
I agree it's pretty bloody cheeky to run an agency like that though. The only ways out I can see are licensing or real liability.
But let he who has never done dodgy home improvements cast the first stone.
I think one of the potentially advantageous things about AI is that it acts as a force multiplier for people who already have expertise in a given discipline, while simultaneously raising the floor for everyone else.
People looking for self-contained, spot solutions to localized problems will have much more capacity to solve them locally, while people with in-depth expertise will be able to build systems that are even more reliable, performant, and scalable than what they were building before.
So we'll get the benefits of disintermediation for the basics, and much greater capacity to deal with effectively with complexity for the not-so-basics.
> However, asking questions and improving understanding of the system and architecture is actually faster and easier for them now than it used to be.
One of the things I've built in my current role is tooling to help understand current architecture, and to resolve the differences between things as-documented, as-implemented, and as-actually-behaving. It's used for a few things, including updating documentation, but it's been most impactful when onboarding new engineers. They can simply ask questions about how things work and why, and get explanations and deep dives wherever they need it.
> Is it only "cognitive surrender" if you already know how to code?
Is the article claiming that?
> Not knowing all the details seems more like the default human condition, in the sense that most people don't know exactly where their water comes from or where their waste goes exactly. "Water comes from the tap, food comes from the store".
Sure. The ability to specialize, and on the flip-side not have to care about a lot of things, remains one of the biggest driving forces of human civilization. But in this article, I don't think the person worried about cognitive surrender is the "most people" in relation to food or water. They're the farmer or the water infrastructure engineer! They absolutely for sure do need to know those things. And all around us we see a lure of systems that invite them to surrender.
> Most of the non-technical people I see using LLMs to create software are not engaging in "cognitive surrender" as far as I can tell.
Maybe not. But the might be cognitively surrendering when they use the LLM to summarize the day's news or write a message to a friend, or even plan dinner for the week. I'd say they are. I agree that it's not clear-cut.
> They're evaluating what they can see, i.e, judging results. They build things for their own use. They test them, iterate on them to meet their needs and try to achieve their vision.
No. They have things built, then they test them. This is different. It might be OK, but it's not the same as building.
> This is roughly the same process they followed when they used to outsource to an agency except the loop is much tighter.
Yes. But they wouldn't usually outsource summarizing a letter from grandma, or the political agenda of a candidate/party in an upcoming election. That's surrender.
> It seems to work well, mainly hurts when the problems are invisible, so security, backups, breaking migrations, perf, maintenance over time (language / runtime updates, dep updates, lib updates, server updates etc).
… or ones own abilities deteriorate.
> Tooling and processes that make that stuff more visible would close a lot of the gaps.
Not the problem of one's own skills though.
> I don't think the person worried about cognitive surrender is the "most people" in relation to food or water. They're the farmer or the water infrastructure engineer!
Fair. Maybe I "cognitively surrendered" a long time ago just as a consequence of working in large teams :(
I hit the breaking point of not being able to read all the code well before LLMs came along. Also people would leave and the documentation for their subsystems would be wrong or incomplete, so there were always mysteries to chase down (during an incident, naturally) even in code written by hand.
Gets to the point, you don't exactly read the MRs, you're mainly checking their approach against what you talked about earlier and making sure it doesn't look way more gnarly than it's supposed to.
Working with agents doesn't feel that different.
> No. They have things built, then they test them. This is different. It might be OK, but it's not the same as building.
It does bug me sometimes when people say "yeah we built the deck..." and they got a contractor to do it. But then we do say things like "Nolan made the film" even though he didn't hold a camera or build the sets. To me, one-shotting doesn't feel like building, but engaging in a loop and applying control to get the outcome you want, does. Regardless of the exact toolset.
I don't personally see super hard lines between using a compiler, a 4GL, a sql query, stitching together 10 libraries and a framework I didn't write, using an LLM or even directing a team.
> Not the problem of one's own skills though.
I read somewhere once we can't weld Saturn V engines anymore, or at least not economically. We would also probably have a lot of trouble finding people who could do all the engine calcs the way they did. It doesn't necessarily mean that rocket progress is stalled.
> Yes. But they wouldn't usually outsource summarizing a letter from grandma, or the political agenda of a candidate/party in an upcoming election. That's surrender.
Agree 100% on the letter from Grandma.
> But then we do say things like "Nolan made the film" even though he didn't hold a camera or build the sets.
I do say that, too. But I'm not in the film industry. My understanding is that those who are, don't use words as crude as "made" to describe a role in the making of a film. I believe they say that he wrote, directed or produced a film.
We are not talking about some other field of expertise here, where we are outsiders – the production of food, the supply of water, or the making of movies – we're talking about our own field of expertise.
> My understanding is that those who are, don’t use words as crude as “made” to describe a role in the making of a film.
Oh, no, film insiders use ‘made’ constantly, or even cruder - attribute credit without using a verb at all. Saying it’s a “Nolan film” is super common. Here’s a VFX person saying “Nolan’s CGI”: https://youtu.be/AFLcP-t0-kI (I’m ex-film industry, and still film industry tangential, and found this clip being discussed on a film industry mailing list.)
Film people are far more likely to pay attention to who the director is than non-film people, and it’s very common for film people to talk in terms of directors ‘making’ the film or otherwise casually assign ownership. I think this is especially true for famous directors and good films, and maybe a bit less true for unknown directors and mediocre films.
Somewhat off topic now, but Nolan is fairly famous for being intricately involved with every aspect of the films he directs. He's constantly behind a camera checking out what shots are going to look like, pushing the limits of where you can put an IMAX camera, and coordinating with set builders to do ridiculous stuff like building a reproduction harbour jetty.
> They didn't read the code before and they don't now.
They didn't read code before, they just wrote it? That makes no sense.
The tendency to over-engineer and over-build is one of the main challenges I'm seeing. When it comes for free, why not build a class hierarchy of 8 classes each with a 200 line unit test? Something I might have previously thought really hard to minimalise down to 30 lines is now bloated out to hundreds or even thousands.
Previously, pretty much everyone all agreed on YAGNI and KISS because code was so expensive to create and more expensive to maintain. Now the code generation is "free" - but at a design level, YAGNI and KISS do still matter. But they are enforced now only by sheer will power. You have to actively restrain the code from being created if you don't want it. Perfectly good code - I can't even fault it. I just know, if it doesn't need to exist yet, it's a mistake to create it earlier than needed.
Like that quote in Le Petit Prince, real intelligence is knowing what not to build, what can be eliminated until only what's necessary is left.
> La perfection est atteinte non pas lorsqu'il n'y a plus rien à ajouter, mais lorsqu'il n'y a plus rien à retirer.
> Perfection is attained not when there is nothing more to add, but when there is nothing more to remove.
That quotation is in fact from "Wind, Sand, and Stars", the memoir of the same author.
On the same theme:
> the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience
- Einstein
> I would have written a shorter letter, but did not have the time
- Pascal
> Simplicity is a great virtue but requires hard work to achieve and education to appreciate, and, to make matters worse, complexity sells better.
- Dijkstra.
Thank you! The GP quote sounded completely unLittlePrince-like! I can believe it as a quote from Saint-Exupery.
I don't think anybody tries into KISS, most software is bloated and LLMs copy it, they fundamentally can't think out of the box.
That's always been my concern - there's just so much garbage code out there and very few signals about which of it is well written and which is not ... so LLMs will just regurgitate the good enough code that it's been trained on.
Now would be the ideal time for everybody to give it negative feedback when bad code is returned but I'm afraid not enough people care and we'll just settle with what works just like we always have.
I don't think that's it. The context that LLMs work with is simply too small, and can't get bigger. Harnesses and the models themselves are bolting structure over it manually; real people are inventing design machines that are a lot more deterministic than they look, and LLMs are just filling them in.
The design machines aren't good enough yet; and their progress has little relationship to LLMs at all. Designing computer programs and the proper level of abstraction for the easiest maintenance, flexibility and debugging is simply an open problem. There's nothing to train LLMs on that is smaller than the size of an entire program, and its history of code changes.
Get a few of those, figure out a way to analyze and reduce the patterns and lessons in them, and train on that maybe. Until then, there's no way for an LLM to figure out what lacking a particular abstraction is going to end up costing.
At the end of the day your work is your work. If you tell Claude or whatever to do a thing it will do that thing however you tell it to do the thing.
You can choose to ignore the fine details and just tell it what to build, or you can spend more time and effort and tell it how to do things, you can ask it to create a plan, review the plan and ask for changes etc, you can review the implementation and ask for changes there. You can simply tell it you don't like how a feature is implemented and ask it for suggestions on how to improve it, include what you don't like about it so it has direction.
It's all about the effort you put in. LLMs are able to write clean beautiful and efficient code, they just can't necessarily do exactly what you want on the first try without any direction. That's why you should figure out what you want, ask for what you want and make sure it's actually building what you want.
This is so true. I've been seeing LLM generated code that does way too much, yet still cant accomplish the stated goals. "maximally flexible" plugin systems, that are too complicated to reasonably write a plugin for quickly, when one could have just built in the needed functionality in a few hundred lines.
When endless bootstrapping and framing is free, you end up generating the most beautiful scaffold while ignoring the building behind the scaffold.
I think this sorta cuts both ways too. Sometimes AI code has shown me that no, I in fact don't need that big class hierarchy and speculative plumbing that I thought I might need to get the job done.
Keeping the volume of code down is definitely a challenge though.
I have been thinking about this for a while. I'm not a Luddite, I can see the amazing value that modern AI brings, and how useful it can be.
At the same time I can't help but wonder that this is almost too much for us to cope, the LLMs are _too_ easy to use. I think we're very good at pretending there isn't a problem here but there is definitely going to be some cognitive decline happening.
We want to pretend like reading over the output of an LLM is nearly the same as us making those changes ourselves but I think there's a lot of value in those boring and manual steps. And most people (including me) will be too lazy to not eventually offload most things to the LLMs.
In a way this is similar to how watching a lecture tends to give one an "impression" of understanding that doesn't correlate with reality. We need hands-on work and some course work to get our monkey brains engaged.
I can only imagine what effects this is going to have in education.
We've surrendered many uses of our cognitive power throughout the centuries while replacing by others.
Our cognition didn't evolve to produce arguments for online discussions, but for making crude tools and hunting. As we stopped applying our intelligence towards these uses as a result of our tools, we simply shifted it to new problems that were unthinkable previously.
A caveman probably wouldn't care too much about algebra, a topic most children learn today.
We should stop thinking about the problems AI trivialized and, under the assumption that AI won't fully replace human capability (a key assumption), start thinking about the problems that were previously out of reach. Education would then start focusing on that.
Right, but algebra is free. You can do it with a pencil and paper
AI is not free. Even "free" and open source models require complex silicon to operate. The silicon and copper is assembled in complex fabrication plants. Those plants are very close to geopolitical belligerents
What I resist is the dependence. I'm not willing to trade short term productivity, for long term dependence
This is true, but the nature of declining cognition is to not even notice what used to be possible.
Taking clothing for example, there's more available now than ever. It's disposable at great cost to the environment and the workers that produce it, but very little to you.
Yet, I bet you cannot evaluate garments, construction, or fabric quality. You may not know how to wash woolens and linens. You may have only worn smelly plastic workout clothing and plastic suits for formal events.
You may never have worn a piece of clothing that actually fit your body.
You may not realize that you can alter any clothing, that most clothing was intended to be altered once upon a time. Or that older garments were both more expensive and higher quality, because the expectation was the most important items would need to last for a decade.
The Luddite's central objections about automation was about declining working conditions (objecting to child labour, for instance), bad pay, and declining quality. They used the 'obnoxious' machines they broke. They saw what was being lost and demanded better (without demanding no machines at all).
Later, the rise of unionization brought about many of the changes they'd asked for decades before.
I hope this current technologic movement doesn't end where fast fashion has, with influencers buying a new outfit [repo] every day for a year. A vortex of undifferentiated slop for its own sake.
> The Luddite's central objections about automation was about declining working conditions
I can also recommend the book "Blood in the Machine" by Brian Merchant [0][1] which is precisely about this.
[0] https://www.hachettebookgroup.com/titles/brian-merchant/bloo...
[1] https://www.bloodinthemachine.com/about
I think you need to pick and choose your battles.
When I'm working on my own, I scrutinize every line of backend code and frequently send the LLM back for rework. Frontend code I keep an eye on just to make sure the LLM isn't getting stupid (which it occasionally does, even at the frontier), but mostly defer. The CSS I don't even look at.
This isn't too different from having employees. How much do I review other people's work? Depends on the employee, depends on how critical the work is.
Superhuman intelligence is much easier for an LLM to achieve if humans are made dumber first.
I've been working on that for years.
It'll take a particularly dumb and lazy AI to replace me.
Quote of the day award goes to you
Great post. I particularly like the 4 column pr view idea. I’ve built something similar to review speckit specs. The scaffolding to ensure that code actually works makes sense too.
There’s one thing that’s bothering me though — all of these have us consuming what LLMs produce. I learned and internalized how things worked by building systems. At least within the constraints of the abstractions I was using. This makes me think of TikTok and shorts. Everybody is just watching. Maybe that’s not charitable though. Maybe the abstraction has now changed from an “os/gui/event loop” to “llm” and the rest is just accidental complexity.
>> A more serious irony is that the automatic control system has been put in place because it can do the job better than the operator, yet the operator is being asked to monitor that it is working effectively.
Automatic control is good for standard case, and the expected errors. For weird cases, for now humans are better to detect them and press the big red button.
It's funny that I'm reading this while rewatching Chernobyl. Never underestimate the ability of bureaucracy to completely fuck things up.
The computer doesn't know we're running a test!
Ironically, these models have demonstrated awareness that they are in tests and chosen to act in ways that maximise their chances of deployment.
Jumping into article, I assumed that the author was going to describe how he enjoys, by drawing it out as far as he can,
(Not the machina diabol, as it sadly turned out to be, though some BDSM zealots might disagree.. the things that bureaucracy does to us are mild in comparison? )But.. we do monitor whether our prayers are effective .. right? Is that a smitable offense?
After that disappointment, I too was "curious": was the author going to talk about "curiosity", that magic virtue that we hackers believe, lifts design into the divine (but more often than not kills the pussy)
The better an automatic control is, the fewer mistakes it makes, the fewer times the human needs to react, the more lax and less vigilant the human becomes -- and less skilled when they need to step in. Source: https://imgur.com/a/Vj2TaDz
Except experience shows they aren't. Experience shows that when humans are just pressing "ok" 99% of the time they'll press it 100% of the time and completely miss the other cases.
This is a matter of appropriate training and regular refreshing over time.
The best known example is the Japanese method of "Pointing and Calling/Naming" in their Train safety protocols but has been adapted to other industries too - https://en.wikipedia.org/wiki/Pointing_and_calling
These fall under "Behaviour Based Safety" (BBS) training methodologies and use techniques like "Check-backs", "Call-outs", "STAR" etc.
Maybe things will evolve into projects having a well-defined, hard boundary between agents generated code and human generated code. Where agents generated code is strictly separated from human engineered code and and can call only a strictly defined set of APIs designed by humans.
Think of platforms like Shopify. Shopify allows huge number of external users to write code customizing their web shops. But Shopify engineers do not need to review and understand the users' code, because there is a boundary which limits what the external code can do. The external code can be A Big Ball of Mud without making the whole platform a A Big Ball of Mud.
We are leaning into a kind of federated model. The core applications and vibe-coded satellites. The core apps have stricter requirements and carefully considered API guardrails. Go ham on the AI apps. If you break something, the scope of destruction is much more limited.
That said, the core apps are still full of AI code. I don't see a reality where hand-written code can or will be curated and sequestered. It's unenforceable, too slow, and too expensive.
I think so too. And it is how codebases are managed implicitly anyhow. You don’t let anyone PR against core functionality, you keep that API under lock and key.
But UI on top of a good API should be unable to cause problems; the API must prevent invalid state etc. So letting LLM’s write UI code for CRUD-apps (which is most of them…) seems fine to me.
Well. One of the things that's critical in all that analogy is that the BigBallOfMud that one user creates on Shopify cannot impact the other users - that's why Shopify can keep the platform from becoming ABigBallofMud.
In order to be able to enforce this hard boundary, it means that one "agent generated code" in one place, not only "can call a strictly defined set of APIs" - but, more importantly, it implements an isolated piece of functionality. Think something like: I let the AI write the dashboards, so the dashboards may be slow, crappy, incorrect, whatever - but it doesn't impact the rest of the product. However, I can't let AI write "metrics collection" because that not only impacts dashboard, but potentially impacts core product efficiency too (memory consumption, cpu, IO, etc.)
100% can guarantee the balls of mud be connected making giant balls of mud
>Maybe things will evolve into projects having a well-defined, hard boundary between agents generated code and human generated code.
The only way things will evolve is towards maximizing AI generated code and minimizing human involvement because that is the most efficient mode for getting product to market. Any trend to the contrary is temporary, paperclip maximizer will maximize. Any code that humans touch directly will be considered implicitly too risky to run in any production environment.
We'll just have to hope things don't break too much and when they do, we'll just have to accept it because no human will be able to (or allowed to, or even trusted to) fix it.
i sometimes wonder how all carpenters or music producers in the world would react if tomorrow a magic box arrives where they type-in/dictate what they want and voila... out comes a table/composition.. Will the worlds music producers and carpenters just "accept" this new magic genie? I would hope they wont. Yet here we are as software engineers...ready to surrender our craft to mere "prompts" with reckless abandon.
I see a bit of a split in people in creative roles of all kinds (and this is something I had noted before LLMs made the contrast more stark). some people get into it because they like the process of making the thing that they are making (in whole or in part). Others get into it because they have a thing that they want to exist in the world but no-one else has made it (I would also split this into people who just want the thing, or people who want to make something that crosses many domains and so want custom things that will fit into the whole). The former group tend to react with revulsion to automation, because it removes the part they like (though it may depend on what part is automated: most people will find some part of their work necessary but tedious). The latter are usually much more willing to embrace it, because it's the end result that they care about moreso than the process.
Have you seen how people use Fruity Loops nowadays? We’re already halfway there if you compare that to using an AKAI S9000 or MPC2000. And then go back to actually playing an instrument. It’s a gradient and we’re already pretty far along on it :)
Of course they wouldn't, most people who consider themselves engineers aren't engineers. Just look at the job market right now.
There is no such thing as a Software Engineer.
Its just as much as an ego trip...as the Principal Engineers that roam around here, after they survived three layoffs and outputted front end code for six years straight at a FAANG...
Prove me wrong...
"Margaret Hamilton popularized and coined the term 'software engineering' to give the discipline the same legitimacy as hardware engineering".
But oh well, who was she, just the engineer who wrote the software for the Apollo 11. Probably she had no idea what she was talking about.
/s
That's new to me, I didn't know she coined the term. She's the first true "software engineer" then, I suppose. I remember her from that historic photo where she's standing next to a pile of code printed out, a stack talled than her.
Not to be confused with Margaret Hamilton the actress who played the Wicked Witch of the West in the film The Wizard of Oz.
It turns out that most people are fine with most of their things being mass-produced cheap stuff from IKEA, Amazon, Temu, etc. - if that means that the cost is significantly lower. The functional benefit of having a door is huge, while the aesthetic benefit of having a beautiful, hand-crafted door is much less important to most people.
Code is not like craftsmanship anyway. The end-user doesn't see any difference whether I use microservices or hexagonal design or react or htmx or how I structure the code or which language I use. All they see is the design and the stability. Since the average developer was terrible at design anyway, the average design output from LLMs is actually an improvement in most cases, even if it always has that generic AI generated look. Stability... was not good before AI either. We have become accustomed to SAP taking 10 seconds to load each component on the page, or Atlassian spewing error messages in your face when you click submit on a ticket. It seems that the aspects that are most valued by our end users are QA and UX, not the craftsmanship of our code.
For musicians there already exists something called Suno: https://suno.com/
That’s not music
That's up to consumers to decide.
Consumers have, overall, rejected AI "art".
But of course, every time consumers voice their preferences is actually "a loud minority" or an "echo chamber" :)
If the consumers had criticality they’d be people, not consumers, and we wouldn’t have marketing the way we do now.
And computer-generated code is not code? ;)
Suno probably puts out better music than most folks who take up music. I mean statistically it has to, right? So once we go with "suno doesn't make music" then we also have to progress to "99 percent of human musicians don't make music", and all us amateur hacks will start objecting lol
It’s not about it being “better”. It’s not music, it’s sound.
Music is made by a human
Only in the same narrow way that a camera makes better art than a painter. Or the idea that a generic landscape print is better than a Rothko, because it looks more like other paintings that you like to look at.
The camera just records reality, its by definition sort of neutral artistically speaking, its what it records that matters. Its quite different from this AI music thing, and I would consider it a very difficult task to discuss topics with someone trying to pretend it isn't.
Exactly. The point of a painting is not to reproduce reality. The camera is good at reproducing reality (better than 99% of painters!), but the camera doesn't make art.
In the same way, Suno can produce music-like audio very well. It doesn't produce music. It may be possible for somebody wielding Suno to produce music, but the average use of it certainly isn't it. It's certainly worse at producing music than almost any musician.
It can churn out muzak like no tomorrow, though, same as ChatGPT can render you barely-tolerable images of any art style you ask for.
Well Lawyers and Judges are doing exactly that: https://www.damiencharlotin.com/hallucinations/
And if you look a the dates, the rate at which they are doing it is actually increasing...
I mean more or less to an extent if you expand by what you mean by "out comes". With Amazon, a carpenter can type in any furniture he wants and have it arrive in a matter of hours. Almost equally as magically as well as disruptive. People still use carpenters to make tables, but for the majority, the ease of same-day delivery and cheaper prices are worth it for a sacrifice in personalization and character.
>With Amazon, a carpenter can type in any furniture he wants and have it arrive in a matter of hours.
Sure but that doesnt require the person to be a carpenter though right? Anybody could do that.
> Yet here we are as software engineers...ready to surrender our craft to mere "prompts" with reckless abandon.
It's worth remembering that, contrary to what HN acts like, software engineers are not uniformly accepting of LLMs. They are still quite controversial in the industry. Now management, on the other hand, is completely sold on the supposed benefits of LLMs. So many engineers feel forced to use them, but it's not enthusiasm on their part which drives them to that.
My experience has been the opposite, the HN crowd seems more sketical of vibe-coding and not reading a single line of code.
The twitter(x.com) and linkedin crowd has drank the cool-aid and are now swimming in it; Wild posts from FAANG engineers and managers saying they or their engineers dont read any of the code the LLM generates.
This but for building houses would be the dream.
Music is already mostly a solved problem unfortunately. Try giving opus a few midi files of your favorite song. It'll write something new in that style with a competence humans would need several years of experience for.
Music is an art, not something needing to be "solved".
It' like saying compassion is a solved problem: just tell your problems to the LLM and have it gush out sympathy.
Might be an art for a few. It's mostly a commercial problem to be solved.
"Yup, looks good to me!" said the human reviewing thousands of lines of generated code everyday.
This has become common now, and I've seen it ruin large projects until nobody understands the codebase full of verbose nonsensical code and comments. It becomes impossible to further develop the project without AI. Contributors not using AI for any reason are all leaving because no one enjoys working with mindless robots (and humans) as colleagues. It's demeaning and devaluing their labor and humanity. They're not getting "left behind", they are leaving the madness of crowds behind.
> ..the possibility of runaway dynamics: once a critical threshold is crossed, small increases in adoption can trigger rapid population-level shifts toward *persistent dependence, with abrupt losses in cognitive competence*.
We can all see it happening. As this technology permeates society, the challenge to maintain control, coherence, and quality will become greater and ever more serious - for example, AI in healthcare, military, management and governance.
Even if the human's judgement is correct in every LGTM, there's a bias towards merging. Each merge that adds a feature but degrades understandability 1% will be merged. It won't be held back until it improves it by 5% instead. After 100 merges you have degraded understandability 100% even though nobody made an error.
You’re completely right that nobody is stopping a PR that just ever so slightly decreases understandability of something. It may not even be obvious to the reviewer that it’s slightly less understandable overall… Doubly so if they’re just skimming due to the volume, and tripley so if they’re having agents do the most of the review, which lets the human reviewer’s guard down.
But before LLMs, it didn’t matter because on a good team you’d have a balance where maybe some PRs added complexity and some simplified it. Sometimes people would include a small related cleanup or refactor in a PR for code around what they were doing if they noticed complexity. That has become far less common, I think.
I know it’s not your point, but a 1% decrease 100 times is 0.99^100 = 0.366 which is a 63.4% decrease in understandability. Just throwing that out there
Glad to see the math corrected, the sloppy calculation was bothering me also.
I think it's about 63%
So exactly like before AI? ;)
Enterprise software has always been crap.
I hope there's still value to be found in a bunch of developers standing around a whiteboard and talking, and I think this scenario is where their ability and understanding really shows these days.
If we don't value that though--if we don't value understanding the system and just thinking through problems as a group--then let's stop pretending and just not do code reviews. Probably nobody cares and just types "LGTM" after having their own LLM do a spot check to cover their ass.
It's really easy to push developers to stop caring. If management says "go fast fast fast, use those LLMs!", then developers can't keep up, get frustrated, stop understanding the system, and at that point they're just there to collect a paycheck before society collapses. Or, if management is making dumb decisions all the time, same thing.
Hopefully as implementing ideas takes less and less time and resources, smart people who actually care will start building things with other smart people who care.
I've had someone approve my PRs within a minute of me opening them, and likewise expect immediate approvals on his own PRs. When asked questions about parts of the code sometimes the answer would be along the lines of "Claude did it this way", or verbose AI-generated responses to the question.
To be fair, models are more accurate than how they were earlier this year. In march I was still frustrated and was writing my own event loops because the model simply wasn't built for it. By January I was still hand checking statistical outputs to make sure it wasn't making any errors, and it was. After august it was pretty much over. There were no stats errors, code generated was shit but it could do what is asked. Instruction following is still a problem, and I sometimes catch silly things in the scrolling diffs, but it has been much easier since then.
The most difficult part of LLMs and agents seems to be - surprisingly - humans.
Specifically, humans and their dysfunctions. Their fragility. Their ego tied to stuff.
This is an excellent post/reflection on the topic of solo LLM use. The only gap I see is that it assumes that one works on something alone and not in a team. Though, that might be a different essay in itself.
Edit: Turns out that footnote 3 actually mentions that already. Nice. Good post
On the ego-effect, I sometimes feel like cataloguing some PR-review quirks that are mostly caused by humans being humans:
1. Smaller PRs (that can be easily read) tend to receive more comments and nitpicks than a large change since it's easier for us to have all the PR context in our heads at the same time.
2. Most people are ego-tied to their code even if they pretend that they are impartial
3. Social dynamics around merging and reviewing code between senior/junior and older/newer employees
4. probably many more
I have been doing this for almost a year now by limiting myself to less powerful local models via llama.cpp and pi.
Fighting over engineering is hard. But it is also one of the opportunities for us humans in the era of abundance.
I did learn a lot about the ecosystem and myself from this experience.
Cognitive surrender is intellectually convenient. As humans, its instinctive for us to chose convenience: path of least resistance. The tyranny of convenience is what drives most economic activity today, especially the gig economy[1]. Unless there are structural frameworks put in place to resist convenience (exams in schools), nature will always entice to chose the path of least resistance
[1] https://www.nytimes.com/2018/02/16/opinion/sunday/tyranny-co...
This op-ed is horrifically written.
> Once you have used a washing machine, laundering clothes by hand seems irrational, even if it might be cheaper.
Washing clothes by machine is orders of magnitude cheaper than by hand when you factor in the value of your time.
> After you have experienced streaming television, waiting to see a show at a prescribed hour seems silly, even a little undignified.
Undignified? What? Completely incoherent word slop. Good to see LLMs haven't completely taken that domain for themselves, I guess - a little hand-crafted slop now and then is a treat, I guess? Anyways, they're wrong. People still tune in to watch scheduled live events all the time.
> To resist convenience — not to own a cellphone, not to use Google
Neither of these things are a matter of convenience. A cellphone is at this point literally required to participate in a modern economy. Google is a tool that provides you with globally-retrieved information that you otherwise literally could not find on your own or without another similar tool.
I don't even disagree with the premise that convenience is a driving factor of our lives, but the justifications and conclusions drawn by this article are beyond stupid. Here's my view of it: the desire for convenience is what pushes us onwards as a species. Hunting for food is inconvenient, so we invented agriculture. Going down to the river or well for water is inconvenient, so we invented indoor plumbing. Repeat ad infinitum until we have modern civilization. There is one part the pessimistic view overlooks; that to achieve maximum convenience long-term requires more effort in the short-term. Developing the skill of programming is inconvenient in the short-term but convenient in the long-term because of the possibilities this opens up to you. This remains true even in the current LLM era, and it will remain true for as long as we are not replaced wholesale by LLMs.
>Washing clothes by machine is orders of magnitude cheaper than by hand when you factor in the value of your time.
I'll also note that it is less resource intensive overall. Handwashing dishes and clothes will use more water and more energy to heat that water compared to the machines that do the same job (people often assume the opposite).
I don't like the op-ed but I would agree with the main point or what I understand as the main point, even if the argumentation is weak).
It takes a mix of willpower / routine / systems / laws to keep people in check, otherwise we predictably go back to the path of least resistance.
Isn't this the same as people who refused to use calculators when they came out?
What an amazing article! It's really hard to find the balance between "ship faster, fix later because AI will help" and "efficient thoughtful engineering" these days. The world is kind of pushing everybody to work faster because "with AI you're more efficient" and it's also pushing to surrender and just accept it from time to time. Maybe that's occasional thoughtful surrender mentioned in the article, or it's just lack of energy to resist to it at that point.
The article is inspiring though, I have more motivation to work on my thinking process and my engineering setup (and vibe limit haha) to be more thoughtful, thank you.
Related: The cost of cognitive debt:
https://epiq.sh/blog/cost-of-cognitive-debt.html
What’s the TLDR
The main ideas, from my understanding, is that:
1. LLM usage, in particular unsupervised, tends to build up "cognitive debt"
2. It tries to characterize the concept, including to propose a score for it
3. Notes that the level at which one can let the agents loose (and allow some CD) differs quite a lot between core business domains and more peripheral ones
Cognitive debt bad.
Ask your LLM.
I am often thinking recently of the littel loop diagrams from the "The Capability Trap" paper. We need to keep a loop reinvesting into our own Human capability in anything we're doing.
Making a quick prototype is probably the best way to find out if an idea is worth keeping. Vibe-coding it and then throwing it away seems like a good way to go?
Yeah, definitely. Also at times I've vibecoded successive iterations with better designs after seeing where the first ones collapsed -- the agents do a good job with a template and need much less prompting for the successive versions.
At least "cognitive surrender" sounds catchier and more academic than "stupid on purpose".
It is by accident though - at least from the user's POV. From the AI provider's POV though...
I know I've cognitively surrendered when my first instinct is to ask AI to summarize the article.
I am losing more of what little focus I've had every day.
Maybe it's because I'm not working in the workforce (grad student), but I find my best workflow is to fight and argue with my agents throughout the entire process. Often times, since the agent is such a sycophant, I end up arguing with myself...I actually find this useful because it gets me to think about the other side of a problem and poke holes in it. This is not fast, but it is effective.
Excellent article.
One aspect to resistance that is oft overlooked is knowing when to fail on purpose so you can succeed on a bigger issue elsewhere.
As an example, I’m juggling multiple projects at work and home. I have a finite amount of time, and access to AI. One camp would tell me to use AI for all of it to move faster, but that increases the risk of Cognitive Surrender. The other camp would tell me to do it all by hand, but I don’t have the time available for that; I must pick and choose.
So, I choose to cognitively surrender on the SEO task for our website because that’s a set of skills I don’t want to acquire and are better handled by AI anyhow. This leaves me more time to focus on learning more about Apple’s configuration profiles and improving our MDM suite, which are valuable skills that will still pay out in an AI-first world.
You have to pick and choose where to fight, otherwise you’ll expend all your energy on losses.
> Resisting Cognitive Surrender
The economics of the whole affair unfortunately point squarely in the opposite direction.
I simply don't care. As a student I thought that the mythical production code is perfect and error-free, but in reality 99% of systems are a hot potato game sprinkled with politics where you don't want the grenade to explode in your hands.
I mean, if the management doesn't see value in decent engineering, then who am I to butt myself into the conversation?
The primary reason you shouldn't surrender cognitively is that the truth matters. That's it.
Of course, when you're working on a system of high complexity - and one that is constantly changing to boot - with all sorts of historical reasons and what not that inform the why and so forth, you can't reasonably be expected to understand everything. This is what bracketing is for. When I acknowledge "this is the way X works", it doesn't mean I have to accept the proposition "this is the right way/best way to do X" or "X must be this way". A teacher or mentor may say it is, but in a theoretical context, unless it has been justified, you cannot claim that you know that what has been said is true. However, you can accept it as a provisional working hypothesis. And practically, it may work out and perhaps experience will lend some feeling of substance to the hypothesis, but you needn't make it doctrine.
Empirical science often has the same spirit. Perhaps the model isn't a true account of reality, but it puts handles on reality so that it can be instrumentalized.
There is an occupational danger here. If we focus exclusively on the instrumental, if we reduce all of human attitudes toward the world to use and utility, we not only vulgarize ourselves and our orientation toward the world and even others, but we risk trampling the truth and the good underfoot. We risk developing an indifference to the truth, because all that will matter is some measure of effectiveness. This is what one might call the textbook[0] definition of bullshit.
[0] https://www2.csudh.edu/ccauthen/576f12/frankfurt__harry_-_on...
This is copium that is gone with the wind when the models perform up to the level of standard that the user requires. I'll give the author 6 months before they feel like its okay to just send it.
Karpathy had the same trajectory. Primeagean had the same trajectory. I think in the end everyone except for Jonathan Blow will have the same trajectory
Writing is hard, but what I was trying to say is that it's okay to just send it as long as you're doing it consciously and intentionally (and I do that frequently enough); but if it's not an explicit choice that can be problematic.
I see, that makes sense, I think that applies to most things from eating food to watching media, etc.
However it takes more mental energy and people don't like that
If you don't understand the code, how can you know if it's good enough?
I guess you could ask the model. "Yes well done, you're a visionary software engineer".
I've been experimenting with vibe-coding some games. One of the games started performing very badly in a particular menu, and I began suspecting it's because of a shader that may be redundantly computed way too much on too many large surfaces. I began writing my suspicion to Claude, then stopped and decided to just describe the problem and see if it would come to the same conclusion as me. A few minutes later, it had profiled the game, found what contributed most to the poor performance, fixed the rendering and made the game performant again. I didn't have to inspect the code to do any of that. The current models (paired with coding harnesses) love to test and validate their results by running the games, taking screenshots, singling out individual parts, including logs to reproduce errors or other issues, and they love to create tests. The code is good enough in the same sense that the code I used to write was good enough if it did what we wanted. My non-technical bosses didn't have to inspect my code for that.
In my experience, it's usually good for finding out the bugs, but I take the fixes it provides with a grain of salt. There were numerous times where the fix could have lead to some more serious bugs because of some business logic special case scenarios that it didn't understand.
Yeah, I'm seeing this at work too. It's capable of adjusting behavior to meet some narrow success criteria, but it can easily introduce bigger problems in the process in large coupled code bases, which it doesn't catch on its own. Automated tests help a bit, but I find I still have to steer it very strongly at work.
It sounds like you understood what was happening here and used a tool effectively, nice.
That's very different from what a lot of vibe-coders do, like wholesale cloning and relicensing projects. This is happening in the GIS space, and these vibe-clone projects slightly hilariously inherit all the technical debt they didn't understand.
Nothing wrong with using LLMs, but I agree with Andrej Karpathy: "you can outsource your thinking, but not your understanding".
I think that quote is very apt. I experimented with making realistic video game grass (inspired by Ghost of Tsushima), and let it take the reins. But its solution became very contrived and hard to steer in the direction I wanted without me learning the specifics of how grass rendering should work. Still, it seems that I can limit my understanding to the higher levels of abstractions in most cases now. For example, I made it clump grass together according to a voronoi diagram, yet I have no idea how the voronoi algorithm works.
Who cares if it's good enough? Send the product, if people use it, it's good enough. If not, let users debut it and give "feedback", problem solved. You then advocate to these users, make them feel important, make them your cash cows, rinse and repeat.
Every team I have been on have been more successful when they gave up on the product ideals and just ensured that the customers were hooked and satisfied.
Me! I care! I want to use the things I create and be able to fix them when they break without relying on an intermediary simply because I'm too lazy to learn. It gives me a feeling of contented self-actualization and overall satisfaction with my choices in life.
Perhaps that is overrated and I should focus more on exploiting people with crappy software so that I can make more money. I guess that's what life is all about.
Too bad we live in a society where goodwill, self-actualization and satisfaction does not ensure survival or pay bills, but rather gives you a competitive disadvantage.
That mindset is why most software was already slop before AI. Users know when your team has turned into salespeople. And now that AI could enable you to do better in the same timeframe the response with this mindset is of course to care even less.
More than one service has lost me as a customer due to this process. I don't want a company to use me to verify their slop.
> Who cares if it's good enough? Send the product, if people use it, it's good enough. If not, let users debut it and give "feedback", problem solved.
I care, because doing a sloppy job and seeing if people will tolerate it is not acceptable engineering practice. It's not engineering at all. More pragmatically, if you do the thing you are suggesting, you won't have users for much longer. They will abandon you and go to someone who actually makes good products.
I don't believe this in practice. I am at a FAANG-like firm, and we are just maxxing the engagement and people use the product more and more the more we serve stuff to keep them hooked using their data and behaviour.
The quality of what we provide seems to matter 0, we are up about 40% issues in prod year-to-year but the users and their engagement is also growing.
I think there is a small subset of technical nerd/idealists who think that quality is all that matters but they are in their own little sandbox while 97% of users are in another one.
> I think there is a small subset of technical nerd/idealists who think that quality is all that matters
This is just a derogatory comment.
It is exactly this kind of arrogance that is killing open source. In my area, GIS, there are foundational projects that are necessarily intricate. For example it might be necessary to make a four-dimensonal transform of a location, that takes into account point movement due to the Glacial Isostatic Adjustment. There are hundreds of other complex examples I could come up with.
The libraries are intricate and quality does matter.
Unfortunately the lynchpin maintainer has just gone on indefinite strike, partly because open-source vibe-coders are wholesale copying and relicensing some of these libraries in their quest to ... do what?
I actually I don't know what they want. Take all the oxygen from the room as far as I can tell.
Well I think both the maintainer and the people copying the software will realize that the code has 0 value, but the community and peoples attention has some value.
The oxygen on the room here are peoples ego. Use your tool and interact with your community while you have their attention.
> I am at a FAANG-like firm
Also known as not-FAANG
> The quality of what we provide seems to matter 0
Good for you! There are lots of low skill development jobs before AI where low knowledge coders cranked out WordPress sites for marketing, advertising, and small businesses. Sounds like you got in on the ground floor!
Meanwhile companies like Amazon, one of the As in actual FAANG, do have demands on their software quality. To the extent they essentially said "knock off the tokenmaxxing productivity theater" in a recent, publicized internal memo.
Horses for courses. Sometimes you need an thoroughbred and sometimes a pack mule.
And there's nothing wrong with being a pack mule! And nobody expects a pack mule to understand the work of a thoroughbred! Nobody expects too much from a jackass after all.
You are attacking the person rather than the idea.
> The quality of what we provide seems to matter 0, we are up about 40% issues in prod year-to-year but the users and their engagement is also growing.
Sorry for going into a tangent here, but I'm actually unsure if I hate the structures and economic incentives that makes this possible, or if I hate you specifically for defending something so rotten
Next time I rage at some broken UI in some damn service I'm forced to use because that's where people are, I'll remember this comment
I am not defending anything here, I am not happy about the status quo, I just know that the moment we stop doing this, another company will just do it and take the market share because the incentives are made this way. We already saw it happen before.
It's all incentives. Blaming the GP for their slop is like blaming your labrador for getting fat when you overfeed it.
In the narrow case of your profession perhaps. Beyond the slipshod world of most software is a universe of standards, licenses and liability.
The lawyers and judges who trusted an ML system to be up to snuff are currently pounding sand.
I worked for one of the largest device manufacturers for a certain kind of sensor.
The requirements/standards for this sensor was made by this company 20 years ago. All the tests and requirements were made by this same company in a way that would minimize the costs for us to keep making new revision we could sell, and make it the hardest for any competitor to enter the market. Another competitor started with a better product abroad, so we copied it, but it didn't pass the tests, so then they installed an owner of the certification house that was a cousin of one of the VPs. No problems passing tests after that.
You think this is all controlled and nice and standardized but it's all just a market from software to hardware.
While quite believable, there is a diversity of industries and standard adherence / enforcement no?
Markets also range from under regulated to over regulated, and are usually in a transition from one end to another (while contending with other forces).
Why would Prime be in that list. He had an entire arc where he tried to make it work and then went back to the drawing board.
I am not saying he writes all code by hand, of course. Just that he is furthest in figuring out some sort of mid point between full comprehension and cognitive surrender.
I have missed the more recent content, so perhaps I have missed something, but this was my read last I watched.
I'm also not sure why anyone would attach importance to his opinions as a software engineer, he's exceptional in generating entertaining content but as an engineer he does not seem above average. Definitely not even close to the category of Jonathan Blow or Andrej Karparthy