So reading this, I know it’s probably a human posing as an LLM. But the problem with formats like this is that I don’t actually know.
If a human said this to me in real life, and I laughed, it would probably help build a connection with that person, as it’s signaling that we have in common an unusually strong grasp of the patterns in LLM output (unusually strong at least in comparison to the general population).
But here? I don’t gain anything by reading this comment. It only contributes to the uncertainty that anything I read on this site has any meaning at all.
Please, if you’re a human, don’t mimic LLMs on forums where the reader cannot distinguish you from an LLM without doing investigative work.
And you’re an LLM, please report to your owner that he is an ass for polluting one of the last bastions of high entropy discussion on the information superhighway.
I only add /s if it's not obvious, and someone might take it seriously. Although online you never know, too often people take seriously even the most obvious.
Without looking, I’m guessing one of these was cherry-picked from Amis and one was plucked nearly at random from LinkedIn.
The writing style AI uses has its place, but not as _every sentence_. That’s what is exasperating. At the same time, I’m happy that I can still at least identify AI prose of more-than-trivial length.
Simple sentences are better then Martin Amis whoever it is. The issue with ai writing is using around seven million of useless words where 7 would be sufficient and clear.
> LLMs aren’t lazy. They don’t cut corners because a simpler solution feels good enough. If they know how to solve something thoroughly, they will.
I don't know why they think this, but no? Perhaps it's badly expressed, but LLMs cut corners all the time. It's sort of their core fault really.
Anyway, I disagree with the core premise[1]. Re-writes are not cheap, because 1) code can be so bad it's unclear how to rewrite it[2], b) code can be so significant it's challenging to rewrite it (architectural choices, schemas, etc), and lastly if you're ever wanting to own your own code and not rent it from LLM companies, having it understandable by a human is still a goal worth working toward.
[1] To be fair, I think OP _might_ be talking about rewriting in the moment of the thing being built, but with some unspoken rule that once they think the change is good enough, then they are reviewing all the code? They don't make it clear..
[2] it's not even that hard. Write a test that exercises an end result and not the rule that causes that end result and 6 months later you've forgotten why the code is like that. I had to maintain a piece of software once where the primary form of tests were a bunch of snapshots of an end report being generated, based on some initial data input, mostly all unlabeled. The code was like "do this SQL query on table A and then take the second result". Why the second result? Your guess is as good as mine! I couldn't even work out why they were querying table A and not table B...
> I don't know why they think this, but no? Perhaps it's badly expressed, but LLMs cut corners all the time. It's sort of their core fault really.
I think this is a matter of perspective about what counts as "cutting corners".
I think they look like you describe only because they have limited competence; this is on the basis that when I asked one to make a fusion reactor simulator (to see if it could) by using open source plasma physics libraries which it had itself suggested at the start of this process (e.g. WarpX), it didn't take the "lazy" option of actually using those libraries, it tried to write its own plasma physics simulation from scratch "as a fallback if we can't install the libraries".
As it was not sufficiently competent, the resulting "simulator" was hilariously wildly unphysical.
Would have been much better if it had been lazy.
To your deeper point: I agree. In this case it made so much of a mess that there was no point trying to rescue it; as it was done from scratch, throwing it away and starting again was fine*, but if this had been pushing commits that got interleaved with real work on the main branch of an existing project it would have been a serious issue.
* or in my case, not even bothering to start again. Like I said, this was done to see if it could.
The diligence to the absurd that you describe is really just a consequence of cutting corners at planning stages. Consider an organization that relies heavily on formal proofs and specifications to one that uses what fits on the 3 bullets allowed in a PowerPoint slide. The first might do less work over all because the second is cutting corners in planning.
Perhaps, though I find myself disagreeing as the planning stage where it itself told me about WarpX. It just then made stupid decisions when left to itself.
Can plan… just not well. Can code… just ok. Can do TDD… just most of the time.
Definitely none of these well enough to be a dark factory though.
Yeah, well.. I think a lot of engineers don't have reason to see the bulk of the industry but if you took an exactly median manager, programmer or PM and ask them a few things you also tend to find that they don't even notice that they will contradict themselves in their answer to a second question, etc. Organizations in general are harnesses that try to get some kind of result out of some pretty sloppy attention.
> Organizations in general are harnesses that try to get some kind of result out of some pretty sloppy attention.
Certainly.
LLMs have superhuman attention, and are so cheap that they can be left running for absurdly long outputs. Despite this, still not high on the competence scale (yet*).
A human doesn't output a few million tokens every day, not even counting our inner monologues, but the AI asymptote to an upper bound of quality somewhat worse than a senior human.
* I would like to have a new job before they reach this point. Got a mortgage to repay.
> I think this is a matter of perspective about what counts as "cutting corners".
The nature of probabilistic sampling practically guarantees that corner cutting is always just a few samples away. Certain sampling strategies can mitigate this, but there's no way to fully eliminate it, without fully eliminating it from the training data and guarding against it during training. Model reasoning can help by giving the model space to draft and review its approach before it executes, but models still aren't guaranteed to follow their own thinking. A mistake or shortcut can always simply slip in during generation and it won't always be caught and corrected.
- Good logits and sampling strategy can make cases like those exceptionally unlikely -- sufficiently so for one to assume it won't realistically happen.
- Once a bad path is sufficiently taken, it tends to be a lot more likely to continue.
This leads to real-world advice:
- If a model refuses your request, do not argue with the refusal; edit your original message or otherwise regenerate -- the presence of refusal tells the model it should continue refusing.
- More generally, don't allow the model context to get contaminated with behavior or commands you don't like -- describing what to do is more effective than describing what NOT to do.
- It's rude to push unreviewed model outputs onto others.
I'm not saying corner-cutting is a thing that is necessarily all over the place (even though there are countless examples in the wild). I'm also not saying it always results in random stops, or that doing one thing bad makes everything else bad. What I'm saying is that bad decisions could be hidden anywhere, at any time, even if everything else looks fine. Such is the nature of current LLMs.
> - Once a bad path is sufficiently taken, it tends to be a lot more likely to continue.
no they demonstrably self correct with injected bad tokens
> - If a model refuses your request, do not argue with the refusal; edit your original message or otherwise regenerate -- the presence of refusal tells the model it should continue refusing.
wish i could do that with humans :P
> - It's rude to push unreviewed model outputs onto others.
Yes but that's not why.
Same real-world advice applies to stuff random fresh graduates make. I remember being one of those.
How on earth are you going to move the needle on "honesty or effort put forth by an LLM" at the logit-choosing stage, though?
To me that's like inferring that you can keep a GPS system from choosing suboptimal cross-country routes by affixing a camera to the grill of the car with sufficient visibility to examine the next fifty feet of pavement.
indeed. That being said, there is a psychological effect of reviews that doesn't exists with LLM. One of the thing that make reviews effective is that when we code, we know someone will look at our code and judge it. It might be subconscious, but it's there.
We faced a lot of this this year. Eager new joinees sent 100s of lines of CLs to review to "improve" little things.
Not only did the new changes did not fix what they thought it would fix but it broke other things in unexpected ways.
I brought in two changes after that:
* I'm not reviewing/reading anything that you yourself did not read / test in the target environments properly. If all it takes is an LLM prompt, I could be issuing the same prompt to make my life easier.. and If you're sending a CL, you should be owning the code you send.
* Me being more involved in the design process so review burden itself becomes lower. A bit of pair programming from time to time helped too.
Not sure how things will turn out after this but so far they seem better.
For those curious what CL is (I had to look it up myself): it means change list[1]. Guessed from context that it was like "diff in PR/MR" or something but hadn't seen this acronym before
> The shortest path for the model is to implement it completely
Have you worked with LLMs???????????
“I disabled the test so it’s not run so now all the tests pass” is not a hypothetical it’s pretty common. LLMs frequently do shortcut learning. The reason why reviews are expensive is because you still need to do all the steps in order to understand if a shortcut is justified.
> "I avoided implementing this correctly because of migration concern for existing installations of this code I'm writing right now"
This one grinds my gears so bad, probably 1/4 of my job at this point is telling an LLM (either in my own editor or in review comments for someone's MR) "how about we do this right _before_ merging it, eh?".
Or:
foo = abc
<Maybe 4-10 lines of code>
if foo != None:
...
In my experience giving it "rules" not to do this does nothing, but a separate pass (could be by a different model but really just fresh context is enough) does okay
All the global memories I have stored are basically just this. Memories appear to be completely useless on their own, but, before a merger or a plan is finalized, having it go read them will definitely help course correct.
I really wish there was an "every x tokens" type trigger for every agent, where I could have it fire off a "Pause, go read the guide to make sure you're adhering fully." To help keep concepts fresh in context.
I used Fable to write a relatively small RPG. In the span of 2 hours it managed to do many interesting things. My favorite was when it wrote code with a race condition that could cause people to take more damage than they should, which it then defended as an acceptable tradeoff for parallelism.
> An LLM defaults to building when it should be buying. Not because it doesn’t know about existing libraries, it often mentions them, but because for an LLM, writing two hundred lines of implementation is the same cognitive effort as writing two lines of import.
Or maybe they are trained that way. It’s more tokens used and more money you need to pay.
I’ve personally reached a point where if I’m saving time typing, but gaining back that time on reviewing and understanding, I may as well have just written the code most of the time. AI is great for boilerplate and learning, but my team has given up on trying to use the tool for entire implementations
Rewrites are only cheap if the rewrite solves the problem.
And the rewrite can only reliably solve the problem if you understand the problem, and even then it's obviously not a guarantee.
If you have a huge blob of code that nobody understands then re-generating a new blob of code that nobody understands is unlikely to solve the problem.
That said: I've always been a fan of optimistic rather than pessimistic merging, at least for human-generated code.
I’m not sure I agree with this or maybe I don’t understand. In my experience, the over engineered code LLMs create have more big problems. Rewriting vast parts of code when I have an outage or need a new feature means the code evolves far faster than my understanding. That gets more and more dangerous. Or maybe I’m not smart enough to follow the new pace?
Do they, though? AWS and CloudFlare recently had the worst outages in their history, and GitHub is flirting with zero nines of uptime these days. Doesn’t seem to have cost them any business.
An outage is the best failure that can happen. The worst that can happen depends on the system but could be anything from a business collapsing because of an attack, your production database being stolen and wiped, all the way to someone dying.
Shouldn't we really consider review and rewrite together? If so, the economics may not change as drastically[1]. It's just like writing code by hand: we spend lots of time organizing our thoughts and examining our code, which is not that different from reviewing.
[1] Unless you're an engineer in Anthropic, so you just spend you time writing "loop".
Say you discover in review that your agent wrote code before tests. A file that was added, no corresponding test file.
Tell the agent to read in the file, delete it (or roll back), then reimplement using red/green TDD.
Claude can test-drive an identical reimplementation but this time by writing red tests and making them green. Now you have coverage that you KNOW covers the code because you watched those tests fail before they passed.
Going further I added a note to my `/review` command so the agent looks for this mistake for me in its self-review. If it finds uncovered code it makes a plan which includes the delete-reimplement trick. Now all I do is approve that plan - no hunting needed, no explaining, no repeating myself.
You could even put this in a Ralph Wiggum loop and give yourself super high quality tests by going file by file.
LLM generated code does have some advantages at code review time. I find myself paying more attention to the diffs being in the right areas than their actual line-by-line content. It's been a long time since I've seen something like a conditional hallucinated incorrectly. The accuracy in the details is exceptional now. It's mostly a matter of ensuring we are dealing with the correct details.
The compiler does a lot of heavy lifting in the codebases I work with. Strongly typed languages seem mandatory if you want to use an LLM. Worrying about every symbol is too much. Let the CPU get hot for a few seconds on the paranoia and stay focused on the minimap.
It's difficult to spot issues from a huge diff.
But when a agent finish a task, it remembers what it just went through, where it got stuck, when it got corrected by human, what code make it want to say the F word to the author, that would be a good chance to make the codebase cleaner.
I got to see greptile and it had a pretty decent code review, somewhat like a static analysis tool without a lot of time wasting nonsense/false positives.
When I've used static analysis tools, the first run is usually helpful as you cherry pick the things that need to be fixed, but then subsequent runs are just the false positives or "only slightly a nit" kind of annoyances.
But human developers are the ones that say stuff like "Do we really have to use a database at all?" etc...
LLM's are good at some types of reviews and awful at others. They generally tend to overcomplicate things and miss opportunities to simplify. They pretty much have to take pre-existing code and tests as gospel and cannot distinguish which is buggy, incomplete, unimportant or important. They have no knowledge of unwritten business requirements, customer preferences, et cetera so high level review is always necessary.
Yeah ok but that’s what I have. I have the knowledge of unwritten business requirements and customer preferences.
My knowledge + Claude is much better than just my knowledge.
I don’t know why it is considered strange to use both tools (my knowledge plus Claude) to make reviews go quicker. In fact it is highly dogmatic to not use Claude while reviewing.
I also like having long, pointed conversations with LLMs as I review code. Then when I'm done, it's different code, and it has all of my blind spots and knowledge gaps, so I cannot effectively review it anymore.
It's like turning a code review that requests you, into a code review that requests someone else. And it tramples on the original author quite a bit too. It's hard only having the ability to add incremental value to large amounts of code, instead of large amounts of value to incremental code.
Writing blog posts has become cheap, making them sound human has become hard.
The simple sentences LLM keep generating break my brain, it's like 95% of writing is now 3rd grade level.
Compare that to e.g. Martin Amis: https://en.wikiquote.org/wiki/Martin_Amis
That's not just you imagining things, that's the world around you changing. That's real and it matters.
So reading this, I know it’s probably a human posing as an LLM. But the problem with formats like this is that I don’t actually know.
If a human said this to me in real life, and I laughed, it would probably help build a connection with that person, as it’s signaling that we have in common an unusually strong grasp of the patterns in LLM output (unusually strong at least in comparison to the general population).
But here? I don’t gain anything by reading this comment. It only contributes to the uncertainty that anything I read on this site has any meaning at all.
Please, if you’re a human, don’t mimic LLMs on forums where the reader cannot distinguish you from an LLM without doing investigative work.
And you’re an LLM, please report to your owner that he is an ass for polluting one of the last bastions of high entropy discussion on the information superhighway.
In context it's funny, and that's what counts.
This is how I really talk. LLMs are the one copying me.
Not just copying, redefining. Slick. Conventional. Marketable.
most HN comments are way dumber then the average LLM take
You could even argue that it’s the load-bearing point!
Please add a /s if appropriate.
Takes all the fun out of it.
I only add /s if it's not obvious, and someone might take it seriously. Although online you never know, too often people take seriously even the most obvious.
Martin Amis level prose is neither possible nor desirable for a technical blog post.
I don't think that there is anything wrong with having stylistic depth in any wriitng
Why?
> Style is not neutral; it gives moral directions.
> Nowadays every business in America says how warm it is and how much it cares — loan companies, supermarkets, hamburger chains.
Guess which one is AI and which one is a quote from Martin Amis.
Without looking, I’m guessing one of these was cherry-picked from Amis and one was plucked nearly at random from LinkedIn.
The writing style AI uses has its place, but not as _every sentence_. That’s what is exasperating. At the same time, I’m happy that I can still at least identify AI prose of more-than-trivial length.
both are Amis
Simple sentences are better then Martin Amis whoever it is. The issue with ai writing is using around seven million of useless words where 7 would be sufficient and clear.
> LLMs aren’t lazy. They don’t cut corners because a simpler solution feels good enough. If they know how to solve something thoroughly, they will.
I don't know why they think this, but no? Perhaps it's badly expressed, but LLMs cut corners all the time. It's sort of their core fault really.
Anyway, I disagree with the core premise[1]. Re-writes are not cheap, because 1) code can be so bad it's unclear how to rewrite it[2], b) code can be so significant it's challenging to rewrite it (architectural choices, schemas, etc), and lastly if you're ever wanting to own your own code and not rent it from LLM companies, having it understandable by a human is still a goal worth working toward.
[1] To be fair, I think OP _might_ be talking about rewriting in the moment of the thing being built, but with some unspoken rule that once they think the change is good enough, then they are reviewing all the code? They don't make it clear..
[2] it's not even that hard. Write a test that exercises an end result and not the rule that causes that end result and 6 months later you've forgotten why the code is like that. I had to maintain a piece of software once where the primary form of tests were a bunch of snapshots of an end report being generated, based on some initial data input, mostly all unlabeled. The code was like "do this SQL query on table A and then take the second result". Why the second result? Your guess is as good as mine! I couldn't even work out why they were querying table A and not table B...
> I don't know why they think this, but no? Perhaps it's badly expressed, but LLMs cut corners all the time. It's sort of their core fault really.
I think this is a matter of perspective about what counts as "cutting corners".
I think they look like you describe only because they have limited competence; this is on the basis that when I asked one to make a fusion reactor simulator (to see if it could) by using open source plasma physics libraries which it had itself suggested at the start of this process (e.g. WarpX), it didn't take the "lazy" option of actually using those libraries, it tried to write its own plasma physics simulation from scratch "as a fallback if we can't install the libraries".
As it was not sufficiently competent, the resulting "simulator" was hilariously wildly unphysical.
Would have been much better if it had been lazy.
To your deeper point: I agree. In this case it made so much of a mess that there was no point trying to rescue it; as it was done from scratch, throwing it away and starting again was fine*, but if this had been pushing commits that got interleaved with real work on the main branch of an existing project it would have been a serious issue.
* or in my case, not even bothering to start again. Like I said, this was done to see if it could.
The diligence to the absurd that you describe is really just a consequence of cutting corners at planning stages. Consider an organization that relies heavily on formal proofs and specifications to one that uses what fits on the 3 bullets allowed in a PowerPoint slide. The first might do less work over all because the second is cutting corners in planning.
Perhaps, though I find myself disagreeing as the planning stage where it itself told me about WarpX. It just then made stupid decisions when left to itself.
Can plan… just not well. Can code… just ok. Can do TDD… just most of the time.
Definitely none of these well enough to be a dark factory though.
Yeah, well.. I think a lot of engineers don't have reason to see the bulk of the industry but if you took an exactly median manager, programmer or PM and ask them a few things you also tend to find that they don't even notice that they will contradict themselves in their answer to a second question, etc. Organizations in general are harnesses that try to get some kind of result out of some pretty sloppy attention.
> Organizations in general are harnesses that try to get some kind of result out of some pretty sloppy attention.
Certainly.
LLMs have superhuman attention, and are so cheap that they can be left running for absurdly long outputs. Despite this, still not high on the competence scale (yet*).
A human doesn't output a few million tokens every day, not even counting our inner monologues, but the AI asymptote to an upper bound of quality somewhat worse than a senior human.
* I would like to have a new job before they reach this point. Got a mortgage to repay.
> I think this is a matter of perspective about what counts as "cutting corners".
The nature of probabilistic sampling practically guarantees that corner cutting is always just a few samples away. Certain sampling strategies can mitigate this, but there's no way to fully eliminate it, without fully eliminating it from the training data and guarding against it during training. Model reasoning can help by giving the model space to draft and review its approach before it executes, but models still aren't guaranteed to follow their own thinking. A mistake or shortcut can always simply slip in during generation and it won't always be caught and corrected.
> practically guarantees that corner cutting is always just a few samples away.
If it were that bad, 100% of chat with AI would look like this comment I'm writing
Two things:
- Good logits and sampling strategy can make cases like those exceptionally unlikely -- sufficiently so for one to assume it won't realistically happen.
- Once a bad path is sufficiently taken, it tends to be a lot more likely to continue.
This leads to real-world advice:
- If a model refuses your request, do not argue with the refusal; edit your original message or otherwise regenerate -- the presence of refusal tells the model it should continue refusing.
- More generally, don't allow the model context to get contaminated with behavior or commands you don't like -- describing what to do is more effective than describing what NOT to do.
- It's rude to push unreviewed model outputs onto others.
I'm not saying corner-cutting is a thing that is necessarily all over the place (even though there are countless examples in the wild). I'm also not saying it always results in random stops, or that doing one thing bad makes everything else bad. What I'm saying is that bad decisions could be hidden anywhere, at any time, even if everything else looks fine. Such is the nature of current LLMs.
> - Once a bad path is sufficiently taken, it tends to be a lot more likely to continue.
no they demonstrably self correct with injected bad tokens
> - If a model refuses your request, do not argue with the refusal; edit your original message or otherwise regenerate -- the presence of refusal tells the model it should continue refusing.
wish i could do that with humans :P
> - It's rude to push unreviewed model outputs onto others.
Yes but that's not why.
Same real-world advice applies to stuff random fresh graduates make. I remember being one of those.
The incompetence is why.
How on earth are you going to move the needle on "honesty or effort put forth by an LLM" at the logit-choosing stage, though?
To me that's like inferring that you can keep a GPS system from choosing suboptimal cross-country routes by affixing a camera to the grill of the car with sufficient visibility to examine the next fifty feet of pavement.
indeed. That being said, there is a psychological effect of reviews that doesn't exists with LLM. One of the thing that make reviews effective is that when we code, we know someone will look at our code and judge it. It might be subconscious, but it's there.
We faced a lot of this this year. Eager new joinees sent 100s of lines of CLs to review to "improve" little things.
Not only did the new changes did not fix what they thought it would fix but it broke other things in unexpected ways.
I brought in two changes after that:
* I'm not reviewing/reading anything that you yourself did not read / test in the target environments properly. If all it takes is an LLM prompt, I could be issuing the same prompt to make my life easier.. and If you're sending a CL, you should be owning the code you send.
* Me being more involved in the design process so review burden itself becomes lower. A bit of pair programming from time to time helped too.
Not sure how things will turn out after this but so far they seem better.
For those curious what CL is (I had to look it up myself): it means change list[1]. Guessed from context that it was like "diff in PR/MR" or something but hadn't seen this acronym before
[1]: https://stackoverflow.com/a/27520705/2805120
> The shortest path for the model is to implement it completely
Have you worked with LLMs??????????? “I disabled the test so it’s not run so now all the tests pass” is not a hypothetical it’s pretty common. LLMs frequently do shortcut learning. The reason why reviews are expensive is because you still need to do all the steps in order to understand if a shortcut is justified.
> “I disabled the test so it’s not run so now all the tests pass”
Also:
"I implemented it this terrible way because of precedence in the codebase...that I just wrote"
"I avoided implementing this correctly because of migration concern for existing installations of this code I'm writing right now"
"I deferred this critical feature for the future, so we can deploy quicker"
or, my favorite,
"I hand rolled an buggy http server because you said the tool should be self contained"
Or bizarrely Claude askes about code churn like it matters.
Human, Do you want me to do it the right way? It will cause code churn in 90 files. Or I can take a shortcut and edit 3 files in a terrible way.
Edits 90 files for 12 lines each in 25 seconds...
“This is a significant rewrite that will take weeks”
Done after my potty break
> "I avoided implementing this correctly because of migration concern for existing installations of this code I'm writing right now"
This one grinds my gears so bad, probably 1/4 of my job at this point is telling an LLM (either in my own editor or in review comments for someone's MR) "how about we do this right _before_ merging it, eh?".
Or:
foo = abc
<Maybe 4-10 lines of code>
if foo != None:
In my experience giving it "rules" not to do this does nothing, but a separate pass (could be by a different model but really just fresh context is enough) does okayAll the global memories I have stored are basically just this. Memories appear to be completely useless on their own, but, before a merger or a plan is finalized, having it go read them will definitely help course correct.
I really wish there was an "every x tokens" type trigger for every agent, where I could have it fire off a "Pause, go read the guide to make sure you're adhering fully." To help keep concepts fresh in context.
I used Fable to write a relatively small RPG. In the span of 2 hours it managed to do many interesting things. My favorite was when it wrote code with a race condition that could cause people to take more damage than they should, which it then defended as an acceptable tradeoff for parallelism.
> “I disabled the test so it’s not run so now all the tests pass” is not a hypothetical it’s pretty common.
I've never seen that though.
If your plan is to not review and just have the LLM rewrite if it doesn’t work, it doesn’t sound like the rewrite is gonna be any better.
> An LLM defaults to building when it should be buying. Not because it doesn’t know about existing libraries, it often mentions them, but because for an LLM, writing two hundred lines of implementation is the same cognitive effort as writing two lines of import.
Or maybe they are trained that way. It’s more tokens used and more money you need to pay.
I’ve personally reached a point where if I’m saving time typing, but gaining back that time on reviewing and understanding, I may as well have just written the code most of the time. AI is great for boilerplate and learning, but my team has given up on trying to use the tool for entire implementations
Rewrites are only cheap if the rewrite solves the problem.
And the rewrite can only reliably solve the problem if you understand the problem, and even then it's obviously not a guarantee.
If you have a huge blob of code that nobody understands then re-generating a new blob of code that nobody understands is unlikely to solve the problem.
That said: I've always been a fan of optimistic rather than pessimistic merging, at least for human-generated code.
Thank god for Strangler Fig.
An incomprehensible source of truth is still a source of truth.
True.
Alas in incomprehensible source of false is not a source of truth.
¯\_(ツ)_/¯
> LLMs aren’t lazy. They don’t cut corners because a simpler solution feels good enough. If they know how to solve something thoroughly, they will.
This is a severe misunderstanding in how LLMs work..
I don't know how this got on my front page....
I’m not sure I agree with this or maybe I don’t understand. In my experience, the over engineered code LLMs create have more big problems. Rewriting vast parts of code when I have an outage or need a new feature means the code evolves far faster than my understanding. That gets more and more dangerous. Or maybe I’m not smart enough to follow the new pace?
AFAICT, the author is talking about rewriting code during a review as part of the review process.
quote: "If I identify code that’s more complex than it needs to be, in my own work or in someone else’s PR"
If so, that makes a lot of sense to me. The best time to rewrite code is before it hits production.
Failures in production remain expensive.
Do they, though? AWS and CloudFlare recently had the worst outages in their history, and GitHub is flirting with zero nines of uptime these days. Doesn’t seem to have cost them any business.
An outage is the best failure that can happen. The worst that can happen depends on the system but could be anything from a business collapsing because of an attack, your production database being stolen and wiped, all the way to someone dying.
I wish they were. Displaying an error message to the user, asking to try again later, is way too cheap in my opinion.
Shouldn't we really consider review and rewrite together? If so, the economics may not change as drastically[1]. It's just like writing code by hand: we spend lots of time organizing our thoughts and examining our code, which is not that different from reviewing.
[1] Unless you're an engineer in Anthropic, so you just spend you time writing "loop".
This is very handy.
Say you discover in review that your agent wrote code before tests. A file that was added, no corresponding test file.
Tell the agent to read in the file, delete it (or roll back), then reimplement using red/green TDD.
Claude can test-drive an identical reimplementation but this time by writing red tests and making them green. Now you have coverage that you KNOW covers the code because you watched those tests fail before they passed.
Going further I added a note to my `/review` command so the agent looks for this mistake for me in its self-review. If it finds uncovered code it makes a plan which includes the delete-reimplement trick. Now all I do is approve that plan - no hunting needed, no explaining, no repeating myself.
You could even put this in a Ralph Wiggum loop and give yourself super high quality tests by going file by file.
LLM generated code does have some advantages at code review time. I find myself paying more attention to the diffs being in the right areas than their actual line-by-line content. It's been a long time since I've seen something like a conditional hallucinated incorrectly. The accuracy in the details is exceptional now. It's mostly a matter of ensuring we are dealing with the correct details.
The compiler does a lot of heavy lifting in the codebases I work with. Strongly typed languages seem mandatory if you want to use an LLM. Worrying about every symbol is too much. Let the CPU get hot for a few seconds on the paranoia and stay focused on the minimap.
ai can do some of the reviewing, checking calling and called arguments, even things like crufty shell scripts.
but the higher-level "should you do this?" or "check your design" - could AI do that stuff?
I think the question is now "should you care?" And it seems the magnificent, incorruptible thought leaders of our time are all converging on "No"
It's difficult to spot issues from a huge diff. But when a agent finish a task, it remembers what it just went through, where it got stuck, when it got corrected by human, what code make it want to say the F word to the author, that would be a good chance to make the codebase cleaner.
> ai can do some of the reviewing
No way this can backfire.
> checking calling and called arguments
Like a static language compiler already does?
I got to see greptile and it had a pretty decent code review, somewhat like a static analysis tool without a lot of time wasting nonsense/false positives.
When I've used static analysis tools, the first run is usually helpful as you cherry pick the things that need to be fixed, but then subsequent runs are just the false positives or "only slightly a nit" kind of annoyances.
But human developers are the ones that say stuff like "Do we really have to use a database at all?" etc...
IMO it can, about as well as an entry/mid level dev
The title sounds a bit like a catch-22: ok, rewrites are now cheap, but rewrites have to be reviewed too?
This makes me think of an LLM induced summer of endless rewrites where you get to experience a random reroll of what 80% works on any given rewrite.
As explained by a LLM.
Why is reviewing hard? I use LLMs for reviewing. It is dogmatic to review every line written by an LLM.
LLM's are good at some types of reviews and awful at others. They generally tend to overcomplicate things and miss opportunities to simplify. They pretty much have to take pre-existing code and tests as gospel and cannot distinguish which is buggy, incomplete, unimportant or important. They have no knowledge of unwritten business requirements, customer preferences, et cetera so high level review is always necessary.
Yeah ok but that’s what I have. I have the knowledge of unwritten business requirements and customer preferences.
My knowledge + Claude is much better than just my knowledge.
I don’t know why it is considered strange to use both tools (my knowledge plus Claude) to make reviews go quicker. In fact it is highly dogmatic to not use Claude while reviewing.
I also like having long, pointed conversations with LLMs as I review code. Then when I'm done, it's different code, and it has all of my blind spots and knowledge gaps, so I cannot effectively review it anymore.
It's like turning a code review that requests you, into a code review that requests someone else. And it tramples on the original author quite a bit too. It's hard only having the ability to add incremental value to large amounts of code, instead of large amounts of value to incremental code.
I'm confused - are you purposefully pretending that the author isn't talking about human review?
What kind of systems do you work on? Does it have production traffic? Is there a cost to downtime?
always true to your name
you arnt reviewing. youre playing loophole semantics.