I'm sure this method has evolved and/or been supplanted over the last 15 years, but one thing that struck me reading this is how much the dynamics of unit test coverage have changed in recent history, with AI-generated commits containing 10x as many unit tests (many of them kind of silly and tautological) as in the olden days. Gonna need to update some of those coefficients in their CRAP1 formula... Or maybe test coverage has/will become too noisy a parameter to use at all.
Humans have been hand-crafting artisanal test cases of equal banality for years, often to meet some metric no one is verifying deeper than an automated report etc. I'm more inclined to believe the null hypothesis here, that the formula doesn't need any change and the signal is still equally valid. If anything changed, the signal is probably more useful now, since you'll never catch a good AI writing test cases that do literally nothing except call a function in a try catch with an empty catch block. (Models may have done that a few years ago but I haven't witnessed that behavior in a long time.) I've witnessed myriad examples of this in pre-AI production code.
Anecdotally, I’ve found that codebases that enforce code coverage metrics often have worse behavior coverage than ones that don’t.
It’s a classic example of Goodhart’s Law in action. Code coverage metrics only measure what percentage of code the test suite causes to run. But it’s very, very easy to write tests that run code without actually confirming that it produces correct output for all possible inputs. And it’s very, very easy to assume that a module with 90+% code coverage also has 90+% behavior coverage, and then become complacent about reviewing the suite for proper behavior coverage.
Djikstra seemed like he was mostly against testing. But only because he was for proofs. A unit test is a single example. The real way to demonstrate the absence of errors is to prove they aren’t there (vis a vis axioms and assumptions).
But most developers don’t have the mathematical sophistication nor the time.
It’s not that unit testing is useless. Just good to know what their limitations are and to plan your testing strategy accordingly.
You also have the problem of potentially having to re-verify everything by hand for every little change. Maybe fine for the kinds of projects Dijkstra was working on, but less practical in a business setting.
Tools like QuickCheck and Hypothesis are an interesting middle ground, though. I strongly prefer them over standard-issue unit testing for verifying algorithm implementations.
Yeah. And TLA+ can confirm that the design is sound, but it can’t confirm that the implementation conforms to the design. QuickCheck style tests can’t solve that problem, but perhaps they can mitigate it.
I sometimes use unit tests to fill in gaps. I just caught a function with almost full coverage except for an if statement.
Playwright e2e tests are harder to maintain than pytest code, so I wrote a test just for that one case, and while I was at it, a few other input strings that seemed like they could be problems.
If the function had no test coverage at all, I probably would have tried to get it with an integration test, because that would mean the callers also weren't getting tested.
The occasional unit test is fine. But you do not need for example `test_helper_test.go` that is just ridiculous.
Its gotten so bad I decided to throw the baby out with the bathwater so to speak and focus on the high value places instead of trying adjudicate every individual usage
I like having both. Unit tests are a decent indicator for local development. Fast, quick to iterate on. Integration tests are slow as molasses and I can’t iterate with them when it takes 20 minutes to setup the suite and run everything. Too slow. But still useful as you say.
A measure is only good if I take action on it and in turn make things better. There are a lot of things that are easy to measure, but there is no useful action I should take on the measure.
Yes, but also all too often “useful” is interpreted to mean “moves the metric”. If that metric is merely a proxy for some more tangible outcome then that may not be good enough.
The one that tech tends to stumble on most often is velocity-type metrics. The problem there is that you can’t pay the bills with velocity. And velocity metrics tend to favor cheap shovelware features that cohere poorly over anything that involves having the team slow down on churning out code long enough to work out elegant solutions to subtle problems.
>Note: This post is rated PG-13 for use of a mild expletive. If you are likely to be offended by the repeated use a word commonly heard in elementary school playgrounds, please don’t read any further.
Mild as this ironic passive aggressiveness is, can't imagine something like this in modern sterile corporate messaging.
"repeated use" seems to be doing the mitigation work here, though it does seem unusual that someone offended by the repetition would be unoffended by a one-off.
I have a goal to make the codebase at work cargo-crap compliant and enforce it with CI. I let an agent run overnight with it once and the diff touched like 40% of our codebase which is untenable for a single merge. So for now I’m doing it piecemeal as the opportunity presents itself.
The pendulum has swung too far in the direction of class, function, cyclomatic complexity (and here, CRAP) and similar idiotic metrics.
This reminds me of a talk Sandi Metz did called "All the Little Things" where she covers the Gilded Rose kata. In the talk, she reworks her solution until there's almost nothing left showing the essence of the problem being solved.
The cyclomatic complexity metric is touted at each step as a proxy for goodness of design and removal of complexity. However, a weakness of the measure itself is that it doesn't account for the control flow indirection that happens through OO method dispatch itself.
At the same time, Kevlin Henney's talk called "Gilding the Rose" takes the same kata and arrives at a far more sane solution he works up to and reveals at the end.
Short functions used to be hot. Uncle Bob used to proselytize "The first rule of functions is that they should be short. The second rule of functions is that they should be shorter than that." Now emphasizing the benefits of longer functions is pretty trendy.
https://github.com/johnousterhout/aposd-vs-clean-code
This industry is pretty idiotic sometimes ¯\_(ツ)_/¯
A while back Hillel Wayne did a talk (whose name I forget) on what empirical evidence on software quality actually says.
As I recall, he concluded that there’s really no support for then-popular ideas like short functions, reducing cyclomatic complexity, avoiding explicit branch statements and loops, or TDD. (Tests yes, just not TDD.)
He made a pretty strong case that only two principles are particularly robust. One was that limiting code volume is good. The other is that working people too hard is bad.
>cyclomatic complexity metric is touted at each step as a proxy for goodness of design and removal of complexity. However, a weakness of the measure itself
Amen, it’s hard to push back against an opaque term (cyclomatic!) when it isn’t really a measure of goodness, it’s a measure of branching, kind of a normal thing in code.
Early on I found that code with low cyclomatic complexity was just usually extremely verbose, lots of passing this to that while avoiding the branching necessary to get something done.
And yes, you can game the metric by hiding the complexity among the confusion of objects and components.
Oh god, yes. The powers that be in my workplace are obsessed with cyclomatic complexity, so now we’re reviewing a flood of LLM diffs that take perfectly good code and extract each loop into its own function with a dozen arguments. The code is now more maintainable on paper and far, far less maintainable (by a human) in practice.
Reasonable cyclomatic complexity is useful (at least) for testability. No metric is perfect and no metric should be a primary goal, but it has its use, and if you read e.g. a bit of leaked Windows you certainly will understand why it matters (and not because it has a particularly low cyclomatic complexity...)
Of course if you introduce new methods of dispatch and do not take them into account into a metric, you end up with something less... precise? useful? But given the primary intent and why and how the metric was created this seems a pretty trivial observation.
Now I agree it is also retarded to attempt to get only very short functions or extremely low cyclomatic complexity everywhere (even if you try to adapt it to count new kind of dispatch), because the only effect that produce is that it moves the complexity in another more abstract place we are less well equipped to manage.
"Short functions used to be hot. Uncle Bob [...]": well yes, Internet and sometimes group of people inspired ultimately by Internet and group effects can be pretty idiotic, but honestly Uncle Bob ideology was never considered serious in actual studies, and it is now even widely recognized mostly bullshit. It is just a kind of tech influencer if you want. Computer science and/or software engineering has more serious branches, where cyclomatic complexity can have its use.
The thing is, AI has no idea when an abstraction is good or not.
The reductio ad absurdum here is that, if abstraction can just be assumed to be bad for quality and maintainability, then perhaps we should go back to hand writing machine code for non-microcoded sequential execution CPU architectures. Conversely, if that idea sounds as preposterous to you as it does to me, then you’re stuck conceding that at least some abstractions are mostly good. So then, before you can automate deciding which ones should and should not count against a code quality metric that’s computed automatically, you need to find an operational definition that can be applied deterministically.
Wow! Did not expect this blast from the past this morning. I worked with Alberto and Bob at the same startup long ago. Hello to any other Agitators who found this today.
One of the many reasons leadership needs to change, they are now more aggressive towards exploiting their customers, especially in cloud, Kurain is ruining that platform, but the investors like it
> Here’s why we think that CRAP1 is a good anti-pattern to detect. Writing automated tests (e.g., using JUnit) for complex and convoluted code is particularly challenging, so crappy code usually comes with few, if any, automated tests.
This is so wrong.
The formula uses code coverage as a fundamental metric, when in reality, a lot of people write code "correct from construction", so coverage is not even applicable. Many times too, people only care the use cases they care about work perfectly.
There are also many other reasons code is not tested, not because it's complex, but because it's simple.
> If the code is simple, the tests aren't much extra work.
Disagree. Even if it isn't much work to write (which is debatable), it still carries costs: CI takes longer to run, and the tests have to be carried and maintained (and possibly updated if the code itself changes).
> High test coverage doesn't mean your code is good, but it at least reduces the rate at which you accidentally break stuff.
Sure, for the most part. But that doesn't mean "more test code" is always a good thing. Tests aren't free, even for simple code.
In its (admittedly weak) defense, if the submitter edits the title back during the few-minute window after submitting where this is possible, the normalizer does not kick back in. But it is a bit opaque, in the sense that the poster needs to: realize that the title has been changed; know that they can edit the title (but not too slow!); and know that it won't be filtered through again.
I have no idea what the website would look like without it, but I have a feeling it does more good than harm.
Maybe it's invisible when it works as intended, but the only times I notice a title has been changed, is when it's changed back to the original title, stripping most of the valuable context in the process. De-editorialization, I suppose.
You get an hour or so to edit the title. There is ample time to fix it if the auto-edits mangle it. The submitter just has to look at the submission after hitting submit one time to see if there was a problem.
It's more of a gut feeling than anything else. I don't know what the normalization rules are, but I think we mostly notice the normalizer when it mangles something [1], and not when it's just quietly humming along. Someone with more spare time could probably figure out a dataset of original to normalized titles and measure that.
[1]: Example from the frontpage right now: "How Big Are Factorials?" probably got normalized to "Big Are Factorials?" originally, based off other previous manglings I have seen previously.
I'm sure this method has evolved and/or been supplanted over the last 15 years, but one thing that struck me reading this is how much the dynamics of unit test coverage have changed in recent history, with AI-generated commits containing 10x as many unit tests (many of them kind of silly and tautological) as in the olden days. Gonna need to update some of those coefficients in their CRAP1 formula... Or maybe test coverage has/will become too noisy a parameter to use at all.
Humans have been hand-crafting artisanal test cases of equal banality for years, often to meet some metric no one is verifying deeper than an automated report etc. I'm more inclined to believe the null hypothesis here, that the formula doesn't need any change and the signal is still equally valid. If anything changed, the signal is probably more useful now, since you'll never catch a good AI writing test cases that do literally nothing except call a function in a try catch with an empty catch block. (Models may have done that a few years ago but I haven't witnessed that behavior in a long time.) I've witnessed myriad examples of this in pre-AI production code.
Anecdotally, I’ve found that codebases that enforce code coverage metrics often have worse behavior coverage than ones that don’t.
It’s a classic example of Goodhart’s Law in action. Code coverage metrics only measure what percentage of code the test suite causes to run. But it’s very, very easy to write tests that run code without actually confirming that it produces correct output for all possible inputs. And it’s very, very easy to assume that a module with 90+% code coverage also has 90+% behavior coverage, and then become complacent about reviewing the suite for proper behavior coverage.
Djikstra seemed like he was mostly against testing. But only because he was for proofs. A unit test is a single example. The real way to demonstrate the absence of errors is to prove they aren’t there (vis a vis axioms and assumptions).
But most developers don’t have the mathematical sophistication nor the time.
It’s not that unit testing is useless. Just good to know what their limitations are and to plan your testing strategy accordingly.
You also have the problem of potentially having to re-verify everything by hand for every little change. Maybe fine for the kinds of projects Dijkstra was working on, but less practical in a business setting.
Tools like QuickCheck and Hypothesis are an interesting middle ground, though. I strongly prefer them over standard-issue unit testing for verifying algorithm implementations.
Hundred percent. All about trade-offs.
Although proof techniques such as proof repair have come a long way, it’s still impractical for a lot of scenarios.
TLA+ is great for systems design and such. Quick check style tests are awesome and a very low bar to clear from unit tests.
Yeah. And TLA+ can confirm that the design is sound, but it can’t confirm that the implementation conforms to the design. QuickCheck style tests can’t solve that problem, but perhaps they can mitigate it.
> Or maybe test coverage has/will become too noisy a parameter to use at all.
It already is, ive banned unit tests via ci checks from our codebases, they were not particularly useful before LLMs and now they are a net negative.
We require int and some e2es and that does all that units do and more.
I sometimes use unit tests to fill in gaps. I just caught a function with almost full coverage except for an if statement.
Playwright e2e tests are harder to maintain than pytest code, so I wrote a test just for that one case, and while I was at it, a few other input strings that seemed like they could be problems.
If the function had no test coverage at all, I probably would have tried to get it with an integration test, because that would mean the callers also weren't getting tested.
The occasional unit test is fine. But you do not need for example `test_helper_test.go` that is just ridiculous.
Its gotten so bad I decided to throw the baby out with the bathwater so to speak and focus on the high value places instead of trying adjudicate every individual usage
Are they slow and brittle?
I like having both. Unit tests are a decent indicator for local development. Fast, quick to iterate on. Integration tests are slow as molasses and I can’t iterate with them when it takes 20 minutes to setup the suite and run everything. Too slow. But still useful as you say.
They don't flake, generally can run a few hundred in 3-4 mins. Iteration times are definitely key.
[dead]
A measure is only good if I take action on it and in turn make things better. There are a lot of things that are easy to measure, but there is no useful action I should take on the measure.
Yes, but also all too often “useful” is interpreted to mean “moves the metric”. If that metric is merely a proxy for some more tangible outcome then that may not be good enough.
The one that tech tends to stumble on most often is velocity-type metrics. The problem there is that you can’t pay the bills with velocity. And velocity metrics tend to favor cheap shovelware features that cohere poorly over anything that involves having the team slow down on churning out code long enough to work out elegant solutions to subtle problems.
I've seen a couple of tools to calculate the CRAP score. I haven't used them in anger though.
For Rust there's https://crates.io/crates/cargo-crap, and for Go there's https://padiazg.github.io/go-crap/
>Note: This post is rated PG-13 for use of a mild expletive. If you are likely to be offended by the repeated use a word commonly heard in elementary school playgrounds, please don’t read any further.
Mild as this ironic passive aggressiveness is, can't imagine something like this in modern sterile corporate messaging.
Man, when I was a kid in the 80s, my parents considered "crap" to be a bad word, and my sister and I would get in trouble for using it.
I'm sure that's still the case for some kids/parents today, but I can't imagine that being remotely common.
There's a good chance it'll be scrubbed now that it's frontpaged here
don't be evil!
every bit of humanity went with the motto
funny enough, the disclaimer comes after the term is used in the title and url.
"repeated use" seems to be doing the mitigation work here, though it does seem unusual that someone offended by the repetition would be unoffended by a one-off.
I have a goal to make the codebase at work cargo-crap compliant and enforce it with CI. I let an agent run overnight with it once and the diff touched like 40% of our codebase which is untenable for a single merge. So for now I’m doing it piecemeal as the opportunity presents itself.
The pendulum has swung too far in the direction of class, function, cyclomatic complexity (and here, CRAP) and similar idiotic metrics.
This reminds me of a talk Sandi Metz did called "All the Little Things" where she covers the Gilded Rose kata. In the talk, she reworks her solution until there's almost nothing left showing the essence of the problem being solved.
The cyclomatic complexity metric is touted at each step as a proxy for goodness of design and removal of complexity. However, a weakness of the measure itself is that it doesn't account for the control flow indirection that happens through OO method dispatch itself.
At the same time, Kevlin Henney's talk called "Gilding the Rose" takes the same kata and arrives at a far more sane solution he works up to and reveals at the end.
Short functions used to be hot. Uncle Bob used to proselytize "The first rule of functions is that they should be short. The second rule of functions is that they should be shorter than that." Now emphasizing the benefits of longer functions is pretty trendy. https://github.com/johnousterhout/aposd-vs-clean-code
This industry is pretty idiotic sometimes ¯\_(ツ)_/¯
A while back Hillel Wayne did a talk (whose name I forget) on what empirical evidence on software quality actually says.
As I recall, he concluded that there’s really no support for then-popular ideas like short functions, reducing cyclomatic complexity, avoiding explicit branch statements and loops, or TDD. (Tests yes, just not TDD.)
He made a pretty strong case that only two principles are particularly robust. One was that limiting code volume is good. The other is that working people too hard is bad.
This talk.
What We Know We Don’t Know • Hillel Wayne. (2019, April 28). Hillel Wayne. https://www.hillelwayne.com/talks/what-we-know-we-dont-know/
>cyclomatic complexity metric is touted at each step as a proxy for goodness of design and removal of complexity. However, a weakness of the measure itself
Amen, it’s hard to push back against an opaque term (cyclomatic!) when it isn’t really a measure of goodness, it’s a measure of branching, kind of a normal thing in code.
Early on I found that code with low cyclomatic complexity was just usually extremely verbose, lots of passing this to that while avoiding the branching necessary to get something done.
And yes, you can game the metric by hiding the complexity among the confusion of objects and components.
> it doesn't account for the control flow indirection that happens through OO method dispatch itself
Every indirect call is a conditional branch, where the condition can be arbitrarily far away in time and space.
Oh god, yes. The powers that be in my workplace are obsessed with cyclomatic complexity, so now we’re reviewing a flood of LLM diffs that take perfectly good code and extract each loop into its own function with a dozen arguments. The code is now more maintainable on paper and far, far less maintainable (by a human) in practice.
Reasonable cyclomatic complexity is useful (at least) for testability. No metric is perfect and no metric should be a primary goal, but it has its use, and if you read e.g. a bit of leaked Windows you certainly will understand why it matters (and not because it has a particularly low cyclomatic complexity...)
Of course if you introduce new methods of dispatch and do not take them into account into a metric, you end up with something less... precise? useful? But given the primary intent and why and how the metric was created this seems a pretty trivial observation.
Now I agree it is also retarded to attempt to get only very short functions or extremely low cyclomatic complexity everywhere (even if you try to adapt it to count new kind of dispatch), because the only effect that produce is that it moves the complexity in another more abstract place we are less well equipped to manage.
"Short functions used to be hot. Uncle Bob [...]": well yes, Internet and sometimes group of people inspired ultimately by Internet and group effects can be pretty idiotic, but honestly Uncle Bob ideology was never considered serious in actual studies, and it is now even widely recognized mostly bullshit. It is just a kind of tech influencer if you want. Computer science and/or software engineering has more serious branches, where cyclomatic complexity can have its use.
I suspect that we could bring this measure into the modern world with a little help from either DFS or ai.
Something like abstractions traversed during interpretation, lines of abstraction v.s. functional implementation, or logic statement dispersion.
It was hard to pin down what was abstraction vs. implementation, but it's much easier now.
The thing is, AI has no idea when an abstraction is good or not.
The reductio ad absurdum here is that, if abstraction can just be assumed to be bad for quality and maintainability, then perhaps we should go back to hand writing machine code for non-microcoded sequential execution CPU architectures. Conversely, if that idea sounds as preposterous to you as it does to me, then you’re stuck conceding that at least some abstractions are mostly good. So then, before you can automate deciding which ones should and should not count against a code quality metric that’s computed automatically, you need to find an operational definition that can be applied deterministically.
(2011)
Wow! Did not expect this blast from the past this morning. I worked with Alberto and Bob at the same startup long ago. Hello to any other Agitators who found this today.
2026 Google would never have some "fun" like this
One of the many reasons leadership needs to change, they are now more aggressive towards exploiting their customers, especially in cloud, Kurain is ruining that platform, but the investors like it
When a measurement becomes a target, it ceases to be a good measure.
Every time I see a software update I cringe inside.
> CRAP1(m) = comp(m)^2 * (1 – cov(m)/100)^3 + comp(m)
and
> Here’s why we think that CRAP1 is a good anti-pattern to detect. Writing automated tests (e.g., using JUnit) for complex and convoluted code is particularly challenging, so crappy code usually comes with few, if any, automated tests.
This is so wrong.
The formula uses code coverage as a fundamental metric, when in reality, a lot of people write code "correct from construction", so coverage is not even applicable. Many times too, people only care the use cases they care about work perfectly.
There are also many other reasons code is not tested, not because it's complex, but because it's simple.
If the code is simple, the tests aren't much extra work.
High test coverage doesn't mean your code is good, but it at least reduces the rate at which you accidentally break stuff.
> If the code is simple, the tests aren't much extra work.
Disagree. Even if it isn't much work to write (which is debatable), it still carries costs: CI takes longer to run, and the tests have to be carried and maintained (and possibly updated if the code itself changes).
> High test coverage doesn't mean your code is good, but it at least reduces the rate at which you accidentally break stuff.
Sure, for the most part. But that doesn't mean "more test code" is always a good thing. Tests aren't free, even for simple code.
Wait, I thought all hand-curated enterprise code was godly pre-LLMs?
Google is crap
Are we just writing tautologies now?
No, we were writing them in 2011.
Title is editorialized. Original: "This code is CRAP" referring to code in review as Change Risk Anti Pattern.
Also, (2011)
I believe the "This" might have gotten swallowed by HN's title normalizer.
This title normalizer is crap.
In its (admittedly weak) defense, if the submitter edits the title back during the few-minute window after submitting where this is possible, the normalizer does not kick back in. But it is a bit opaque, in the sense that the poster needs to: realize that the title has been changed; know that they can edit the title (but not too slow!); and know that it won't be filtered through again.
I have no idea what the website would look like without it, but I have a feeling it does more good than harm.
Maybe it's invisible when it works as intended, but the only times I notice a title has been changed, is when it's changed back to the original title, stripping most of the valuable context in the process. De-editorialization, I suppose.
(Or when the auto-renamer does a funny!)
You get an hour or so to edit the title. There is ample time to fix it if the auto-edits mangle it. The submitter just has to look at the submission after hitting submit one time to see if there was a problem.
What is the "good" when considering the lack of aggressive truncation/overflow concerns here?
It's more of a gut feeling than anything else. I don't know what the normalization rules are, but I think we mostly notice the normalizer when it mangles something [1], and not when it's just quietly humming along. Someone with more spare time could probably figure out a dataset of original to normalized titles and measure that.
[1]: Example from the frontpage right now: "How Big Are Factorials?" probably got normalized to "Big Are Factorials?" originally, based off other previous manglings I have seen previously.
Neutering Popular Science clickbait titles like “This Koala has a Secret Trick!”
While that might be the case, best to normalize that: Title Normalizers are Crap.
Submitters can rectify the title after submitting.
Isn't that what this title is? Which part is editorialized?
(2011)