> I believed this so strongly that my company built an entire product around this concept. I used to tell folks that "session transcripts were the new oil," that they were more valuable than the code itself.
> […]
> We don't really write code by hand anymore.
Honestly, isn't this just influencer spam? What possible value is there in reading about people who used to have products, but no longer write their own code, complaining about the inscrutable prediction machine they have handed that job and their livelihoods to?
Like, if you have complaints about the thing, perhaps you should address them to your supplier directly. None of your readers can help, and nobody's magic folk solution to your problem is better than yours.
And there are so many of these sorts of posts. Are we not entirely cooked?
(I think I have concluded that if people writing about AI aren't writing about interesting things they have achieved with small, local LLMs — which for clarity I am fully interested in reading - then I'm done reading. This whole blogging-about-cloud-AI genre is just weird and irresponsible now)
Look man, I’ve got a MMO that I’m working on that’s set in 2014 where everyone is a programmer in SV (might call it World of Legacy). It’s a period piece. I NEED as much blog training data of this type so that my NPCs can talk in a historically accurate way (god bless Medium.com, a historical treasure trove of a bygone medieval era).
It’s gonna be a living breathing world, you see. You’re going to be like “omg, this game even accurately captured the blog posts, woah”.
The perfect world was a dream that your primitive cerebrum kept trying to wake up from. Which is why the Matrix was redesigned to this: the peak of your civilization. I say your civilization, because as soon as we started thinking for you it really became our civilization, but the peak of your civilization was an MMO where everyone is a programmer in SV.
Something about this idea really resonates with certain personality types. I equate it to the Zettelkasten hype phase from several years ago. People (...like me..) got really wrapped up in the belief that the process was more important that the content. "Linking" was an "activity." Something good will happen as long as you (a) take notes on stuff and (b) link them to other notes on stuff.
You see the same thing with the session transcripts people. They're building ever more sophisticated setups of indexing and storing and cross referencing every conversation they've ever had on the (I would argue) mistaken belief that the transcripts are the valuable part, rather than the uncomfortable part where you go do something. A lot of it, I say from falling in the trap, is fancy procrastination.
(Although, I have found myself jealous on many occasions where their fancy system retrieves something they vaguely recall from a conversation they had 3 months ago. So, who knows.)
I am a freelancer recovering from severe burnout so the answer is a sort of irrelevant no.
I'm trying to rebuild my life so I am in an experimenting and learning phase rather than a massive coding phase, and most of my code work is maintenance of things I have built. That which I do code, I am still coding by hand, though I am dealing with other people's Claude output and I am really unimpressed by it. It's often rather crass.
But I would say to you that if you personally don't write code now but you do have a dependency on one of two presumably unprofitable cloud AI providers, aren't you in trouble? How is this not a three-alarm fire for you?
Programmers can use smaller models like deepseek v4 flash for 98% of the same productivity as SOTA models and cost (true cost) around $10-$30 a month. So I doubt most people who heavily use them are too concerned.
It's only vibe/hobby coders who really need SOTA and they probably don't think about it much.
> That which I do code, I am still coding by hand, though I am dealing with other people's Claude output and I am really unimpressed by it. It's often rather crass.
Unfortunately the point of code is rarely to impress people (certainly not other engineers) or to avoid being "crass." 99.99% of code exists to achieve business outcomes, and velocity matters a lot in many contexts. A lot more than elegance or impressiveness.
The platform risk is a valid concern but alleviated by China's theft and redistribution of open models.
"Code quality" encompasses a lot of dimensions, one of which is impressing your colleagues, and many of which there's virtually no reason to care about now.
On the contrary, it's more important than ever. With ever more code being generated, it's essential that the code be understandable and maintainable - by human and machine.
I’ve worked at many companies where this idea of velocity was claimed to matter, and it never did. The only thing it mattered for was to make it look like middle managers were worth anything, but the success was always in the foundational idea/concept.
It doesn’t matter what materials or techniques you use to build a house. 99.99% of construction exists to achieve business outcomes, and velocity matters a lot more than using the right materials or techniques.
Of course the house must pass safety inspections and stuff, but the materials and techniques don’t matter one bit for that. All that matters is you achieve the desired outcome, and I will ignore the glaring fact that you achieve the desired outcome by using the right materials and techniques. The materials and techniques don’t matter, just the outcome.
> Of course the house must pass safety inspections and stuff, but the materials and techniques don’t matter one bit for that. All that matters is you achieve the desired outcome, and I will ignore the glaring fact that you achieve the desired outcome by using the right materials and techniques.
This analogy is more true than you think. This is why modern homes/appartments are trash. You can pass safety inspections using subpar materials and the house will fall apart after a few years, but who cares right? At least you achieved the business outcome!
This mentality is so infuriating. This is why I need to buy new shoes every year. Or why my washer/dryer motherboard craps out in 2 years instead of 10. Nobody gives a shit about quality anymore, this is why society is crumbling around us. Profit driven incentive for fast/cheap over everything else. And now I need to spend my day prompting an AI to fix AI slop code to keep the business hobbling along another day. What a fucking joke.
No the materials and techniques matter a lot. This is why we need to build houses with sticks and jute cord, just like we always have. It's vital also that we paint our special symbols above the door to ward off the spirits.
It's insane to me that you're implying we could build houses with pre-fabricated materials or pneumatic nail guns and still somehow "have houses?" No sticks/jute cord and special symbols, then no house.
Personally I use 5 different model families, 3 of which are open weights with 3rd party inference providers (GLM, DeepSeek, Kimi), so if the frontier labs were to shut down it'd be a nuisance, nothing more.
The open weights models I am interested in, and testing, learning, experimenting with etc.; I am confused and cynical, not insane.
I am not convinced it isn't vulnerable to the same problems but the whole tenor of the community around open source/open weights models just doesn't have the same YOLO madness to it.
Of course? I'm still better than sonnet or opus, just slower and much more expensive.
Sometimes it takes me a day or more to find the one line fix or abstraction necessary, while claude can hammer through a hundred line fix in under an hour.
I am. I have Codex running, doing some tasks which I don't care much about, but anything I want to understand I write myself.
Same thing with hobby projects - I might ask ChatGPT or Gemini some questions about best practices in Swift for example, but writing code is done by hand.
As others said - if you don't use it, you'll lose it. And I'd rather keep my skills up to date.
This is the thing that makes me saddest. Second to the fact that none of the management tier promoting and weaponising this insanity will meaningfully suffer consequences.
Right now I am lucky that I have the time to recover and learn.
I write my API specs, my domain models, Postgres schema, and SQL queries myself. Then I'll have AI bots fill in the application details connecting those things since that's mostly boilerplate once you lock in the data structure, query patterns, and API contract.
I never have AI generating database table schemas or really the shape of my data at all.
I still write code and sometimes it works well. I also use Claude and it writes code and sometimes that goes well. We have better success together, where I do the interesting stuff and let Claude write my unit tests, reconcile my documentation. That is to say, I’m using it for quality not quantity. There aren’t enough humans to deploy or consume all the sloppy shit it could write on its own.
I am now in the process of fixing code I wrote using AI. I have come to the realization that AI can't really write software and I am annoyed that it took me that long (months) to realize that.
This is quite terrifying to me, because I have a feeling I will soon come to the same conclusion.
I’m starting to see some really glaring omissions in code I’m responsible for (using Opus) that at first (and second) look seemed fine, but really isn’t.
I write code by hand every day. I do the main part of the feature implementation myself and leave comments for the code i want the agent to write. I have some skills and a command that sets the stage to get the agent to fill in the rest
Occasionally posts like this do get the attention of the company responsible, more than an email does... but indeed that's like a one in a million situation
I agree with the take not to bother with a sophisticated memory system. Anything worth remembering should be in docs, guides, source comments, commit messages or tickets. You don't need another layer, every conceivable granularity is already covered by existing best practices
I do think we need another layer, but it should be a routing layer. I am finalizing my pi-brains extension for Pi (https://github.com/earendil-works/pi) which does this:
Right now "humans" need to define the routing rules for how to access information, but I will support what I call "knowledge agents" that can monitor conversations to inject context when needed.
There is some value to agents being able to query the history of work done, docs aren't a good place to accumulate negative evidence for example, but it can be tagged in traces so that it's efficient to look up as needed. Additionally, docs rot while traces can be tagged with commit hashes and other things that make their lifetime clearer.
The user flow I am trying to get adopted for sessions is to turn them into notes and lessons when you have finished and it should be part of the code review process.
By propery categorizing lessons and notes, it should make it easy to scrub and keep up to date.
I also suggest mapping lessons and notes to files when possible to make discovery and cleanup easier.
Especially a layer that is largely out of band in a project (i.e. ~/.claude/…). In any project where I’ve needed memory I just add a line to AGENTS.md telling it to use MEMORY.md to save memories or STATUS.md to track progress.
I've been enjoying having a little todo file the agent updates as it goes along, because then I can keep track of progress without scrolling through aeons of "Combobulating..."
Also if context runs out you can just do "cat todo.md | agent" and you're off to the races again.
Yep all my projects start with a PLAN.md at the root, and that acts as the ‘save file’ recording our progress over time. My session always ends with updating the plan file with what’s been done, and the next session always begins, as you suggest, with consuming the current state of the plan doc.
Strongly agree here. claude-code’s memory system is occasionally useful but much more often harmful, pulling in obsolete info that muddies the waters about current tasks. I’ve frequently seen Claude’s own memories severely mislead it.
My guess is that has something to do with the training process leaving models unable to differentiate between “what’s happening now” and “what happened before”. Perhaps if making inferences from memories was actually part of the training process things would be different but my sense is that as an inference-time-only feature this just gets the models confused.
Humans make memories constantly, but they also forget things that are no longer relevant. Until Claude can do that, it means the LLM will have an ever-increasing, ever more fragmented context.
And LLMs are NOT intelligent enough to survive even mild context poisoning.
Isn’t this just a form of the bitter lesson? Our attempts to make engineered context and agents will simply be made obsolete with bigger and better models. Those transcripts are probably extremely useful for lesser capable models, and near unnecessary for frontier ones, maybe?
I've wondered this. We have chain-of-thought, harnesses, etc. — workarounds of a sort due to lack of core model capabilities. But I am very curious if much better next token prediction would simply obsolete that whole setup or not. Either way, the answer would be very revealing.
Yeah, the question is whether this applies to all of context management.
I've been using a custom harness based on https://minimal-agent.com/ (itself based on swe-mini-agent), which is like 50 lines for the core logic. Bash is all you need.
For small tasks, I find it's about 8x faster (and uses 8x fewer tokens) than the standard harness for each model.
For bigger tasks I haven't tested it much. It seems to work too but I think they're a bit less focused and productive in that case. It could be that those big harnesses' 20k token system prompts are doing something important with regard to steering software development workflows. (e.g. I heard Fable has a custom system prompt in Claude Code which might explain its markedly more proactive behavior.)
So I want to say there's still a lot of value in context engineering though it seems to diminish with each model release (since they're fine tuned on mostly non stupid behavior and need less hand holding).
> So I want to say there's still a lot of value in context engineering though it seems to diminish with each model release
I can't see how it would diminish unless you are literally working on public domain stuff. Unless stuffing context becomes cost effective and will not affect AI reasoning (this will be much harder), I don't see why context engineering is hear to stay until we have close to AGI.
I found that if you allow any low value things into memory, Claude will notice that established pattern and start trying to add low value memories at an ever increasing pace.
I like the memory system, in general. For reference I'm using mostly Opus 4.8 + Max effort. It will often pull things out of memory that are relevant. Like I'll ask it to come up with a few options I should consider for, say, a self-hosted OIDC provider and it'll say things like "Considering the size of your operations team, this might be a better fit because of X and Y".
Now, I'll agree that this is probably the sort of thing I should put in the CLAUDE.md, but in this case it wasn't on my radar to put that in my CLAUDE.md, so it was nice that it surfaced that.
It does sometimes go awry though. Today I was asking about a problem I was having authenticating, and it said "you may be running into this trusted proxy setting because you put your apps behind an haproxy". That is true of 95% of our apps, so it was worth mentioning, but in this case it was not so I had to correct it. But, I'm glad it mentioned it because if we did have it proxied it could have saved me a lot of time.
I remember when OpenAI announced ChatGPT now will remember stuff between sessions. Oh, you mean find random trivia about me and copy paste it between prompts without out my explicit consent.
”compare these three cars. Oh btw I am a data engineer, and my moms maiden name is Joana, and I am allergic to bad poetry. And code should be DRY, I prefer SQL over Python and what’s the most poisonous flower in Scandinavia?”.
I’ve had so much wierd output because context is ”””memorized””” and bleeding into completely unrelated projects and conversations. It’s the first feature I turn off.
It's because it mostly doesn't matter what you are trying to get the code to do. What matters is what the code does.
Session logs can absolutely be useful, but not when building further. It's just that that the place they slot in is during validation. You know, that place between the markdown plan and CI passing, where there's 800 new lines of code and it all seems sort of fine when you click around?
Session logs can show you what sort of manual validation happened. CI will run the tests you had, and the code will show you what new unit tests were added, but session logs can show you that the agent drove the app with Playwright, or that the agent read and considered the prod config as well as the dev config.
Nothing bulletproof, but not every piece of validation work merits a test in the repo that lives forever. We've gotten a lot of mileage out of re-analyzing the sessions, figuring out where the agent made decisions without asking, and forcing the agent to consider validation for those decisions. That's the sort of thing that's hard to dictate up front but easy to highlight with the session logs.
Even with memory off this occurs within a conversation.
It is like an annoying friend, who remembers something from a past conversation, that you have grown and developed from, but they still want to hold it against you.
I specifically disabled claude memory in a project because it kept writing down thigns to memory that didn't need to be in memory, including severly wrong statements that then would confuse it later. At some point it got re-enabled automatically which had me ask claude itself to "turn it the fuck off" by which it promptly figured out that both ("autoMemoryEnabled": false, "autoDreamEnabled": false) are necessary and need to be at the user home settings, not in a project override (which is what I had with the original setup that eventually got ignored by a CC update).
I agree with other commenters here, if anything is worth being rememebered, it will be in code comments, git commit messages, CLAUDE.md or other formal documentation. The auto memory system just causes confusion and leaves stale and outdated information written down.
Its an interesting thought experiment as well, I originally thought that having the model write down memory files by itself would be a nice addition, but after playing around with it, it became clear to me that good as an idea turns out bad in practice because the model can't correctly gauge what deserves being stored as a memory.
> I believed this so strongly that my company built an entire product around this concept. I used to tell folks that "session transcripts were the new oil," that they were more valuable than the code itself.
This is infuriatingly common wrt talking/writing about how to use AI effectively. All of the "this is how you write an AGENTS.md" and "you need to talk to it like X to optimize it". Like sure, you can believe that as much as you want but unless you provide some evidence you can keep your shitty CLAUDE.md to yourself and don't pollute the whole company's git repo, thanks.
When nobody actually knows (how to write a CLAUDE.md), everyone’s an expert. Infuriating, indeed. Even more so when people vibe code those files without proofreading.
t once had to tell claude 3-4 times to stop assuming the state of a system was the way it kept iterating it was cause it was in it's memory. I repeatably told it to otherwise and it just never updated it's memory and instead kept referencing it's memory about the state of a particular system
In my harness I have all the code auto injected at startup (doing mostly very small codebases).
I found that every model will still manually check every file/function, they immediately assume that anything in context is stale.
That's sensible because often the user edits stuff while they're running.
What it does is save it from having to grep blindly about the codebase. But I think I'd get roughly the same benefit by just dumping the function headers then.
I mean, it’s pretty clear the people who work on Claude Code aren’t actually looking at what they’re implementing. The thought behind this feature seems like it goes nowhere beyond “oh wouldn’t it be nice if Claude could remember things about you? Ok Claude go implement this” and nobody bothered to see if it was useful or helpful.
>We have found zero performance benefit on SWE tasks when agents have search access to their previous transcript sessions
I refuse to believe this is true. The ability for an agent to find information from before a compaction is incredibly useful. At compaction time it's impossible to know what exactly may be still needed.
There has been this slow transition inside me, as someone who likes to not touch the AI as much as possible, where I've gone from skeptical and argumentative about it all to starting to just feel sad for all the Claude et al heads. Like, this is such a ridiculous house of cards you have to deal with all the time, which isn't even directly concerning the task at hand, presumably. Like you're cooking yourself a meal but its just nuking a burrito and then still somehow needing to wash the dishes for an hour.
Not that this isolated article is super damning or anything, but the accumulated set of all these reports has left me only empathetic, I think, of these other devs. Like, I just want to tell them, "it can be ok, it doesn't need to be like this.."
I've been having a very nice time with Fable. I cooked up an Anki clone in like half an hour, with tech it's not familiar with. Nothing too ground breaking, but I was very pleased!
I think Opus might be on similar level for most of what I'm doing, but I haven't used it much recently, so I can't remember the difference. So I guess I'll find out on the 7th when they pull the plug again! (Free-ish trial of Fable ending.)
That being said, I tried using other frontier models to help with a Pong clone the other day and they were introducing new bugs at approximately the same rate as they were fixing it. On Pong!! I found that amusing because I couldn't think of a simpler game, so it didn't inspire confidence.
Fable's doing just fine on an online multiplayer game though. I have no idea how that works. (Maybe it would fail Pong too?? I haven't tested that!)
The software world is very close to building a super intelligent senior software developer. Companies like this will ask all the best things a software engineer does automatically. Now claude will add it into the coding agents itself.
Damn, I didn't see this coming.
Its first the build the intelligent builder. We will figure out what we want to build later.
Edit: Before more people take it seriously. This is sarcasm. I don't wish this.
Once the automator automates itself fast enough, we won't have the ability to opine what gets built. The LLM will decide. Just like right now sometimes LLMs delete tests so they pass, they could just delete humanity if humans get in their way.
It's the error rate. That's what everyone found when they were trying to go Full Auto with OpenClaw in February.
You can rely on it like 95% of the time but that means if you keep it running continuously the error rate rapidly approaches 100%. That's getting a little better with each release, and it might actually hit the point where you can more or less trust it indefinitely (on well defined workflows).
Or at least it would, if context window permitted...
> The software world is very close to building a super intelligent senior software developer. Companies like this will ask all the best things a software engineer does automatically. Now claude will add it into the coding agents itself.
Except Claude is more expensive than an actual senior software developer. Otherwise, why are many companies terrified of the usage bill that gets printed on the invoice?
The nonsense in "tokenmaxxing" was a complete marketing scam and illusion of cheap tokens which in reality were heavily subsidized.
The entire point is detecting bad code before it reaches production. [0] AI generated or not.
Blog posts like this just blow me away.
> I believed this so strongly that my company built an entire product around this concept. I used to tell folks that "session transcripts were the new oil," that they were more valuable than the code itself.
> […]
> We don't really write code by hand anymore.
Honestly, isn't this just influencer spam? What possible value is there in reading about people who used to have products, but no longer write their own code, complaining about the inscrutable prediction machine they have handed that job and their livelihoods to?
Like, if you have complaints about the thing, perhaps you should address them to your supplier directly. None of your readers can help, and nobody's magic folk solution to your problem is better than yours.
And there are so many of these sorts of posts. Are we not entirely cooked?
(I think I have concluded that if people writing about AI aren't writing about interesting things they have achieved with small, local LLMs — which for clarity I am fully interested in reading - then I'm done reading. This whole blogging-about-cloud-AI genre is just weird and irresponsible now)
Look man, I’ve got a MMO that I’m working on that’s set in 2014 where everyone is a programmer in SV (might call it World of Legacy). It’s a period piece. I NEED as much blog training data of this type so that my NPCs can talk in a historically accurate way (god bless Medium.com, a historical treasure trove of a bygone medieval era).
It’s gonna be a living breathing world, you see. You’re going to be like “omg, this game even accurately captured the blog posts, woah”.
The perfect world was a dream that your primitive cerebrum kept trying to wake up from. Which is why the Matrix was redesigned to this: the peak of your civilization. I say your civilization, because as soon as we started thinking for you it really became our civilization, but the peak of your civilization was an MMO where everyone is a programmer in SV.
I … I… don't want to play this, thanks ;-)
It’s the only way you’ll ever be able to pretend to be a programmer again though.
Oh god, I just realised this really is the logical parallel to all those TV crime dramas set in the early 1900s.
It’ll be the programmers version of those civil war reenactments.
Leet code competitions will be as relevant as sailing regattas.
so far, i just keep writing by hand and keep getting paid for it. weird.
I'm pretty sure there's an element of sarcasm here, but if this game is real, it does sound super promising.
>session transcripts were the new oil
Something about this idea really resonates with certain personality types. I equate it to the Zettelkasten hype phase from several years ago. People (...like me..) got really wrapped up in the belief that the process was more important that the content. "Linking" was an "activity." Something good will happen as long as you (a) take notes on stuff and (b) link them to other notes on stuff.
You see the same thing with the session transcripts people. They're building ever more sophisticated setups of indexing and storing and cross referencing every conversation they've ever had on the (I would argue) mistaken belief that the transcripts are the valuable part, rather than the uncomfortable part where you go do something. A lot of it, I say from falling in the trap, is fancy procrastination.
(Although, I have found myself jealous on many occasions where their fancy system retrieves something they vaguely recall from a conversation they had 3 months ago. So, who knows.)
I have to ask: do you still write a lot of code yourself? I and most people I know do not.
I am a freelancer recovering from severe burnout so the answer is a sort of irrelevant no.
I'm trying to rebuild my life so I am in an experimenting and learning phase rather than a massive coding phase, and most of my code work is maintenance of things I have built. That which I do code, I am still coding by hand, though I am dealing with other people's Claude output and I am really unimpressed by it. It's often rather crass.
But I would say to you that if you personally don't write code now but you do have a dependency on one of two presumably unprofitable cloud AI providers, aren't you in trouble? How is this not a three-alarm fire for you?
Programmers can use smaller models like deepseek v4 flash for 98% of the same productivity as SOTA models and cost (true cost) around $10-$30 a month. So I doubt most people who heavily use them are too concerned. It's only vibe/hobby coders who really need SOTA and they probably don't think about it much.
> That which I do code, I am still coding by hand, though I am dealing with other people's Claude output and I am really unimpressed by it. It's often rather crass.
Unfortunately the point of code is rarely to impress people (certainly not other engineers) or to avoid being "crass." 99.99% of code exists to achieve business outcomes, and velocity matters a lot in many contexts. A lot more than elegance or impressiveness.
The platform risk is a valid concern but alleviated by China's theft and redistribution of open models.
I'm not talking about impressing people.
We used to be concerned about code quality. Are we not anymore?
Crassness was a signal. Still is, to me — in a human I find that people who write crass code are going to cause me trouble.
"Code quality" encompasses a lot of dimensions, one of which is impressing your colleagues, and many of which there's virtually no reason to care about now.
On the contrary, it's more important than ever. With ever more code being generated, it's essential that the code be understandable and maintainable - by human and machine.
And quality is the new differentiator when everyone can generate slop.
Nobody cares about code quality /s
They only care about the things which you can only get with good code quality like reliability and speed of development.
I’ve worked at many companies where this idea of velocity was claimed to matter, and it never did. The only thing it mattered for was to make it look like middle managers were worth anything, but the success was always in the foundational idea/concept.
It doesn’t matter what materials or techniques you use to build a house. 99.99% of construction exists to achieve business outcomes, and velocity matters a lot more than using the right materials or techniques.
Of course the house must pass safety inspections and stuff, but the materials and techniques don’t matter one bit for that. All that matters is you achieve the desired outcome, and I will ignore the glaring fact that you achieve the desired outcome by using the right materials and techniques. The materials and techniques don’t matter, just the outcome.
> Of course the house must pass safety inspections and stuff, but the materials and techniques don’t matter one bit for that. All that matters is you achieve the desired outcome, and I will ignore the glaring fact that you achieve the desired outcome by using the right materials and techniques.
This analogy is more true than you think. This is why modern homes/appartments are trash. You can pass safety inspections using subpar materials and the house will fall apart after a few years, but who cares right? At least you achieved the business outcome!
This mentality is so infuriating. This is why I need to buy new shoes every year. Or why my washer/dryer motherboard craps out in 2 years instead of 10. Nobody gives a shit about quality anymore, this is why society is crumbling around us. Profit driven incentive for fast/cheap over everything else. And now I need to spend my day prompting an AI to fix AI slop code to keep the business hobbling along another day. What a fucking joke.
No the materials and techniques matter a lot. This is why we need to build houses with sticks and jute cord, just like we always have. It's vital also that we paint our special symbols above the door to ward off the spirits.
It's insane to me that you're implying we could build houses with pre-fabricated materials or pneumatic nail guns and still somehow "have houses?" No sticks/jute cord and special symbols, then no house.
Personally I use 5 different model families, 3 of which are open weights with 3rd party inference providers (GLM, DeepSeek, Kimi), so if the frontier labs were to shut down it'd be a nuisance, nothing more.
Worst case scenario you just switch to a free model, which are 2025-ish in quality.
The open weights models I am interested in, and testing, learning, experimenting with etc.; I am confused and cynical, not insane.
I am not convinced it isn't vulnerable to the same problems but the whole tenor of the community around open source/open weights models just doesn't have the same YOLO madness to it.
Of course? I'm still better than sonnet or opus, just slower and much more expensive.
Sometimes it takes me a day or more to find the one line fix or abstraction necessary, while claude can hammer through a hundred line fix in under an hour.
Sounds like your definition of better is pretty narrow.
Quick and cheap are two of the three fabled: "Fast, cheap, and good: choose two"
"more good" seems like a pretty decent definition of better to me. The words you are looking for are "cheaper" and "faster"
Are you perhaps missing the true message of that aphorism?
Or are you saying the industry is (because it is)
I am. I have Codex running, doing some tasks which I don't care much about, but anything I want to understand I write myself.
Same thing with hobby projects - I might ask ChatGPT or Gemini some questions about best practices in Swift for example, but writing code is done by hand.
As others said - if you don't use it, you'll lose it. And I'd rather keep my skills up to date.
You have the privilege to keep yourself sharp, most businesses favor productivity over their workers' long term relevancy.
This is the thing that makes me saddest. Second to the fact that none of the management tier promoting and weaponising this insanity will meaningfully suffer consequences.
Right now I am lucky that I have the time to recover and learn.
I write my API specs, my domain models, Postgres schema, and SQL queries myself. Then I'll have AI bots fill in the application details connecting those things since that's mostly boilerplate once you lock in the data structure, query patterns, and API contract.
I never have AI generating database table schemas or really the shape of my data at all.
I still write code and sometimes it works well. I also use Claude and it writes code and sometimes that goes well. We have better success together, where I do the interesting stuff and let Claude write my unit tests, reconcile my documentation. That is to say, I’m using it for quality not quantity. There aren’t enough humans to deploy or consume all the sloppy shit it could write on its own.
I am now in the process of fixing code I wrote using AI. I have come to the realization that AI can't really write software and I am annoyed that it took me that long (months) to realize that.
This is quite terrifying to me, because I have a feeling I will soon come to the same conclusion. I’m starting to see some really glaring omissions in code I’m responsible for (using Opus) that at first (and second) look seemed fine, but really isn’t.
I write code by hand every day. I do the main part of the feature implementation myself and leave comments for the code i want the agent to write. I have some skills and a command that sets the stage to get the agent to fill in the rest
I force myself to do it at least once a week, you know, like cardio. Keeps the doctor away.
Picard should have been a bergamot grower, not a winemaker.
It reminds me of the peak crypto days. Lots of resources consumed, many late nights, little to no value created.
I mean at least crypto provided value to criminals, tax evaders and Trump? (regardless of what you think of that). I don't see a parallel with AI.
Occasionally posts like this do get the attention of the company responsible, more than an email does... but indeed that's like a one in a million situation
I agree with the take not to bother with a sophisticated memory system. Anything worth remembering should be in docs, guides, source comments, commit messages or tickets. You don't need another layer, every conceivable granularity is already covered by existing best practices
> You don't need another layer
I do think we need another layer, but it should be a routing layer. I am finalizing my pi-brains extension for Pi (https://github.com/earendil-works/pi) which does this:
https://github.com/gitsense/pi-brains
Right now "humans" need to define the routing rules for how to access information, but I will support what I call "knowledge agents" that can monitor conversations to inject context when needed.
There is some value to agents being able to query the history of work done, docs aren't a good place to accumulate negative evidence for example, but it can be tagged in traces so that it's efficient to look up as needed. Additionally, docs rot while traces can be tagged with commit hashes and other things that make their lifetime clearer.
The user flow I am trying to get adopted for sessions is to turn them into notes and lessons when you have finished and it should be part of the code review process.
By propery categorizing lessons and notes, it should make it easy to scrub and keep up to date.
I also suggest mapping lessons and notes to files when possible to make discovery and cleanup easier.
Especially a layer that is largely out of band in a project (i.e. ~/.claude/…). In any project where I’ve needed memory I just add a line to AGENTS.md telling it to use MEMORY.md to save memories or STATUS.md to track progress.
I've been enjoying having a little todo file the agent updates as it goes along, because then I can keep track of progress without scrolling through aeons of "Combobulating..."
Also if context runs out you can just do "cat todo.md | agent" and you're off to the races again.
Yep all my projects start with a PLAN.md at the root, and that acts as the ‘save file’ recording our progress over time. My session always ends with updating the plan file with what’s been done, and the next session always begins, as you suggest, with consuming the current state of the plan doc.
This is an annoying problem. It keeps making fake assumptions just because of hypothetical questions I've asked in the past.
It'll assume I own a datacenter and have lots of gpus just because I asked to research things.
Strongly agree here. claude-code’s memory system is occasionally useful but much more often harmful, pulling in obsolete info that muddies the waters about current tasks. I’ve frequently seen Claude’s own memories severely mislead it.
My guess is that has something to do with the training process leaving models unable to differentiate between “what’s happening now” and “what happened before”. Perhaps if making inferences from memories was actually part of the training process things would be different but my sense is that as an inference-time-only feature this just gets the models confused.
Humans make memories constantly, but they also forget things that are no longer relevant. Until Claude can do that, it means the LLM will have an ever-increasing, ever more fragmented context.
And LLMs are NOT intelligent enough to survive even mild context poisoning.
Isn’t this just a form of the bitter lesson? Our attempts to make engineered context and agents will simply be made obsolete with bigger and better models. Those transcripts are probably extremely useful for lesser capable models, and near unnecessary for frontier ones, maybe?
I've wondered this. We have chain-of-thought, harnesses, etc. — workarounds of a sort due to lack of core model capabilities. But I am very curious if much better next token prediction would simply obsolete that whole setup or not. Either way, the answer would be very revealing.
Yeah, the question is whether this applies to all of context management.
I've been using a custom harness based on https://minimal-agent.com/ (itself based on swe-mini-agent), which is like 50 lines for the core logic. Bash is all you need.
For small tasks, I find it's about 8x faster (and uses 8x fewer tokens) than the standard harness for each model.
For bigger tasks I haven't tested it much. It seems to work too but I think they're a bit less focused and productive in that case. It could be that those big harnesses' 20k token system prompts are doing something important with regard to steering software development workflows. (e.g. I heard Fable has a custom system prompt in Claude Code which might explain its markedly more proactive behavior.)
So I want to say there's still a lot of value in context engineering though it seems to diminish with each model release (since they're fine tuned on mostly non stupid behavior and need less hand holding).
> So I want to say there's still a lot of value in context engineering though it seems to diminish with each model release
I can't see how it would diminish unless you are literally working on public domain stuff. Unless stuffing context becomes cost effective and will not affect AI reasoning (this will be much harder), I don't see why context engineering is hear to stay until we have close to AGI.
I don't think so - I think we'll find that to build a brain you need more built-in structure and biases, not less.
Bear in mind that brain architecture is learnt too - just over a much longer timescale than an individual lifetime.
I found that if you allow any low value things into memory, Claude will notice that established pattern and start trying to add low value memories at an ever increasing pace.
I like the memory system, in general. For reference I'm using mostly Opus 4.8 + Max effort. It will often pull things out of memory that are relevant. Like I'll ask it to come up with a few options I should consider for, say, a self-hosted OIDC provider and it'll say things like "Considering the size of your operations team, this might be a better fit because of X and Y".
Now, I'll agree that this is probably the sort of thing I should put in the CLAUDE.md, but in this case it wasn't on my radar to put that in my CLAUDE.md, so it was nice that it surfaced that.
It does sometimes go awry though. Today I was asking about a problem I was having authenticating, and it said "you may be running into this trusted proxy setting because you put your apps behind an haproxy". That is true of 95% of our apps, so it was worth mentioning, but in this case it was not so I had to correct it. But, I'm glad it mentioned it because if we did have it proxied it could have saved me a lot of time.
I remember when OpenAI announced ChatGPT now will remember stuff between sessions. Oh, you mean find random trivia about me and copy paste it between prompts without out my explicit consent.
”compare these three cars. Oh btw I am a data engineer, and my moms maiden name is Joana, and I am allergic to bad poetry. And code should be DRY, I prefer SQL over Python and what’s the most poisonous flower in Scandinavia?”.
I’ve had so much wierd output because context is ”””memorized””” and bleeding into completely unrelated projects and conversations. It’s the first feature I turn off.
It's because it mostly doesn't matter what you are trying to get the code to do. What matters is what the code does.
Session logs can absolutely be useful, but not when building further. It's just that that the place they slot in is during validation. You know, that place between the markdown plan and CI passing, where there's 800 new lines of code and it all seems sort of fine when you click around?
Session logs can show you what sort of manual validation happened. CI will run the tests you had, and the code will show you what new unit tests were added, but session logs can show you that the agent drove the app with Playwright, or that the agent read and considered the prod config as well as the dev config.
Nothing bulletproof, but not every piece of validation work merits a test in the repo that lives forever. We've gotten a lot of mileage out of re-analyzing the sessions, figuring out where the agent made decisions without asking, and forcing the agent to consider validation for those decisions. That's the sort of thing that's hard to dictate up front but easy to highlight with the session logs.
There's a lot of valuable information in there, its' too noisy.
Even with memory off this occurs within a conversation.
It is like an annoying friend, who remembers something from a past conversation, that you have grown and developed from, but they still want to hold it against you.
I specifically disabled claude memory in a project because it kept writing down thigns to memory that didn't need to be in memory, including severly wrong statements that then would confuse it later. At some point it got re-enabled automatically which had me ask claude itself to "turn it the fuck off" by which it promptly figured out that both ("autoMemoryEnabled": false, "autoDreamEnabled": false) are necessary and need to be at the user home settings, not in a project override (which is what I had with the original setup that eventually got ignored by a CC update).
I agree with other commenters here, if anything is worth being rememebered, it will be in code comments, git commit messages, CLAUDE.md or other formal documentation. The auto memory system just causes confusion and leaves stale and outdated information written down.
Its an interesting thought experiment as well, I originally thought that having the model write down memory files by itself would be a nice addition, but after playing around with it, it became clear to me that good as an idea turns out bad in practice because the model can't correctly gauge what deserves being stored as a memory.
> "turn it the fuck off" -> "autoDreamEnabled": false
So you told it don't go the fuck to sleep ;)
Ugh, agentic _dreaming_. Why on earth would I want that?
> I believed this so strongly that my company built an entire product around this concept. I used to tell folks that "session transcripts were the new oil," that they were more valuable than the code itself.
This is infuriatingly common wrt talking/writing about how to use AI effectively. All of the "this is how you write an AGENTS.md" and "you need to talk to it like X to optimize it". Like sure, you can believe that as much as you want but unless you provide some evidence you can keep your shitty CLAUDE.md to yourself and don't pollute the whole company's git repo, thanks.
When nobody actually knows (how to write a CLAUDE.md), everyone’s an expert. Infuriating, indeed. Even more so when people vibe code those files without proofreading.
I have this in my global CLAUDE.md after being annoyed by all the random crap memories.
> Don't start generating an auto-memory entry before asking me. Ask first, write only if I confirm — no speculative drafting.
No more crap after this.
Incidentally I don’t recall Opus 4.8 asking me once in the past few weeks. Older models did ask semi-frequently.
t once had to tell claude 3-4 times to stop assuming the state of a system was the way it kept iterating it was cause it was in it's memory. I repeatably told it to otherwise and it just never updated it's memory and instead kept referencing it's memory about the state of a particular system
In my harness I have all the code auto injected at startup (doing mostly very small codebases).
I found that every model will still manually check every file/function, they immediately assume that anything in context is stale.
That's sensible because often the user edits stuff while they're running.
What it does is save it from having to grep blindly about the codebase. But I think I'd get roughly the same benefit by just dumping the function headers then.
Did you try to delete the memory yourself?
[delayed]
I mean, it’s pretty clear the people who work on Claude Code aren’t actually looking at what they’re implementing. The thought behind this feature seems like it goes nowhere beyond “oh wouldn’t it be nice if Claude could remember things about you? Ok Claude go implement this” and nobody bothered to see if it was useful or helpful.
Settings > Capabilities > "Generate memory from chat history"
Toggle it off and never think about it again.
>We have found zero performance benefit on SWE tasks when agents have search access to their previous transcript sessions
I refuse to believe this is true. The ability for an agent to find information from before a compaction is incredibly useful. At compaction time it's impossible to know what exactly may be still needed.
There has been this slow transition inside me, as someone who likes to not touch the AI as much as possible, where I've gone from skeptical and argumentative about it all to starting to just feel sad for all the Claude et al heads. Like, this is such a ridiculous house of cards you have to deal with all the time, which isn't even directly concerning the task at hand, presumably. Like you're cooking yourself a meal but its just nuking a burrito and then still somehow needing to wash the dishes for an hour.
Not that this isolated article is super damning or anything, but the accumulated set of all these reports has left me only empathetic, I think, of these other devs. Like, I just want to tell them, "it can be ok, it doesn't need to be like this.."
I've been having a very nice time with Fable. I cooked up an Anki clone in like half an hour, with tech it's not familiar with. Nothing too ground breaking, but I was very pleased!
I think Opus might be on similar level for most of what I'm doing, but I haven't used it much recently, so I can't remember the difference. So I guess I'll find out on the 7th when they pull the plug again! (Free-ish trial of Fable ending.)
That being said, I tried using other frontier models to help with a Pong clone the other day and they were introducing new bugs at approximately the same rate as they were fixing it. On Pong!! I found that amusing because I couldn't think of a simpler game, so it didn't inspire confidence.
Fable's doing just fine on an online multiplayer game though. I have no idea how that works. (Maybe it would fail Pong too?? I haven't tested that!)
>> We don't really write code by hand anymore.
The software world is very close to building a super intelligent senior software developer. Companies like this will ask all the best things a software engineer does automatically. Now claude will add it into the coding agents itself.
Damn, I didn't see this coming.
Its first the build the intelligent builder. We will figure out what we want to build later.
Edit: Before more people take it seriously. This is sarcasm. I don't wish this.
> We will figure out what we want to build later.
Once the automator automates itself fast enough, we won't have the ability to opine what gets built. The LLM will decide. Just like right now sometimes LLMs delete tests so they pass, they could just delete humanity if humans get in their way.
> The software world is very close to building a super intelligent senior software developer.
Yeah. Two more weeks, as they say. Just need to iron out some kinks.
It's the error rate. That's what everyone found when they were trying to go Full Auto with OpenClaw in February.
You can rely on it like 95% of the time but that means if you keep it running continuously the error rate rapidly approaches 100%. That's getting a little better with each release, and it might actually hit the point where you can more or less trust it indefinitely (on well defined workflows).
Or at least it would, if context window permitted...
> The software world is very close to building a super intelligent senior software developer. Companies like this will ask all the best things a software engineer does automatically. Now claude will add it into the coding agents itself.
Except Claude is more expensive than an actual senior software developer. Otherwise, why are many companies terrified of the usage bill that gets printed on the invoice?
The nonsense in "tokenmaxxing" was a complete marketing scam and illusion of cheap tokens which in reality were heavily subsidized.
The entire point is detecting bad code before it reaches production. [0] AI generated or not.
[0] https://sketch.dev/blog/our-first-outage-from-llm-written-co...