This is a problem with long context models. To put it as simple and as bluntly as possible: just because they claim you can use 1M tokens in your context doesn't mean its true and you should do that.
Due to extreme quantization of models and the context's KV cache, and also just really shitty samplers provided to the user (hell, most are just getting rid of sampler knobs altogether), this problem will absolutely continue.
Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defects will go away.
> Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defects will go away.
This is just not true. Any local LLM you can host on consumer-accessible hardware has all of these defects, too. Adjusting the knobs doesn’t solve everything.
The closest you can get to frontier performance is Kimi K3, but you’re not hosting that unless your budget is on the order of a nice house in a good metro area.
I like my local LLMs as much as the next person and my office is currently uncomfortably warm from the amount of compute happening, but I would never agree that local LLMs solve all of the common LLM defects. This is peak wishful thinking.
In my experience, the local models and even the larger ones that we can’t run at home suffer more from long context degradation than the frontier models. You are exactly right that you need to manage context length, but even at fp16/bf16 the local models have a lower ceiling for usable context length in my experience.
How do they go away with local models? It's a bug of all LLMs not just cloud vs local. As mentioned in another comment, they did test local models here too and those failed as well.
As parent implies, they're testing the wrong control mechanism. Why are you using policies instead of real controls over the weights and inference pipeline?
Well the answer is that VC-backed companies decided AI is not a domain expert tool for highly competent technical users, it's a magic oracle for the lowest common denominator. So you don't get any of the actually useful controls, just context engineering like that's fucking sane at all. It's like trying to program by navigating a git history. Not writing any new code, you don't have the ability to do that. No, you exclusively have the ability to move around a git history and cherry pick things. It's insulting that they want to charge money for this shit.
> just context engineering like that's fucking sane at all.
Meanwhile every model is at a battle to figure out how to stop you from jailbreaking it, which I assume takes a toll on your ability to prompt. Heck, if you read r/Claude and related subs during the early release of Fable there were a lot of complaints that it would downgrade you heavily.
Look, I enjoy local LLMs as much as anyway, but I think there is some motivated reasoning happening in this thread to try to make local LLMs sound like a utopia against those evil VCs.
At my org we've been building AI agents and one internal rule we have is to use at most 50% of the models context window with the recommendation to not go over 25% for large context window models.
Anytime I see a "1M Context Window", my brain always goes "Gotcha so a 250k usable window"
Why 25% and not 12% or 40%? Is this an arbitrary vibestimate or you had some tests done that pointed you to 25%? I am genuinely interested in how others deal with context issues. Also I would think the usable context window is variable depending on the task, for example summarizing documents vs analyzing large, scattered and complex instructions.
250k because thats what the smaller models supported and theres better training data in that part of the window
there have been some papers suggesting that the useful context is even smaller, and stays fixed as you change the context window size.
as a more general case though, i think the possibilities for what youd need to include in training to have many different paths of text be well represented enough over the long window means the later tokens will almost always be a lot more random than early ones?
I'm not the above poster, but my experiences generally line up with theirs.
The first 20% of context is usually very solid. Agent stays on task, is responsive and focused on the original goal.
The last 20% of context is usually pretty bad. Agent tends to loop, repeat, drift away into sub-goals (or non-goals).
The middle 60% is a bit of a grab-bag. Sometimes it's fine, sometimes it's not. It varies by task, prompt, and luck.
The above 25% and 50% limits seem like reasonable and simple rules to keep agents in the productive zones of usage.
It's generally not worth doing extensive testing to try to find your exact threshold unless you're running exactly the same prompt at scale, on similar data.
If you are in that spot, I won't actually recommend langchain tooling (I've been somewhat frustrated with it) but I think the concepts are worth understanding.
I've used this for testing an agent that handles provisioning and permissions for developers with SaaS tooling (ex - if a developer asks, go provision an account [saml/oauth/etc] with the 3rd party, configure access, and let them know how to use it).
I'm about 6 months out of date at this point, so I won't speak for the latest models, but what was available early this year absolutely suffers from extended context lengths.
The needle benchmarks show, that models extended context works for the part, that can be explained as: "I can access/adress that part of the input".
I have no idea, why in that context, the number of attention heads isn't mentioned. Models have a limited set of them and obviously, a model can focus at N max things at a time, which has to put an upper bound of long context support in some way. There's just more things to lose focus to (or, mismanage the limited attention heads resources - per token)
There are no "attention heads" or fixed number of things a model can pay attention to ... or at least not exactly.
After every prompt the model decides "I have a weight of 1 to distribute between every token in my context". If you have ten tokens, each gets a weight of 0.1 ...
... except it's not that simple, because the LLMs don't distribute that "attention budget" equally. If your prompt was "where is Paris", then any tokens in context it can associate with Paris will get a greater share. If the word Paris is in your context, it might get 0.3 or 0.4 weight, and close by tokens might get 0.2, while other (unrelated) tokens get 0.03 or something.
Now, add lots of context, and you start to see the problem: more context = greater distribution of the attention budget. Even if the prompt is about Paris, and the Paris tokens get higher weights, they are only getting (say) 0.002 ... while unrelated tokens are getting 0.001.
All LLM "answers" are just math, computation, based on the context and those weights. If it can't "focus it's attention" because it's distributed among too much context, it's far more likely to miss the relevant tokens (eg. the Paris ones) and give you an answer that ignores key parts of context.
But again, there's no fixed number of things it can pay attention to: it's a gradual degradation of the chance of it seeing key info it should, based on the amount of context.
Just because something is called an "attention head" doesn't mean the terminology makes sense.
If I'm reading the article right, hungryhobbit is accurately describing a single attention head.
And... what having multiple attention heads means is that you do the "single attention head" thing several times, and average the results. There is no part of hungryhobbit's comment that changes if you make this adjustment.
In particular, he is clearly responding to this clause in the comment above:
>> the number of attention heads isn't mentioned. Models have a limited set of them and obviously, a model can focus at N max things at a time
Those are the "attention heads" that hungryhobbit is saying don't exist. You'll notice that this definition of "attention head" does not correspond to any part of an LLM; I fail to see the problem with saying that "attention heads", as defined by this clause, don't exist.
There quite literally are internals that are called attention heads in modern LLMs. Saying 'there are no attention heads' is objectively false, which definition you follow is irrelevant.
One of my early experiments last year with open source models and context size was with GPT-OSS 20B (the mxfp4, the "smart" 4-bit quantization). Even though it boasted a 128k context size it was bad at recall around 32k characters (didn't bother to implement the tokenizer for counting).
The recall text was a simple hash generated, filler text from a dictionary file and a request at the end to return only the hash from the beginning of the prompt. Past 32k characters the response contained hallucinations of characters or full hashes.
Just having large context size doesn't paint a full picture of capabilities, prompt adherence and other quality metrics.
This is conspiratorial speculation though. They did test many open weights models which spanned the full spectrum of performance: Nemotron 3 Ultra second-worst, GLM 5.2 top five https://arxiv.org/html/2607.25398v1
If you have a semi-recent MacBook with even 32GB, you can run 20GB models that are pretty damn smart, with room to spare for the rest of your toolchain.
If you’re reasonably connected to the code you’re writing and prompting the AI at the level of the code, not the level of the feature, you can get some fantastic results.
Sure, it’s not the completely automated dreamland that’s been sold, but it’s still a speed-up on par from going from assembler to a higher-level language, which is still immense.
And for effectively free, if you have a machine that would otherwise have been considered “developer-grade” for a lot of tasks anyway.
What models have you found to work for which tasks? I find the local models very useful but not in a way that's replaced cloud models (yet, I remain hopeful).
A good setup will cost you the same as a decent house in a major metropolitan area. And then the price of a used car to upgrade your electrical to handle that kind of load, and get the cooling setup you need.
Long-context models do work as advertised... just not when combined with multi-turn conversation sessions.
It's my understanding that, for all current models having long-context capability, the "parts" of the model that allow them to do long-context processing, are mostly-untouched descendants of academic model architectures; where these academic models were designed around a very constrained use-case assumption: that anyone using more than a "normal" amount of context would specifically be trying to apply a reasonably-sized prompt to an unreasonably-sized block of plain-old embedded data, as a single-shot conversation. (Consider a task like e.g. "summarize this entire book." Context = [prompt] + [text of book].) This is the use-case all these architectures were refined and benchmarked around. And even today, this is still basically the only way these models' long-context processing capabilities work.
---
But I think it goes deeper than that. It's not just about how we've made long-context work so far. It's about what "attending to context" really means, and what that implies about what we'll ever expect a long-context model to be able to do.
I would describe long-context models as seeming to have two distinct types of "attention state" used when attending to the layer-1 input-vector / context window:
- They have some small amount (e.g. 8-32k) of regular "high-quality" attention state. This is likely inherited from the base model they are trained on top of.
- They also have a large amount of "low-quality" attention state. This is what the long-context training process has created.
The "high-quality attention state" isn't limited to referencing the first 8-32k of the context window, to be clear. The high-quality attention mechanism is just limited to "choosing" 8-32k of arbitrary data samples, each from its own arbitrary position in the context window, to form a bounded-sized vector upon which high-quality attention is then computed.
(You could think of this as the high-quality attention mechanism computing attention over the entire context, and then doing top-K sampling, i.e. zeroing out all but the K largest-valued elements of the resulting latent vector, where K=8-32k. That's not what it's actually doing—that'd require paying the very quadratic scaling cost that prevents us from just increasing the short-context window size—but it's a useful naive model for what it's doing.)
---
It's harder to directly explain what the low-quality attention mechanism does, so let me try an analogy.
(While no one can yet give a definitive accounting of the abstract computational processes the matrix-multiply-ops against a given LLM's weights encode, we can still speak of possibility-spaces, or analogies via information/computability theory on what sorts of abstract operations the LLM "must" be performing to achieve the results it achieves. This is that kind of thing.)
Think of an LLM's evolved context-window attendance mechanisms as being somewhat akin to the LLM being a regular CPU program, that has access to a 1. block of raw addressable memory (i.e. the context itself), and 2. a block of pointers into that memory (i.e. the learned first-layer attention over the context.)
Our LLM program's logic, through mutations during training, gradually evolves into a form where it could be seen as implicitly declaring and using a growing set of variables: some bound to static addresses in the block of raw memory (i.e. static context-window positions), and others bound to static addresses in the block of pointers, where the raw-memory-address-value of the pointer (and thus what data from the context-window it will read as when "dereferenced") is the result of a dynamic calculation over one or more other variables (i.e. an arbitrary weighting of the layer-1 input vector, where the resulting vector element is used as a selector, in layer 2, among other data that got sampled into "passthrough" vector-elements back in layer 1.)
Under this analogy, a model's "high-quality attention state" is the set of "named" pointer variables that it directly references in its logic.
As should be obvious, there can only be so many of these; you really fundamentally can't scale them up, without literally having "more logic" that defines and makes use of them (i.e. scaling up the size-in-weights of the model's layers.)
A model's "low-quality attention state", on the other hand, is the block of pointer memory itself, which can be as large as you like. Presuming the computational substrate permits the logic to compute on the addresses of memory within that pointer-memory block and then double-dereference them to get context data (and it does seem to!), your LLM's evolved logic can do "dynamic random-access reads" of arbitrary parts of the long context.
But, of course, the model won't be doing anything special with that data. It can attend to it, but not in a way that's unique to that data. It pulled in such data through generic logic that doesn't "know what it's looking at", after all.
---
IMHO, this analogy helps to make clear that while giving a model a raw "large-context capability" might help said model to attend to more data, it doesn't really help in making a model able to attend to a larger prompt.
Just by information theory, an instruction-following capability must imply that something analogous to logic keyed on those instructions gets embedded into the weights; and that logic, in order to activate and work, presumably requires its own state — i.e. requires that individual values from the context are getting attended into additional top-level named variables.
And while the short-context attendance mechanism does that, the long-context attendance mechanism inherently does not / cannot.
And this means that there is likely a practical upper limit (in terms of practical model-weight size + latent-vector size + etc) to "how much prompt" a model can follow during a single inference step. Regardless of how much data the model can "see" on a given inference step, its active "program logic" is only defined in terms of so many concurrent stateful computations.
---
Thus: trying to use a current long-context model's "low quality" attention to attend to a large prompt is just going to result in nonsense. None of these model families have ever been trained to follow a million sentences of rules simultaneously when answering a question. They're just going to attend with their inherited bounded "high quality" attention to your prompt as best as they're able, while forgetting literally everything about the prompt that doesn't fit in that bounded "high quality" window.
And anything that allows current long-context models to appear to interact well in multi-turn conversations with memory + KV caching + etc layers in operation — while also making use of its "low quality" attention to attend to large data — is some kind of hack, and will break like a hack.
Yeah checks out with my anecdotal experience with Claude. It is pretty great at following instructions - for about 10 minutes, after which it seems to ignore things I told it before.
I have quite explicit and strong instructions (e.g. don't write massive comments, use existing functionality, etc.) in CLAUDE.md files which seem to get bypassed surprisingly quickly when doing real tasks. Yet if I tell it these things in a prompt during the task, it performs way better.
Result is I'm trying to resist adding more and more things to CLAUDE.md files which in some scenarios it does well but in other scenarios totally ignores and messes up.
This is not what the article is talking about. Its talking about policy documents not it forgetting something 5 prompts ago. In fact you adding things to CLAUDE.md is more what its talking about.
Conceptually the same thing though. Claude.md is sourced at the beginning of the session, so will be pretty far back in context, just like user prompts from the beginning of the session.
Claude is a next-token predictor, appending to a long text document. Prompts aren't an independent kind of thing from policy documents. It's all text in the backscroll.
I’ve had a lot of success using the root Claude.md for a handful of high level application wide rules and directions (I keep it pretty small), module specific claude.md in subfolders alongside the code with more specific rules and direction, and a custom rules backed /code-review skill that enforces it all and catches anything that was missed during implementation.
I believe the correct static instructions are about getting it at the right starting point for whatever class of projects you're working on; not as a continued referencable or "HOWTO" of what it's doing. They're all just "grooming" the LLM for future instructions.
The coding harness is what's getting it to continually align to your current instructions.
Ok so what is the correct way to tell it "I don't care what is happening, you must uphold these rules at all times"? If it's not any configuration of .md files?
You need to let go of the idea that this is something LLM's can do. They can't. At best, they can bias towards rules conformance with more or less likelihood, but coverage and conformance both go down super-linearly as you accumulate more rules, more context, and more output in a session. That's simply the nature of how these tools work and you need to engineer your workflows around it if you want to use them.
If you absolutely need some rules enforced, you need to adopt some framework for validating those rules that then rejects, reprocesses, or repeats any session that fails to satisfy them. In the best case scenario, this is some traditional deterministic validator (like a linter, compiler, analyzer, exhaustive test suite, etc in coding) but if you need to process in stochastic space because its something rich and ambiguous like natural language itself, then you want to dispatch a swarm very narrow, task-focused subagents that each validate against a very constrained subset. (And prepare yourself to have those to fail sometimes too. LLM's are noisy and cannot deliver strict rule enforcement on their own.)
You need to make the rule concrete somehow. I call it a "control". So for example, instead of instructing it "always run tests before committing", you (or you have it) make a git commit hook that always runs the tests first and that refuses the commit if they don't pass.
In this case, it is an advisory control only, because the LLM can also unhook that hook. And of course, it could also just disable the failing test(s) with some bullshit reason. But it is far better than assuming it will comply every time.
Then there is the "hard control", which is the inviolable that the LLM cannot bypass.
You need to move as much as is technically possible to either hard or (failing that) advisory controls.
>In this case, it is an advisory control only, because the LLM can also unhook that hook
you can in principle make a hook in the harness itself that will run an auditor prompt that checks it adhered to the policy, correct the model and also make it known in advance.
I've seen pretty much every model including Claude bypass commit hooks constantly if there's even a tiny bit of friction, instead of pausing and asking me to help fix why it can't run the hooks (usually something that needs to be done outside the sandbox like `npm ci`).
That's why it's critical to have these checks run in a context that the model can't bypass, such as via github actions that block PR merges. Generally works pretty well for stuff at work (where we have all this stuff set up as part of continuous integration checks), but it means that for personal stuff I have to set up quite a bit more infrastructure to ensure that models don't just skip running tests (which they LOVE to do).
I think things like CI, linters, hooks, etc started as defense against humans eventually ignoring it (low blood sugar, under pressure, never cared in the first place, etc).
These seem like they would naturally extend to LLMs as well.
(I'm not pro-LLM on balance, but the solutions of today feel similar for humans and LLMs.)
Ask it to come back with a filled checklist and hand it over to a different agent with a fresh context window (three lines model). Or make it collapse the context and get back to the checklist.
Subagents whose only job is to review the actions of your other agents for rule compliance? It works reasonably well for me in complex workflows using Claude Code.
Can I ask how you set this up? Like is there some way to have that run automatically, similar to “auto mode” for approvals, or do you have to invoke it regularly?
Sure! I use an orchestrator main agent whose only job is to run subagents through the workflow process I've defined. Part of that workflow is to invoke a review subagent at particular points to check the spec, the implementation plan, and the code for rule conformance.
The review subagent has its own definition and gets invoked with a specific target, so the context is very focused on just rule enforcement and I don't have problems with it skipping rules.
The whole workflow is packaged up as a plugin, but you don't need that to get this approach to work. It should be sufficient to have the rules you want enforced written out somewhere, and to either kick off a focused review agent manually referencing them, or do something like I did and have it be a defined part of your workflow (depending on the automation level you want).
in the plugins I'm using, it's basically _always_ adding information to the context. You're not going to do it manually; you can't go back in the context and add it because that'll break the cache. The way your programming harness works is by constantly reminding the LLM of the tools available.
A good programming harness is basically a stack. A good stack keeps building each layer. You _cannot_ pull things off the bottom of the stack because that's an expensive cache hit; but you can pull things off the top. So what your harness should be doing: <tools> <rules> <user content> onto each request _then_, when you get to the next request or result, pulling that out if there's some change.
So you can see it's the cache that's either exponentially growing or having to cache bust to keep it fresh.
As a hobbyist, I find it difficult to figure out how to make Claude stick with some repeating things I want it to do after every major action, like re-evaluate the completeness of tests, update the documentation, etc. And CLAUDE.md/AGENTS.md definitely did NOT help there, sadly.
If the harness runs on your localhost, but the inference doesn't, it usually means it's calling some API. Whether you want to break TOS of your provider like that or simply buy tokens from our friends from UTC+8 timezone is an open question.
This is the way. Making your own agent to have a sticky memory context that is prepended to every execution is necessary to ensure each task is bounded by those precepts.
The trick I'm doing -- the model is given a tool that runs a prompt in the current thread to consolidate it's working memory and identity (it has a memory tool bound to the agent persona). When the prompt ends, the parts of memory that are marked as identity are merged together into a new system prompt, then the context restarts with only system prompt and this tool call surviving. Then it just keeps going.
<project_prompts> //Duplicated with new user messsage
<llm_response>
It wouldn't bust cache, it would just make your input prompts slightly larger. Technically inefficient as you're duplicating the same rules over and over but I imagine for a smallish checklist/principles that it is tremendously more efficient than a cache break every message
so first, we treat context like a stack. Things are popped onto the stack; if we try to remove anything but the top of the stack, we'll break the cache.
So a good harness should be popping tool prompts, user hints after every return from the top so the context of tools and user hints never repeats and are allways prefixed to the current user prompt.
I've never inspected any harness, but opencode with the dynamic context compaction plugin appears to do this well.
What I'm currently doing in a large refactor, is I created a super-run script; the super run script is devided into super-dev (Setup dev), super-test (run all tests), super-build (build artificats), super-e2e (test all artifacts), super-deploy (deploy finished).
Each super's sub functions should _fail hard_, and each script should be highly detailed; of course I'm not doing it myself, but in small increments of directed work, it can build up the necessary harness.
What I get is a CI that just starts with "run super-run.sh" and that gives it context, then each sub script provides context depending on if it succeeds or fails. If it fails, the agent is provided what it needs.
It's basically, you have to design the products of the AI to give itself the context. Another technique I'm testing out is a parallel set of files like <subject-module>.js, <subject-module>.test.js, <subject-module>.md which get pulled up if the Agent is looking for a file.
Any model with a good score on this benchmark would have a good claim on superhuman abilities. Humans are pretty terrible at being thrown a long policy document and being expected to follow it
And while we shouldn't anthropomorphize these models too much, I wouldn't be surprised if many of the core reasons for failures are similar. Working memory is a limited resource; you can only focus on so many things at once; reasoning depth is limited; and many real-world policies are not actually meant to be implemented in the same way they are written and have insufficient specification of edge cases
With humans, we usually do the equivalent of RLHF, both via "training" with simulated cases, and via feedback while on the job. You would never hand a newbie a 124 page policy document and expect them to correctly apply it on the first task, or to do it reliably in the first month
The challenge with comparing these things to humans, is that humans learn. A newbie might not respect your organization’s set of policies on day one, but what about 3 months in? Or 3 years? Meanwhile there’s still no reasonable mechanism for automatically fine tuning LLMs or adjusting their harnesses to make them better at completing your organization’s objectives more successfully. They’re still overwhelmingly governed by the shared weights and harness policies found to be successful for the average case.
LoRAs or even full fine-tunes would be much cheaper than that, and with some investment in the right infra could be updated regularly. And at least LoRAs can be swapped in and out cheaply, making them usable in large-scale inference providers. But there seems to be limited appetite in offering this. Both Anthropic and OpenAI no longer offer fine tuning for current models
No, it's more than that. We can't remember everything. I hired, say, two years ago; as part of my onboarding process I had to read a bunch of policy and procedure documents, which were full of stuff that I didn't understand because I wasn't really in the context yet. So at the time, to me, those documents were full of arbitrary text that I didn't really understand. Some of it was rules that I had to follow, but at the time I didn't understand why, so it's just arbitrary rules.
How many arbitrary rules can you memorize? Do you even remember them two years later? If you do, then we can get to your statement.
And your statement is true. Humans do not run every action through a memorized list of rules, to see if any of them block the action. We don't. We're not going to, either, no matter how badly the policy manual writers want us to.
That's a great comparison, human vs ai on a wall of text.
The problem is that it doesn't fit the sales pitch of LLMs and agents - humanlike or better, repeatably, 24/7, for a fraction of the price, you just need to make sure that you give it all the rules.
Unfortunately we can't really have a meaningful conversation until the money vampires have left so we will need to reschedule this until after the bubble.
This is real. I was very angry that AI kept breaking the rules I wrote, so I asked Claude to crawl its own history logs. And then I found: every time after it breaks a rule, the chance of breaking rules goes up.
I feel this is like few-shot in reverse. Few-shot is supposed to be good examples AI should follow. But when it breaks a rule, we correct it, it keeps breaking, and this whole thing actually raises the probability of more violations.
I wanted to know if there is any difference between writing rules in the prompt, writing them in CLAUDE.md, or not writing at all. So I did some short tests before. I asked Claude to open brand new sessions, test different topics with the rules I want to apply. The result turns out to be: in a fresh session, no matter the rule is in a prompt or in CLAUDE.md, models (Opus 4.8, 5 or Fable) all follow it fine, across models. Even Opus 4.8, the one always violates rules in our conversations, does it well.
I suspect it's the long context that breaks rules. But simulating a long conversation experiment is kind of hard, I still haven't found a good way to test it. So seeing this paper now, it completely answers the question I was stuck on these few days.
Besides, something caught my eyes in this paper: sometimes the model does run the check by the rules, and it really finds the violation, but its narrative still insists on its original wrong output.
My current approach is same as everyone here: use a separate hook or post-check to fix things. Because if you let the model fix it during generation, its narrative or main generation part sometimes just rejects the rule error it found.
Evoking Gödel via GPT, with the requisite em-dash: "No sufficiently expressive formal governor can be simultaneously consistent, complete, and able to prove every relevant proposition—including all propositions about itself."
But "Read AGENTS.md, including the middle" sure helps.
Most people don't understand that 'agentic AI' is a completely synthetic, force fed capability by extensive Reinforcement Learning on synthetic domain specific 'agentic' datasets on post training.
If the LLM wasn't post-trained to adhere to specific handbook, it just won't work. If the LLM wasn't trained on an use case the lab decided was worth making a synthetic agentic dataset, it won't work as well as you want.
There's a reason the main agentic task LLMs excel at are coding tasks, it's the way of working of the creators, and they understand intimately the flow and can train for it.
I believe the true way will be able to easily fine tune models on your agentic use cases, but it would require a big company to compile a huge dataset on it's way of working and I don't think anyone wants to be the first.
In terms of long context, accurate attention retrieval from early tokens is just impossible, given the expansion of RoPE encoding for the positions, or in case of Kimi that don't use it anymore, as well as deepseek, early context is heavily compressed you lose accurate information.
If people spent more time studying about AI and how it works, they would realize that the default should be to one shot prompt your task with a big, cached system prmopt, with an user prompt that is just dynamic data, specified to the cheapest model that can do the job.
Unless you really can't do this given your problem, you should try to make a graph of well defined, step by step oneshot prompts, and THEN if your problem still can't be solved with that, then you start leveraging agents.
Despite this giving better results, and being more cost efficient, is evidently too much work then just letting the AI do all the work.
Most people jump straight to agents when what they actually need is a graph.
Example: a mining company receives free-text reports from field geologists. You could have:
Geologist report -> LLM call extracts minerals we are looking for (you inject a db query result on the user prompt), locations, assay mentions and risks into structured fields -> LLM call classifies evidence into positive indicators, negative indicators and unknowns -> LLM call estimates deposit potential and confidence -> database lookups inject regional ore demand, nearby deposits, infrastructure and historical yield data -> LLM call combines geological evidence with business context -> LLM call generates an investment recommendation and rationale.
That's what I mean by graph. Every step is a separate LLM call with a well-defined responsibility, consuming the output of the previous node. Each node can be tested, benchmarked, retrained, replaced, or monitored independently.
Why would you use an agent here? You can cache every single system prompt on each call making your total token output much cheaper than having a full 'output' only token generation workflow which is what happens with agents.
There is nothing to discover. The workflow is already known. The company already knows how geologists evaluate prospects. The company already knows what data sources matter. The company already knows what the final output should look like.
You don't want the model deciding which tools to call, which reasoning path to take, or which pieces of information are important every single run. You want the exact same process applied to every report so results are consistent, measurable, auditable and debuggable.
My default is:
One-shot prompt -> if not enough -> graph of LLM calls -> if not enough -> agent.
A surprising amount of enterprise AI is really just:
Unstructured input -> extraction -> classification -> enrichment from databases -> decision support.
Not:
Unstructured input -> autonomous agent spends 20 steps deciding what to do next.
Agents make sense when the workflow itself is unknown.
If the workflow is already understood, a graph is usually cheaper, more reliable, easier to evaluate, easier to debug, and less dependent on whatever synthetic "agentic" behaviors happened to get reinforced during post-training. I am sure people default to agents mostly because it's less engineering work than explicitly modeling the process.
Thanks, that helped. I get it. Yea, that’s the harness executing the workflow with the LLM being called at the right time. That ensures the process is consistent no matter what, in contrast to the agent calling out to tools and possibly doing different things every time. Basically, standard code being in control rather than the LLM being in control. Fully agree with this model. We should use deterministic code when we want the same process or algorithm every time and choose LLM callouts when we want “fuzzy” processing that is not as deterministic.
I jump straight to graphs (unless it truly cannot be solved with a graph), but then the stakeholders get upset that I didn't use agents. Doesn't fit their marketing plan, I suppose. I don't mind dealing with that, but I can see why most jump to agents in order to avoid the human conflict.
> In terms of long context, accurate attention retrieval from early tokens is just impossible, given the expansion of RoPE encoding for the positions, or in case of Kimi that don't use it anymore, as well as deepseek, early context is heavily compressed you lose accurate information.
"Every gambler knows the secret to survivin' is knowing what to throw away, and knowing what to keep." - Kenny Rogers
Humans have limited context, just like AI. The difference is that humans - at least some of the time - can figure out which pieces are more likely to be important, and therefore prioritize keeping those in the context.
I don't understand why a model has to follow the instructions. Don't get surprised when it shows its true color. Plus, do users (not the researchers) really check whether the response follows the instructions?
Long policy documents are also a problem for human agents. Without special training no one will retain 180 pages HR employee handbook, fire codes, OSHA safety rules, FCC regulations, the US legal code.
If the stakes are high, e.g, proceeding in ignorance could lead to prison time, people will favor inaction, even if the policy technically permits a corner case. If the stakes are low, people will completely override policy for the path of least resistance.
There was an article a few years ago called "Lost in the Middle: How Language Models Use Long Contexts" https://arxiv.org/abs/2307.03172
From my experience this holds true to this day. It was one of my core observations for similarity to the limitations of human working memory on "Engineering for Bounded Cognition"
This article to me also implies that there are some potential issues with large Spec based development flows, which I haven't been able to pin down lately.
Specifically, having agent implementation drift from the Spec.
Drift is huge between any large spec and agent implementations.
I've done a ton of testing and the model doesn't matter, fable or sol still miss a ton of detail and drift.
I'm building http://engine.build which closes the gap and makes sure the implementation matches the spec.
It's not the same as the satisfaction you get when solving complex problems with code yourself but writing clear specs and thinking through the problem is still very satisfying to me.
For Claude, I used a UserPromptSubmit hook running inject_rules.py which reads RULES.md from the disk and prepends the whole thing to every prompt. That helps the rules to stop fading as context fills because it is reinforced every prompt.
Sure, it uses tokens slightly faster in the prompt, but I find it reduces overall token use, you can use it with pro, but of course, nothing works 100% of the time, but it's better. Emptying the memory helps too to avoid Claude making up stuff that messes with how I want it to act.
The general gist of inject_rules.py is :
RULES_PATH points at RULES.md
reads it with encoding='utf-8-sig' so the BOM is stripped
wraps it in a JSON object — hookSpecificOutput.hookEventName = "UserPromptSubmit", additionalContext = a preamble plus the full rules text
the preamble is the line you see above the rules: rules are in force for this turn, run rule 33's five tests before raising anything unasked
prints that JSON to stdout, which is how Claude Code takes it in
on OSError it returns 0 silently — if RULES.md is missing or unreadable, nothing is injected and the turn proceeds with no rules
Control vectors for the win? It feels like the way to fix this is to pull out a control vector immediately after processing the handbook and use that to steer later inference. That should stop the drift over long distances, but it doesn't guard against the handbook itself being too big.
I noticed this behaviour a few months back, I think I was using Sonnet 4.6 at the time... I have strict rules about comments in the codebase, this all for personal projects, and the reason I restrict comments is to keep the token count low.
At some point between the model i was using and the previous version of it, Claude started inserting massive comments with references to tickets and other tasks. All this while having specific directives on the CLAUDE.md
Since then I resorted to developing my crapware as if I was the floor manager of a vehicle assembly line, and I have a few highly-specialized sub-agents running errands around the main session, but only ever taking care of a single concern. The main session builds with the knowledge contained in things like CLAUDE.md but the sub agents make sure things like the no/low-comments directives are either enforced, or factored into the final product.
LLMs are not people. Employee handbooks, like all other social rules, are not self-executing; they depend on the fact that human beings are innately social animals.
simonw has been correctly asserting that there is no deterministic way to prevent hallucinations. I’d argue that this obstinance could be the seed of a strong argument in favor of their capacity to eventually coalesce some form of consciousness
Codex has been pushing things to my main branch all week despite me repeatedly telling it not to and adding to my AGENTS.md very clear instructions for creating feature branches and putting up a PR. It keeps doing it in spite of all that.
I'm probably going to need to enable branch protection on my personal projects... What a pain.
What really helped me was to run any .md/prompts through an LLM to find contradictions, duplicates or ambiguities, repeatedly. That led to agents much better following instructions.
When "performance" breaks down over long lengths, one could attribute it to a lapse in attention, but one could equally suspect inconsistent instructions.
The fewer instructions and conditions that need to be simultaneously met the easier it is to comply, but with more and more instructions one is bound to introduce internal inconsistencies within the instructions.
For my own (rather idiosyncratic) harness I've been experimenting [0] with "compiling" long markdown specifications into small executable logic programs. It's too early to tell for sure, but I believe that this approach does have its merits when you want some guarantees about how your agents behave for longer tasks.
Interesting idea. I’ve been noodling about something similar myself for a few months, but I haven’t moved forward with testing it. What sort of outcomes are you seeing with it? IMO, we’re never going to get to AGI without fusing “soft” AI decision making with “hard” logic and symbolic algorithmic reasoning. Humans don’t realize this most of the time, but we routinely use them all.
Opus 4.8 (max thinking) scored highest and Grok 4.3 lowest
It's hard to understand what's going on with Grok. It's like it has capabilities in a theoretical sense but maybe the training is so focused on being in x.com/grok.com with the web search tool enabled for "is this true?11" type queries that with any API type usage with document workflow instructions, tool use, code gen etc it completely falls over
Dealing with agents/LLMs based on "instructions & guidelines" has taught me - nothing (un)reliably governs agents other than agents themselves or (rather i.e.) their motherships (assuming they can and they intend to). Or if you add ton of local tooling.
Why would anyone think that models optimized for efficient context management, giving much more weight to a short sliding window, would attend to distant, heavily diluted tokens?
Plus the model's capacity to take more context into account and actually integrate it to the output is simply limited by the number of activated parameters. If you give it a playbook, you are forcing to choose it between attending to the playbook and the task at hand.
If you want to force it to work step-by-step, you need to present the steps one-by-one. Ideally with rules for the current step at hand and maybe relevant input again, depending on overall task size.
Why did you think models love to re-read files before editing them? It increases recall quality and thus edit precision and thus benchmarks.
To run at decent speed, all models try hard to use only most likely relevant part of the context and most likely relevant weights (MoE) to predict the next token. Doing the math in full is unfeasible.
yep all the advice about context engineering, harness whatever is so silly. ai doesnt give a flying fuck about some IMPORTANT instruction in your claude.md.
It does what its has been trained to do. So find out what its trained to do and just use it to do that. This is not general intelligence.
I got pumped seeing the OKF format from Google (which is just a standardization of wiki pattern) but quickly realized it could not yet combine many subtle concepts together in an efficient way.
This is what spec driven development tries to solve.
Multiple rounds of generating small contacts documents that grow from the original idea , trying to keep each slice small enough to process for a human to approve/disprove .
Eventually it leads to a long list of tasks grouped by functionality. You start a new context and the orchestrator agent dispatches tasks to sub agents with a limited amount of information provided to each sub agent.
Also should have adversarial review and approval gates with other agents and roles.
Having only dipped my toes in generative ai recently I'm surprised how small even a 1mio window is. A semi serious project can take several session in one sitting. Especially as degradation sets in waay before the window is full.
Dont try and handle the entire project in context.
Use a well structured filesystem layout for your code with a few lines in an AGENTS.md describing the layout and core architectural requirements (no more than that, as per the article!).
Then work on small-medium tasks at a time with a fresh context.
At the end of your task, ask the agent if there are any key points about the project layout or architecture it would want to add to memory - audit those manually and amend your AGENTS.md accordingly.
If your code is well structured and you keep your tasks localised, you can get away with seemingly minuscule context windows.
It's also worth noting that high effort models love to slurp up whatever context they can. You almost never need/want high effort for non cross-cutting tasks.
HANDBOOK.md is a benchmark for long-context agentic instruction following, modeled on how enterprise employees follow company handbooks in their day-to-day work. Each task is a unique RL environment with internal tools and external MCP servers, spanning five enterprise domains: Finance, Medical Billing, Insurance, Logistics, and HR.
The prompts reflect the actual jobs enterprise workers perform every day. Each task drops an AI agent into a live company environment, requiring them to cross-reference an extensive, multi-section handbook against a cluttered inbox, a multi-channel Slack workspace, Jira queues, and a stack of files (spreadsheets, PDFs), and working out both what to do and what the handbook forbids.
This is a problem with long context models. To put it as simple and as bluntly as possible: just because they claim you can use 1M tokens in your context doesn't mean its true and you should do that.
Due to extreme quantization of models and the context's KV cache, and also just really shitty samplers provided to the user (hell, most are just getting rid of sampler knobs altogether), this problem will absolutely continue.
Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defects will go away.
> Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defects will go away.
This is just not true. Any local LLM you can host on consumer-accessible hardware has all of these defects, too. Adjusting the knobs doesn’t solve everything.
The closest you can get to frontier performance is Kimi K3, but you’re not hosting that unless your budget is on the order of a nice house in a good metro area.
I like my local LLMs as much as the next person and my office is currently uncomfortably warm from the amount of compute happening, but I would never agree that local LLMs solve all of the common LLM defects. This is peak wishful thinking.
In my experience, the local models and even the larger ones that we can’t run at home suffer more from long context degradation than the frontier models. You are exactly right that you need to manage context length, but even at fp16/bf16 the local models have a lower ceiling for usable context length in my experience.
Do you work for a frontier lab by any chance?
How do they go away with local models? It's a bug of all LLMs not just cloud vs local. As mentioned in another comment, they did test local models here too and those failed as well.
As parent implies, they're testing the wrong control mechanism. Why are you using policies instead of real controls over the weights and inference pipeline?
Well the answer is that VC-backed companies decided AI is not a domain expert tool for highly competent technical users, it's a magic oracle for the lowest common denominator. So you don't get any of the actually useful controls, just context engineering like that's fucking sane at all. It's like trying to program by navigating a git history. Not writing any new code, you don't have the ability to do that. No, you exclusively have the ability to move around a git history and cherry pick things. It's insulting that they want to charge money for this shit.
> just context engineering like that's fucking sane at all.
Meanwhile every model is at a battle to figure out how to stop you from jailbreaking it, which I assume takes a toll on your ability to prompt. Heck, if you read r/Claude and related subs during the early release of Fable there were a lot of complaints that it would downgrade you heavily.
And what is the difference between cloud modeld vs local inference regarding this? It is still unclear in this thread - asking out of curiosity.
> Well the answer is that VC-backed companies
Look, I enjoy local LLMs as much as anyway, but I think there is some motivated reasoning happening in this thread to try to make local LLMs sound like a utopia against those evil VCs.
Local LLMs suffer from the same problems.
At my org we've been building AI agents and one internal rule we have is to use at most 50% of the models context window with the recommendation to not go over 25% for large context window models.
Anytime I see a "1M Context Window", my brain always goes "Gotcha so a 250k usable window"
Why 25% and not 12% or 40%? Is this an arbitrary vibestimate or you had some tests done that pointed you to 25%? I am genuinely interested in how others deal with context issues. Also I would think the usable context window is variable depending on the task, for example summarizing documents vs analyzing large, scattered and complex instructions.
250k because thats what the smaller models supported and theres better training data in that part of the window
there have been some papers suggesting that the useful context is even smaller, and stays fixed as you change the context window size.
as a more general case though, i think the possibilities for what youd need to include in training to have many different paths of text be well represented enough over the long window means the later tokens will almost always be a lot more random than early ones?
there's a sheer amount of bits problem.
I'm not the above poster, but my experiences generally line up with theirs.
The first 20% of context is usually very solid. Agent stays on task, is responsive and focused on the original goal.
The last 20% of context is usually pretty bad. Agent tends to loop, repeat, drift away into sub-goals (or non-goals).
The middle 60% is a bit of a grab-bag. Sometimes it's fine, sometimes it's not. It varies by task, prompt, and luck.
The above 25% and 50% limits seem like reasonable and simple rules to keep agents in the productive zones of usage.
It's generally not worth doing extensive testing to try to find your exact threshold unless you're running exactly the same prompt at scale, on similar data.
If you are in that spot, I won't actually recommend langchain tooling (I've been somewhat frustrated with it) but I think the concepts are worth understanding.
You can do something like https://docs.langchain.com/oss/python/langchain/test/evals. Trajectory evals in particular seem to suffer as context length grows (https://docs.langchain.com/oss/python/langchain/test/evals#t...).
I've used this for testing an agent that handles provisioning and permissions for developers with SaaS tooling (ex - if a developer asks, go provision an account [saml/oauth/etc] with the 3rd party, configure access, and let them know how to use it).
I'm about 6 months out of date at this point, so I won't speak for the latest models, but what was available early this year absolutely suffers from extended context lengths.
In my own experience (both Claude and Gemini 1M), you start running into problems around 60-70%. A 25% ceiling sounds like total cargo culting to me.
The needle benchmarks show, that models extended context works for the part, that can be explained as: "I can access/adress that part of the input".
I have no idea, why in that context, the number of attention heads isn't mentioned. Models have a limited set of them and obviously, a model can focus at N max things at a time, which has to put an upper bound of long context support in some way. There's just more things to lose focus to (or, mismanage the limited attention heads resources - per token)
There are no "attention heads" or fixed number of things a model can pay attention to ... or at least not exactly.
After every prompt the model decides "I have a weight of 1 to distribute between every token in my context". If you have ten tokens, each gets a weight of 0.1 ...
... except it's not that simple, because the LLMs don't distribute that "attention budget" equally. If your prompt was "where is Paris", then any tokens in context it can associate with Paris will get a greater share. If the word Paris is in your context, it might get 0.3 or 0.4 weight, and close by tokens might get 0.2, while other (unrelated) tokens get 0.03 or something.
Now, add lots of context, and you start to see the problem: more context = greater distribution of the attention budget. Even if the prompt is about Paris, and the Paris tokens get higher weights, they are only getting (say) 0.002 ... while unrelated tokens are getting 0.001.
All LLM "answers" are just math, computation, based on the context and those weights. If it can't "focus it's attention" because it's distributed among too much context, it's far more likely to miss the relevant tokens (eg. the Paris ones) and give you an answer that ignores key parts of context.
But again, there's no fixed number of things it can pay attention to: it's a gradual degradation of the chance of it seeing key info it should, based on the amount of context.
> There are no "attention heads"
Yes there quite literally is internally in an LLM.
https://bactra.org/notebooks/nn-attention-and-transformers.h...
Just because something is called an "attention head" doesn't mean the terminology makes sense.
If I'm reading the article right, hungryhobbit is accurately describing a single attention head.
And... what having multiple attention heads means is that you do the "single attention head" thing several times, and average the results. There is no part of hungryhobbit's comment that changes if you make this adjustment.
In particular, he is clearly responding to this clause in the comment above:
>> the number of attention heads isn't mentioned. Models have a limited set of them and obviously, a model can focus at N max things at a time
Those are the "attention heads" that hungryhobbit is saying don't exist. You'll notice that this definition of "attention head" does not correspond to any part of an LLM; I fail to see the problem with saying that "attention heads", as defined by this clause, don't exist.
There quite literally are internals that are called attention heads in modern LLMs. Saying 'there are no attention heads' is objectively false, which definition you follow is irrelevant.
Imagine the following conversation:
How was your trip to Los Angeles?
There were no movie stars!
Is the second person's statement "objectively false"?
Or is it more likely that you, Catloafdev, enjoy intentionally misinterpreting simple statements?
You might notice that in the remainder of the sentence you quoted, hungryhobbit takes the trouble to define the attention heads that don't exist.
Llm's have come on quite a bit beyond the fundamentals nowadays. Attention heads really are a thing.
One of my early experiments last year with open source models and context size was with GPT-OSS 20B (the mxfp4, the "smart" 4-bit quantization). Even though it boasted a 128k context size it was bad at recall around 32k characters (didn't bother to implement the tokenizer for counting).
The recall text was a simple hash generated, filler text from a dictionary file and a request at the end to return only the hash from the beginning of the prompt. Past 32k characters the response contained hallucinations of characters or full hashes.
Just having large context size doesn't paint a full picture of capabilities, prompt adherence and other quality metrics.
This is conspiratorial speculation though. They did test many open weights models which spanned the full spectrum of performance: Nemotron 3 Ultra second-worst, GLM 5.2 top five https://arxiv.org/html/2607.25398v1
>Want it to go away, almost like magic? Local inference.
Ah yes, magic that costs the same as a new car.
Not necessarily!
If you have a semi-recent MacBook with even 32GB, you can run 20GB models that are pretty damn smart, with room to spare for the rest of your toolchain.
If you’re reasonably connected to the code you’re writing and prompting the AI at the level of the code, not the level of the feature, you can get some fantastic results.
Sure, it’s not the completely automated dreamland that’s been sold, but it’s still a speed-up on par from going from assembler to a higher-level language, which is still immense.
And for effectively free, if you have a machine that would otherwise have been considered “developer-grade” for a lot of tasks anyway.
What models have you found to work for which tasks? I find the local models very useful but not in a way that's replaced cloud models (yet, I remain hopeful).
A good setup will cost you the same as a decent house in a major metropolitan area. And then the price of a used car to upgrade your electrical to handle that kind of load, and get the cooling setup you need.
I've noticed this trend of the sampler knobs being removed. Can you explain why this might be the case?
Would you mind sharing some of your setup? Which model and which params do you tweak (e.g. temperature)?
Long-context models do work as advertised... just not when combined with multi-turn conversation sessions.
It's my understanding that, for all current models having long-context capability, the "parts" of the model that allow them to do long-context processing, are mostly-untouched descendants of academic model architectures; where these academic models were designed around a very constrained use-case assumption: that anyone using more than a "normal" amount of context would specifically be trying to apply a reasonably-sized prompt to an unreasonably-sized block of plain-old embedded data, as a single-shot conversation. (Consider a task like e.g. "summarize this entire book." Context = [prompt] + [text of book].) This is the use-case all these architectures were refined and benchmarked around. And even today, this is still basically the only way these models' long-context processing capabilities work.
---
But I think it goes deeper than that. It's not just about how we've made long-context work so far. It's about what "attending to context" really means, and what that implies about what we'll ever expect a long-context model to be able to do.
I would describe long-context models as seeming to have two distinct types of "attention state" used when attending to the layer-1 input-vector / context window:
- They have some small amount (e.g. 8-32k) of regular "high-quality" attention state. This is likely inherited from the base model they are trained on top of.
- They also have a large amount of "low-quality" attention state. This is what the long-context training process has created.
The "high-quality attention state" isn't limited to referencing the first 8-32k of the context window, to be clear. The high-quality attention mechanism is just limited to "choosing" 8-32k of arbitrary data samples, each from its own arbitrary position in the context window, to form a bounded-sized vector upon which high-quality attention is then computed.
(You could think of this as the high-quality attention mechanism computing attention over the entire context, and then doing top-K sampling, i.e. zeroing out all but the K largest-valued elements of the resulting latent vector, where K=8-32k. That's not what it's actually doing—that'd require paying the very quadratic scaling cost that prevents us from just increasing the short-context window size—but it's a useful naive model for what it's doing.)
---
It's harder to directly explain what the low-quality attention mechanism does, so let me try an analogy.
(While no one can yet give a definitive accounting of the abstract computational processes the matrix-multiply-ops against a given LLM's weights encode, we can still speak of possibility-spaces, or analogies via information/computability theory on what sorts of abstract operations the LLM "must" be performing to achieve the results it achieves. This is that kind of thing.)
Think of an LLM's evolved context-window attendance mechanisms as being somewhat akin to the LLM being a regular CPU program, that has access to a 1. block of raw addressable memory (i.e. the context itself), and 2. a block of pointers into that memory (i.e. the learned first-layer attention over the context.)
Our LLM program's logic, through mutations during training, gradually evolves into a form where it could be seen as implicitly declaring and using a growing set of variables: some bound to static addresses in the block of raw memory (i.e. static context-window positions), and others bound to static addresses in the block of pointers, where the raw-memory-address-value of the pointer (and thus what data from the context-window it will read as when "dereferenced") is the result of a dynamic calculation over one or more other variables (i.e. an arbitrary weighting of the layer-1 input vector, where the resulting vector element is used as a selector, in layer 2, among other data that got sampled into "passthrough" vector-elements back in layer 1.)
Under this analogy, a model's "high-quality attention state" is the set of "named" pointer variables that it directly references in its logic.
As should be obvious, there can only be so many of these; you really fundamentally can't scale them up, without literally having "more logic" that defines and makes use of them (i.e. scaling up the size-in-weights of the model's layers.)
A model's "low-quality attention state", on the other hand, is the block of pointer memory itself, which can be as large as you like. Presuming the computational substrate permits the logic to compute on the addresses of memory within that pointer-memory block and then double-dereference them to get context data (and it does seem to!), your LLM's evolved logic can do "dynamic random-access reads" of arbitrary parts of the long context.
But, of course, the model won't be doing anything special with that data. It can attend to it, but not in a way that's unique to that data. It pulled in such data through generic logic that doesn't "know what it's looking at", after all.
---
IMHO, this analogy helps to make clear that while giving a model a raw "large-context capability" might help said model to attend to more data, it doesn't really help in making a model able to attend to a larger prompt.
Just by information theory, an instruction-following capability must imply that something analogous to logic keyed on those instructions gets embedded into the weights; and that logic, in order to activate and work, presumably requires its own state — i.e. requires that individual values from the context are getting attended into additional top-level named variables.
And while the short-context attendance mechanism does that, the long-context attendance mechanism inherently does not / cannot.
And this means that there is likely a practical upper limit (in terms of practical model-weight size + latent-vector size + etc) to "how much prompt" a model can follow during a single inference step. Regardless of how much data the model can "see" on a given inference step, its active "program logic" is only defined in terms of so many concurrent stateful computations.
---
Thus: trying to use a current long-context model's "low quality" attention to attend to a large prompt is just going to result in nonsense. None of these model families have ever been trained to follow a million sentences of rules simultaneously when answering a question. They're just going to attend with their inherited bounded "high quality" attention to your prompt as best as they're able, while forgetting literally everything about the prompt that doesn't fit in that bounded "high quality" window.
And anything that allows current long-context models to appear to interact well in multi-turn conversations with memory + KV caching + etc layers in operation — while also making use of its "low quality" attention to attend to large data — is some kind of hack, and will break like a hack.
Yeah checks out with my anecdotal experience with Claude. It is pretty great at following instructions - for about 10 minutes, after which it seems to ignore things I told it before.
I have quite explicit and strong instructions (e.g. don't write massive comments, use existing functionality, etc.) in CLAUDE.md files which seem to get bypassed surprisingly quickly when doing real tasks. Yet if I tell it these things in a prompt during the task, it performs way better.
Result is I'm trying to resist adding more and more things to CLAUDE.md files which in some scenarios it does well but in other scenarios totally ignores and messes up.
This is not what the article is talking about. Its talking about policy documents not it forgetting something 5 prompts ago. In fact you adding things to CLAUDE.md is more what its talking about.
Conceptually the same thing though. Claude.md is sourced at the beginning of the session, so will be pretty far back in context, just like user prompts from the beginning of the session.
Claude is a next-token predictor, appending to a long text document. Prompts aren't an independent kind of thing from policy documents. It's all text in the backscroll.
I’ve had a lot of success using the root Claude.md for a handful of high level application wide rules and directions (I keep it pretty small), module specific claude.md in subfolders alongside the code with more specific rules and direction, and a custom rules backed /code-review skill that enforces it all and catches anything that was missed during implementation.
This is exactly what I have and it doesn't work well
I believe the correct static instructions are about getting it at the right starting point for whatever class of projects you're working on; not as a continued referencable or "HOWTO" of what it's doing. They're all just "grooming" the LLM for future instructions.
The coding harness is what's getting it to continually align to your current instructions.
This is very obvious with local models.
Ok so what is the correct way to tell it "I don't care what is happening, you must uphold these rules at all times"? If it's not any configuration of .md files?
> you must uphold these rules at all times"?
You need to let go of the idea that this is something LLM's can do. They can't. At best, they can bias towards rules conformance with more or less likelihood, but coverage and conformance both go down super-linearly as you accumulate more rules, more context, and more output in a session. That's simply the nature of how these tools work and you need to engineer your workflows around it if you want to use them.
If you absolutely need some rules enforced, you need to adopt some framework for validating those rules that then rejects, reprocesses, or repeats any session that fails to satisfy them. In the best case scenario, this is some traditional deterministic validator (like a linter, compiler, analyzer, exhaustive test suite, etc in coding) but if you need to process in stochastic space because its something rich and ambiguous like natural language itself, then you want to dispatch a swarm very narrow, task-focused subagents that each validate against a very constrained subset. (And prepare yourself to have those to fail sometimes too. LLM's are noisy and cannot deliver strict rule enforcement on their own.)
You need to make the rule concrete somehow. I call it a "control". So for example, instead of instructing it "always run tests before committing", you (or you have it) make a git commit hook that always runs the tests first and that refuses the commit if they don't pass.
In this case, it is an advisory control only, because the LLM can also unhook that hook. And of course, it could also just disable the failing test(s) with some bullshit reason. But it is far better than assuming it will comply every time.
Then there is the "hard control", which is the inviolable that the LLM cannot bypass.
You need to move as much as is technically possible to either hard or (failing that) advisory controls.
>In this case, it is an advisory control only, because the LLM can also unhook that hook
you can in principle make a hook in the harness itself that will run an auditor prompt that checks it adhered to the policy, correct the model and also make it known in advance.
This appears to be what the claude code-review is doing:
https://github.com/anthropics/claude-code/blob/7ef6eec9d9ba8...
I've seen pretty much every model including Claude bypass commit hooks constantly if there's even a tiny bit of friction, instead of pausing and asking me to help fix why it can't run the hooks (usually something that needs to be done outside the sandbox like `npm ci`).
That's why it's critical to have these checks run in a context that the model can't bypass, such as via github actions that block PR merges. Generally works pretty well for stuff at work (where we have all this stuff set up as part of continuous integration checks), but it means that for personal stuff I have to set up quite a bit more infrastructure to ensure that models don't just skip running tests (which they LOVE to do).
So you are essentially saying "you can't, you can only safeguard from effects of LLM eventually ignoring it"
I think things like CI, linters, hooks, etc started as defense against humans eventually ignoring it (low blood sugar, under pressure, never cared in the first place, etc).
These seem like they would naturally extend to LLMs as well.
(I'm not pro-LLM on balance, but the solutions of today feel similar for humans and LLMs.)
Ask it to come back with a filled checklist and hand it over to a different agent with a fresh context window (three lines model). Or make it collapse the context and get back to the checklist.
Subagents whose only job is to review the actions of your other agents for rule compliance? It works reasonably well for me in complex workflows using Claude Code.
Can I ask how you set this up? Like is there some way to have that run automatically, similar to “auto mode” for approvals, or do you have to invoke it regularly?
Sure! I use an orchestrator main agent whose only job is to run subagents through the workflow process I've defined. Part of that workflow is to invoke a review subagent at particular points to check the spec, the implementation plan, and the code for rule conformance.
The review subagent has its own definition and gets invoked with a specific target, so the context is very focused on just rule enforcement and I don't have problems with it skipping rules.
The whole workflow is packaged up as a plugin, but you don't need that to get this approach to work. It should be sufficient to have the rules you want enforced written out somewhere, and to either kick off a focused review agent manually referencing them, or do something like I did and have it be a defined part of your workflow (depending on the automation level you want).
Hooks..
CI runs. Local Git hooks. Cursor also has hooks built into their agent. Other agent APIs probably have something similar.
in the plugins I'm using, it's basically _always_ adding information to the context. You're not going to do it manually; you can't go back in the context and add it because that'll break the cache. The way your programming harness works is by constantly reminding the LLM of the tools available.
A good programming harness is basically a stack. A good stack keeps building each layer. You _cannot_ pull things off the bottom of the stack because that's an expensive cache hit; but you can pull things off the top. So what your harness should be doing: <tools> <rules> <user content> onto each request _then_, when you get to the next request or result, pulling that out if there's some change.
So you can see it's the cache that's either exponentially growing or having to cache bust to keep it fresh.
As a hobbyist, I find it difficult to figure out how to make Claude stick with some repeating things I want it to do after every major action, like re-evaluate the completeness of tests, update the documentation, etc. And CLAUDE.md/AGENTS.md definitely did NOT help there, sadly.
Hooks can be pretty useful for that. A hook when it is finished ”run tests suite and check coverage” ”check if your changes require updating the docs”
Don't use the default harness, write your own instead.
Any way to do that AND use subscription instead of per-token pricing from SOTA providers?
If the harness runs on your localhost, but the inference doesn't, it usually means it's calling some API. Whether you want to break TOS of your provider like that or simply buy tokens from our friends from UTC+8 timezone is an open question.
This is the way. Making your own agent to have a sticky memory context that is prepended to every execution is necessary to ensure each task is bounded by those precepts.
The trick I'm doing -- the model is given a tool that runs a prompt in the current thread to consolidate it's working memory and identity (it has a memory tool bound to the agent persona). When the prompt ends, the parts of memory that are marked as identity are merged together into a new system prompt, then the context restarts with only system prompt and this tool call surviving. Then it just keeps going.
Doesn’t that blow your token cache hit rate and balloon your costs (essentially everything is billed at the input token rate, not the cached rate)?
If it moved a block like
<system_prompt>
<tools_etc>
<project_prompts>
<user_call>
<llm_response>
=>
<system_prompt>
<tools_etc>
//Removed project_prompts
<user_call>
<llm_response>
<user_call>
<project_prompts> //reinserted
<llm_response>
Cache would break but if you did instead
<system_prompt>
<tools_etc>
<project_prompts>
<user_call>
<llm_response>
<user_call>
<project_prompts> //Duplicated with new user messsage
<llm_response>
It wouldn't bust cache, it would just make your input prompts slightly larger. Technically inefficient as you're duplicating the same rules over and over but I imagine for a smallish checklist/principles that it is tremendously more efficient than a cache break every message
so first, we treat context like a stack. Things are popped onto the stack; if we try to remove anything but the top of the stack, we'll break the cache.
So a good harness should be popping tool prompts, user hints after every return from the top so the context of tools and user hints never repeats and are allways prefixed to the current user prompt.
I've never inspected any harness, but opencode with the dynamic context compaction plugin appears to do this well.
What I'm currently doing in a large refactor, is I created a super-run script; the super run script is devided into super-dev (Setup dev), super-test (run all tests), super-build (build artificats), super-e2e (test all artifacts), super-deploy (deploy finished).
Each super's sub functions should _fail hard_, and each script should be highly detailed; of course I'm not doing it myself, but in small increments of directed work, it can build up the necessary harness.
What I get is a CI that just starts with "run super-run.sh" and that gives it context, then each sub script provides context depending on if it succeeds or fails. If it fails, the agent is provided what it needs.
It's basically, you have to design the products of the AI to give itself the context. Another technique I'm testing out is a parallel set of files like <subject-module>.js, <subject-module>.test.js, <subject-module>.md which get pulled up if the Agent is looking for a file.
Any model with a good score on this benchmark would have a good claim on superhuman abilities. Humans are pretty terrible at being thrown a long policy document and being expected to follow it
And while we shouldn't anthropomorphize these models too much, I wouldn't be surprised if many of the core reasons for failures are similar. Working memory is a limited resource; you can only focus on so many things at once; reasoning depth is limited; and many real-world policies are not actually meant to be implemented in the same way they are written and have insufficient specification of edge cases
With humans, we usually do the equivalent of RLHF, both via "training" with simulated cases, and via feedback while on the job. You would never hand a newbie a 124 page policy document and expect them to correctly apply it on the first task, or to do it reliably in the first month
The challenge with comparing these things to humans, is that humans learn. A newbie might not respect your organization’s set of policies on day one, but what about 3 months in? Or 3 years? Meanwhile there’s still no reasonable mechanism for automatically fine tuning LLMs or adjusting their harnesses to make them better at completing your organization’s objectives more successfully. They’re still overwhelmingly governed by the shared weights and harness policies found to be successful for the average case.
Models learn. It just costs $10B and 1 year to do what a human does every night.
LoRAs or even full fine-tunes would be much cheaper than that, and with some investment in the right infra could be updated regularly. And at least LoRAs can be swapped in and out cheaply, making them usable in large-scale inference providers. But there seems to be limited appetite in offering this. Both Anthropic and OpenAI no longer offer fine tuning for current models
does lora do a good job at teaching the model new things that werent in the training data?
without trillions of examples of following instructions at a million context length, im not convinced the behaviour is in the weights to begin with
isn't it because there are too many contradictions and ambiguity? the reason it works for humans is because we don't apply everything at once either.
No, it's more than that. We can't remember everything. I hired, say, two years ago; as part of my onboarding process I had to read a bunch of policy and procedure documents, which were full of stuff that I didn't understand because I wasn't really in the context yet. So at the time, to me, those documents were full of arbitrary text that I didn't really understand. Some of it was rules that I had to follow, but at the time I didn't understand why, so it's just arbitrary rules.
How many arbitrary rules can you memorize? Do you even remember them two years later? If you do, then we can get to your statement.
And your statement is true. Humans do not run every action through a memorized list of rules, to see if any of them block the action. We don't. We're not going to, either, no matter how badly the policy manual writers want us to.
That's a great comparison, human vs ai on a wall of text.
The problem is that it doesn't fit the sales pitch of LLMs and agents - humanlike or better, repeatably, 24/7, for a fraction of the price, you just need to make sure that you give it all the rules.
Unfortunately we can't really have a meaningful conversation until the money vampires have left so we will need to reschedule this until after the bubble.
This is real. I was very angry that AI kept breaking the rules I wrote, so I asked Claude to crawl its own history logs. And then I found: every time after it breaks a rule, the chance of breaking rules goes up.
I feel this is like few-shot in reverse. Few-shot is supposed to be good examples AI should follow. But when it breaks a rule, we correct it, it keeps breaking, and this whole thing actually raises the probability of more violations.
I wanted to know if there is any difference between writing rules in the prompt, writing them in CLAUDE.md, or not writing at all. So I did some short tests before. I asked Claude to open brand new sessions, test different topics with the rules I want to apply. The result turns out to be: in a fresh session, no matter the rule is in a prompt or in CLAUDE.md, models (Opus 4.8, 5 or Fable) all follow it fine, across models. Even Opus 4.8, the one always violates rules in our conversations, does it well.
I suspect it's the long context that breaks rules. But simulating a long conversation experiment is kind of hard, I still haven't found a good way to test it. So seeing this paper now, it completely answers the question I was stuck on these few days.
Besides, something caught my eyes in this paper: sometimes the model does run the check by the rules, and it really finds the violation, but its narrative still insists on its original wrong output.
My current approach is same as everyone here: use a separate hook or post-check to fix things. Because if you let the model fix it during generation, its narrative or main generation part sometimes just rejects the rule error it found.
Evoking Gödel via GPT, with the requisite em-dash: "No sufficiently expressive formal governor can be simultaneously consistent, complete, and able to prove every relevant proposition—including all propositions about itself."
But "Read AGENTS.md, including the middle" sure helps.
Most people don't understand that 'agentic AI' is a completely synthetic, force fed capability by extensive Reinforcement Learning on synthetic domain specific 'agentic' datasets on post training.
If the LLM wasn't post-trained to adhere to specific handbook, it just won't work. If the LLM wasn't trained on an use case the lab decided was worth making a synthetic agentic dataset, it won't work as well as you want.
There's a reason the main agentic task LLMs excel at are coding tasks, it's the way of working of the creators, and they understand intimately the flow and can train for it.
I believe the true way will be able to easily fine tune models on your agentic use cases, but it would require a big company to compile a huge dataset on it's way of working and I don't think anyone wants to be the first.
In terms of long context, accurate attention retrieval from early tokens is just impossible, given the expansion of RoPE encoding for the positions, or in case of Kimi that don't use it anymore, as well as deepseek, early context is heavily compressed you lose accurate information.
If people spent more time studying about AI and how it works, they would realize that the default should be to one shot prompt your task with a big, cached system prmopt, with an user prompt that is just dynamic data, specified to the cheapest model that can do the job.
Unless you really can't do this given your problem, you should try to make a graph of well defined, step by step oneshot prompts, and THEN if your problem still can't be solved with that, then you start leveraging agents.
Despite this giving better results, and being more cost efficient, is evidently too much work then just letting the AI do all the work.
What do you mean by a graph of one shot prompts?
Most people jump straight to agents when what they actually need is a graph. Example: a mining company receives free-text reports from field geologists. You could have:
Geologist report -> LLM call extracts minerals we are looking for (you inject a db query result on the user prompt), locations, assay mentions and risks into structured fields -> LLM call classifies evidence into positive indicators, negative indicators and unknowns -> LLM call estimates deposit potential and confidence -> database lookups inject regional ore demand, nearby deposits, infrastructure and historical yield data -> LLM call combines geological evidence with business context -> LLM call generates an investment recommendation and rationale.
That's what I mean by graph. Every step is a separate LLM call with a well-defined responsibility, consuming the output of the previous node. Each node can be tested, benchmarked, retrained, replaced, or monitored independently. Why would you use an agent here? You can cache every single system prompt on each call making your total token output much cheaper than having a full 'output' only token generation workflow which is what happens with agents.
There is nothing to discover. The workflow is already known. The company already knows how geologists evaluate prospects. The company already knows what data sources matter. The company already knows what the final output should look like. You don't want the model deciding which tools to call, which reasoning path to take, or which pieces of information are important every single run. You want the exact same process applied to every report so results are consistent, measurable, auditable and debuggable. My default is: One-shot prompt -> if not enough -> graph of LLM calls -> if not enough -> agent. A surprising amount of enterprise AI is really just: Unstructured input -> extraction -> classification -> enrichment from databases -> decision support. Not: Unstructured input -> autonomous agent spends 20 steps deciding what to do next.
Agents make sense when the workflow itself is unknown.
If the workflow is already understood, a graph is usually cheaper, more reliable, easier to evaluate, easier to debug, and less dependent on whatever synthetic "agentic" behaviors happened to get reinforced during post-training. I am sure people default to agents mostly because it's less engineering work than explicitly modeling the process.
Thanks, that helped. I get it. Yea, that’s the harness executing the workflow with the LLM being called at the right time. That ensures the process is consistent no matter what, in contrast to the agent calling out to tools and possibly doing different things every time. Basically, standard code being in control rather than the LLM being in control. Fully agree with this model. We should use deterministic code when we want the same process or algorithm every time and choose LLM callouts when we want “fuzzy” processing that is not as deterministic.
I jump straight to graphs (unless it truly cannot be solved with a graph), but then the stakeholders get upset that I didn't use agents. Doesn't fit their marketing plan, I suppose. I don't mind dealing with that, but I can see why most jump to agents in order to avoid the human conflict.
> In terms of long context, accurate attention retrieval from early tokens is just impossible, given the expansion of RoPE encoding for the positions, or in case of Kimi that don't use it anymore, as well as deepseek, early context is heavily compressed you lose accurate information.
"Every gambler knows the secret to survivin' is knowing what to throw away, and knowing what to keep." - Kenny Rogers
Humans have limited context, just like AI. The difference is that humans - at least some of the time - can figure out which pieces are more likely to be important, and therefore prioritize keeping those in the context.
can downvoters explain? this has been by experience with these tools too.
i thought it was well known that claude code got good at coding because anthropic bought tons of coding data from companies like mercor.
I don't understand why a model has to follow the instructions. Don't get surprised when it shows its true color. Plus, do users (not the researchers) really check whether the response follows the instructions?
Long policy documents are also a problem for human agents. Without special training no one will retain 180 pages HR employee handbook, fire codes, OSHA safety rules, FCC regulations, the US legal code.
If the stakes are high, e.g, proceeding in ignorance could lead to prison time, people will favor inaction, even if the policy technically permits a corner case. If the stakes are low, people will completely override policy for the path of least resistance.
So what is tre answer then?
I feel like “discretion” parties missing. Do we need some kind of special discretion model?
There was an article a few years ago called "Lost in the Middle: How Language Models Use Long Contexts" https://arxiv.org/abs/2307.03172
From my experience this holds true to this day. It was one of my core observations for similarity to the limitations of human working memory on "Engineering for Bounded Cognition"
Richard Hendricks solved this decisively with middle-out compression
What inspired him to take this novel approach?
I didn't get the reference, but it looks like im going to have to watch that series now :D
It holds up really well. I’m envious you get to watch it for the first tome, enjoy!
This article to me also implies that there are some potential issues with large Spec based development flows, which I haven't been able to pin down lately.
Specifically, having agent implementation drift from the Spec.
Drift is huge between any large spec and agent implementations.
I've done a ton of testing and the model doesn't matter, fable or sol still miss a ton of detail and drift.
I'm building http://engine.build which closes the gap and makes sure the implementation matches the spec.
It's not the same as the satisfaction you get when solving complex problems with code yourself but writing clear specs and thinking through the problem is still very satisfying to me.
For Claude, I used a UserPromptSubmit hook running inject_rules.py which reads RULES.md from the disk and prepends the whole thing to every prompt. That helps the rules to stop fading as context fills because it is reinforced every prompt.
Sure, it uses tokens slightly faster in the prompt, but I find it reduces overall token use, you can use it with pro, but of course, nothing works 100% of the time, but it's better. Emptying the memory helps too to avoid Claude making up stuff that messes with how I want it to act.
The general gist of inject_rules.py is :
RULES_PATH points at RULES.md
reads it with encoding='utf-8-sig' so the BOM is stripped
wraps it in a JSON object — hookSpecificOutput.hookEventName = "UserPromptSubmit", additionalContext = a preamble plus the full rules text
the preamble is the line you see above the rules: rules are in force for this turn, run rule 33's five tests before raising anything unasked
prints that JSON to stdout, which is how Claude Code takes it in
on OSError it returns 0 silently — if RULES.md is missing or unreadable, nothing is injected and the turn proceeds with no rules
why this vs having some response hook check the next outputs against the bunch of rules, and injects only when it goes off track?
Control vectors for the win? It feels like the way to fix this is to pull out a control vector immediately after processing the handbook and use that to steer later inference. That should stop the drift over long distances, but it doesn't guard against the handbook itself being too big.
I noticed this behaviour a few months back, I think I was using Sonnet 4.6 at the time... I have strict rules about comments in the codebase, this all for personal projects, and the reason I restrict comments is to keep the token count low.
At some point between the model i was using and the previous version of it, Claude started inserting massive comments with references to tickets and other tasks. All this while having specific directives on the CLAUDE.md
Since then I resorted to developing my crapware as if I was the floor manager of a vehicle assembly line, and I have a few highly-specialized sub-agents running errands around the main session, but only ever taking care of a single concern. The main session builds with the knowledge contained in things like CLAUDE.md but the sub agents make sure things like the no/low-comments directives are either enforced, or factored into the final product.
LLMs are not people. Employee handbooks, like all other social rules, are not self-executing; they depend on the fact that human beings are innately social animals.
simonw has been correctly asserting that there is no deterministic way to prevent hallucinations. I’d argue that this obstinance could be the seed of a strong argument in favor of their capacity to eventually coalesce some form of consciousness
I guess that means my code is conscious cuz it often doesn’t do what I want either :)
I absolutely believe it.
Codex has been pushing things to my main branch all week despite me repeatedly telling it not to and adding to my AGENTS.md very clear instructions for creating feature branches and putting up a PR. It keeps doing it in spite of all that.
I'm probably going to need to enable branch protection on my personal projects... What a pain.
Policy documents do not govern agents at all. Conformance is distributed and completely unreliable.
How many times have you told an agent not to do something then had to correct it?
You must always flip the frame. Objective analysis is way better with llms than steering via skills.
This is just a small example of why "loops" became popular for a minute and now it is "graphs"
I'd be very interested to see a comparison with actual humans on the same benchmark.
What really helped me was to run any .md/prompts through an LLM to find contradictions, duplicates or ambiguities, repeatedly. That led to agents much better following instructions.
Attention vs. Consistency
When "performance" breaks down over long lengths, one could attribute it to a lapse in attention, but one could equally suspect inconsistent instructions.
The fewer instructions and conditions that need to be simultaneously met the easier it is to comply, but with more and more instructions one is bound to introduce internal inconsistencies within the instructions.
For my own (rather idiosyncratic) harness I've been experimenting [0] with "compiling" long markdown specifications into small executable logic programs. It's too early to tell for sure, but I believe that this approach does have its merits when you want some guarantees about how your agents behave for longer tasks.
[0] https://github.com/deepclause/deepclause-sdk
Interesting idea. I’ve been noodling about something similar myself for a few months, but I haven’t moved forward with testing it. What sort of outcomes are you seeing with it? IMO, we’re never going to get to AGI without fusing “soft” AI decision making with “hard” logic and symbolic algorithmic reasoning. Humans don’t realize this most of the time, but we routinely use them all.
Opus 4.8 (max thinking) scored highest and Grok 4.3 lowest
It's hard to understand what's going on with Grok. It's like it has capabilities in a theoretical sense but maybe the training is so focused on being in x.com/grok.com with the web search tool enabled for "is this true?11" type queries that with any API type usage with document workflow instructions, tool use, code gen etc it completely falls over
After they acquired Cursor, Grok 4.5 seems like a completely new model, performing at Opus 4.6 level, I'd say. But much cheaper and faster.
maybe it is Grokimi?
https://venturebeat.com/technology/cursors-composer-2-was-se...
Dealing with agents/LLMs based on "instructions & guidelines" has taught me - nothing (un)reliably governs agents other than agents themselves or (rather i.e.) their motherships (assuming they can and they intend to). Or if you add ton of local tooling.
Why would anyone think that models optimized for efficient context management, giving much more weight to a short sliding window, would attend to distant, heavily diluted tokens?
Plus the model's capacity to take more context into account and actually integrate it to the output is simply limited by the number of activated parameters. If you give it a playbook, you are forcing to choose it between attending to the playbook and the task at hand.
If you want to force it to work step-by-step, you need to present the steps one-by-one. Ideally with rules for the current step at hand and maybe relevant input again, depending on overall task size.
Why did you think models love to re-read files before editing them? It increases recall quality and thus edit precision and thus benchmarks.
> limited by the number of activated parameters
not sure I got it?
Separately, the frontier labs are kinda pushing us into that behaviour by releasing models with ever-larger context windows.
To run at decent speed, all models try hard to use only most likely relevant part of the context and most likely relevant weights (MoE) to predict the next token. Doing the math in full is unfeasible.
This is a problem with soft rules and LLM in general, and it makes sense that it gets worse on complicated tasks with long context.
Glad to be able to put some numbers on it.
In my experience, the more structure you enforce on models, the worse they perform and the less they actually do what you want.
yep all the advice about context engineering, harness whatever is so silly. ai doesnt give a flying fuck about some IMPORTANT instruction in your claude.md.
It does what its has been trained to do. So find out what its trained to do and just use it to do that. This is not general intelligence.
I got pumped seeing the OKF format from Google (which is just a standardization of wiki pattern) but quickly realized it could not yet combine many subtle concepts together in an efficient way.
Hard to take this seriously when it has all the hallmarks and annoying tics indicating it has been written by Claude.
Well no shit obviously. Just cause you tell the random text generator to follow some rules doesn't mean it will.
This is what spec driven development tries to solve.
Multiple rounds of generating small contacts documents that grow from the original idea , trying to keep each slice small enough to process for a human to approve/disprove .
Eventually it leads to a long list of tasks grouped by functionality. You start a new context and the orchestrator agent dispatches tasks to sub agents with a limited amount of information provided to each sub agent.
Also should have adversarial review and approval gates with other agents and roles.
Having only dipped my toes in generative ai recently I'm surprised how small even a 1mio window is. A semi serious project can take several session in one sitting. Especially as degradation sets in waay before the window is full.
Few pointers:
Dont try and handle the entire project in context.
Use a well structured filesystem layout for your code with a few lines in an AGENTS.md describing the layout and core architectural requirements (no more than that, as per the article!).
Then work on small-medium tasks at a time with a fresh context.
At the end of your task, ask the agent if there are any key points about the project layout or architecture it would want to add to memory - audit those manually and amend your AGENTS.md accordingly.
If your code is well structured and you keep your tasks localised, you can get away with seemingly minuscule context windows.
It's also worth noting that high effort models love to slurp up whatever context they can. You almost never need/want high effort for non cross-cutting tasks.
Good points, but it still seems to me that the technology is far from mature.
HANDBOOK.md is a benchmark for long-context agentic instruction following, modeled on how enterprise employees follow company handbooks in their day-to-day work. Each task is a unique RL environment with internal tools and external MCP servers, spanning five enterprise domains: Finance, Medical Billing, Insurance, Logistics, and HR.
The prompts reflect the actual jobs enterprise workers perform every day. Each task drops an AI agent into a live company environment, requiring them to cross-reference an extensive, multi-section handbook against a cluttered inbox, a multi-channel Slack workspace, Jira queues, and a stack of files (spreadsheets, PDFs), and working out both what to do and what the handbook forbids.
https://github.com/surge-ai/handbook/tree/main
Please don't paste walls of text into the comment field without quotation marks. It wastes all of our time.