Seems like this would break the cache often. That would increase billing rates with certain providers and, for local models, take a while to generate responses, especially with long-running agentic sessions. Interesting idea though.
No? from a quick skim it doesnt look like it goes into the system prompt everytime, you just use search/grep over it. Pretty much most memory approaches relying on a big set of info where agent chooses what to 'recall'. Its like any other tool
If there are no summaries then when context is full messages need to get evicted. If doing so one by one then it would indeed destroy the cache. Of course... maybe the implementation evicted 50% of messages at once, I didnt verify in code
Is the date really useful? Usually I want the AI to remember rules, like "always do X before committing Y" and a timestamp doesn't change anything there.
When I look at Claude's memory markdown it tends to list dates for significant requests already, so it can untangle this sort of thing. And if it can't it will just ask directly.
If you can make an agent reliably follow instructions, you've solved alignment completely. So I'd be particularly surprised if you can square that circle as a hobby project!
I can't resist the opportunity to paste some of the text I wrote in the Anthropic AI interview last December on an idea I had about adding temporal awareness to Claude.
Anthropic: Let's focus on AI with capabilities that feel within reach—not necessarily limited to exactly what exists today, but the kind of AI tools you can reasonably imagine existing in the near future. What would you want AI to help you with?
Me: There is something fundamental missing in today's capabilities that is well within reach of the tech today. I thought I would make a Github repo with a prototype but too much in my life right now. If a human in Anthropic reads this, please let me know if you run with this idea: The missing thing is lack of the bot's initiative. It is always the human who takes initiative and writes something, then the bot makes a (perhaps) elaborate response and shuts up. So something seemingly very simple like telling the bot: "Write me a limerick about a swedish chef, wait 50 seconds, then write a recipe for a danish christmas dinner not involving duck". The bot will not be able to do this. It has no concept of time. No ability to take initiative. There are many ways to frame this but you get the point. I am confident there must be a way to implement this. I read some research on this but all articles I could find are (in my opinion) unnecessarily complicated. The current chat style is Query, Response. The bot waits forever for the next input. This is not natural. When you (A) chat with customer support (B) (a real human), the context is such that it is natural for both A and B to give many inputs in a row before the other party gives their input. And, crucially, if either A or B takes too long to give input, the other party will react to that. This applies to most conversations between humans. So a way to get this behavior from a bot is as follows: the moment either the human or the bot gives any input, the chat system tries to understand the current context, and immediately decides how long time it is natural to wait until the bot should say something by its own initiative, and what it is that it should say. Then the system has a simple timer in the background that waits this amount of time and then says the thing. Very simple. Would it work? I have no idea, but it is worth a shot. This should probably be a separate mode, Proactive Messaging. it could be linked to an agent dynamically. So for example, i have a conversation in Proactive Messaging asking claude to follow the lowest price of something and let me now if it goes below a certain value. Then a few weeks down the line, i get a popup from claude about this if it happens. There should also be a menu in this case with what tasks like this have been defined.
> So for example, i have a conversation in Proactive Messaging asking claude to follow the lowest price of something and let me now if it goes below a certain value. Then a few weeks down the line, i get a popup from claude about this if it happens. There should also be a menu in this case with what tasks like this have been defined.
I've been looking for a new cooker. I have a Hermes agent running on VM that does this. I told it to keep an eye on prices. It created a cron job that triggers the scrapes and messages me if anything interesting happened since the last time it checked.
There is a KISS way to do this with our current models:
use a time string that updates:
every second (that it is not responding) erase a byte string at the end, then send a new byte string (that represents time from last response): bot either outputs a null terminating token, or if the number gets high enough, instruct/train it to take initiative then.
That only really gets you a limited sense, certainly better than nothing though.
The approach I would like to see would be more expensive token wise, but I don't think that can be avoided for fully interactive real time AI.
Add another dimension to position encoding to allow for multiple simultaneous streams. Run a model on each stream with an additional "I must speak" output value. A simple adjudicator model (maybe just softmax would work) decides which token(s) are emitted, each tagged by the channel that emitted them, which in-turn goes to the additional position encoding dimension. Then reinforcement learn them all at once with attention being allowed to look at all streams.
Then the model is always emitting tokens but while waiting it might just be emitting dum-dee-dum twiddle thumbs, on a channel that doesn't go out to the user.
It needs a way to focus attention though because you need a much larger potential context.
Is there any work on attention being limited to looking at a subset of embeddings defined by a parameterisable function because if a model could emit special tokens to change those parameters, it would potentially be able to scan it's own memory. Could be tricky to train but a training process that reduced the number of locations attended to over time might force it to compensate for the loss in scope by focusing.
Claude can definitely wait 50 seconds and then do a second thing. I see it all the time when running tools that never stop but iterate towards a goal. It can either run a bash `sleep 50` or use the ScheduleWakeup tool.
I’ve seen some good research where xx% of the time, all LLMs essentially just get lazy and won’t actually look things up.
Been working on an informational retrieval system similar, but endless Nerfs combined with non deterministic and finicky behaviors make it hard to reliably depend on.
It may seem useful at first, but eventually you'll hit a wall where this doesn't work, and you have to add another memory technique. Eventually you end up with a complex multi-layered system, because what people want by "memory" is actually 10 different things which all need their own solution.
Seems like this would break the cache often. That would increase billing rates with certain providers and, for local models, take a while to generate responses, especially with long-running agentic sessions. Interesting idea though.
No? from a quick skim it doesnt look like it goes into the system prompt everytime, you just use search/grep over it. Pretty much most memory approaches relying on a big set of info where agent chooses what to 'recall'. Its like any other tool
If there are no summaries then when context is full messages need to get evicted. If doing so one by one then it would indeed destroy the cache. Of course... maybe the implementation evicted 50% of messages at once, I didnt verify in code
Is the date really useful? Usually I want the AI to remember rules, like "always do X before committing Y" and a timestamp doesn't change anything there.
The specific design of this system uses the raw memories and allows rebuilding the operational memory from the raw memory (the LLL "Left Leg Layer").
To do this requires that there is an ordering of which memory came last.
Which of these is the current state? Without the date, it is not possible to rebuild the operational state of the rule from the raw records.When I look at Claude's memory markdown it tends to list dates for significant requests already, so it can untangle this sort of thing. And if it can't it will just ask directly.
The point is to build systems that don't require the agent to "ask".
The human becomes the bottleneck as systems become more agentic.
OP's system is first ingesting and storing the individual messages and then recompiling it into "working memory".
Oh cool. I'm working on exactly that here!
https://innerloop.works/breadcrumb
If you can make an agent reliably follow instructions, you've solved alignment completely. So I'd be particularly surprised if you can square that circle as a hobby project!
Its working pretty well to be honest! here’s some more background :)
https://innerloop.works/blog/introducing-breadcrumb
Am interested but don't want to sign up for discord to try something out.
I can't resist the opportunity to paste some of the text I wrote in the Anthropic AI interview last December on an idea I had about adding temporal awareness to Claude.
Anthropic: Let's focus on AI with capabilities that feel within reach—not necessarily limited to exactly what exists today, but the kind of AI tools you can reasonably imagine existing in the near future. What would you want AI to help you with?
Me: There is something fundamental missing in today's capabilities that is well within reach of the tech today. I thought I would make a Github repo with a prototype but too much in my life right now. If a human in Anthropic reads this, please let me know if you run with this idea: The missing thing is lack of the bot's initiative. It is always the human who takes initiative and writes something, then the bot makes a (perhaps) elaborate response and shuts up. So something seemingly very simple like telling the bot: "Write me a limerick about a swedish chef, wait 50 seconds, then write a recipe for a danish christmas dinner not involving duck". The bot will not be able to do this. It has no concept of time. No ability to take initiative. There are many ways to frame this but you get the point. I am confident there must be a way to implement this. I read some research on this but all articles I could find are (in my opinion) unnecessarily complicated. The current chat style is Query, Response. The bot waits forever for the next input. This is not natural. When you (A) chat with customer support (B) (a real human), the context is such that it is natural for both A and B to give many inputs in a row before the other party gives their input. And, crucially, if either A or B takes too long to give input, the other party will react to that. This applies to most conversations between humans. So a way to get this behavior from a bot is as follows: the moment either the human or the bot gives any input, the chat system tries to understand the current context, and immediately decides how long time it is natural to wait until the bot should say something by its own initiative, and what it is that it should say. Then the system has a simple timer in the background that waits this amount of time and then says the thing. Very simple. Would it work? I have no idea, but it is worth a shot. This should probably be a separate mode, Proactive Messaging. it could be linked to an agent dynamically. So for example, i have a conversation in Proactive Messaging asking claude to follow the lowest price of something and let me now if it goes below a certain value. Then a few weeks down the line, i get a popup from claude about this if it happens. There should also be a menu in this case with what tasks like this have been defined.
> So for example, i have a conversation in Proactive Messaging asking claude to follow the lowest price of something and let me now if it goes below a certain value. Then a few weeks down the line, i get a popup from claude about this if it happens. There should also be a menu in this case with what tasks like this have been defined.
I've been looking for a new cooker. I have a Hermes agent running on VM that does this. I told it to keep an eye on prices. It created a cron job that triggers the scrapes and messages me if anything interesting happened since the last time it checked.
There is a KISS way to do this with our current models:
use a time string that updates:
every second (that it is not responding) erase a byte string at the end, then send a new byte string (that represents time from last response): bot either outputs a null terminating token, or if the number gets high enough, instruct/train it to take initiative then.
That only really gets you a limited sense, certainly better than nothing though.
The approach I would like to see would be more expensive token wise, but I don't think that can be avoided for fully interactive real time AI.
Add another dimension to position encoding to allow for multiple simultaneous streams. Run a model on each stream with an additional "I must speak" output value. A simple adjudicator model (maybe just softmax would work) decides which token(s) are emitted, each tagged by the channel that emitted them, which in-turn goes to the additional position encoding dimension. Then reinforcement learn them all at once with attention being allowed to look at all streams.
Then the model is always emitting tokens but while waiting it might just be emitting dum-dee-dum twiddle thumbs, on a channel that doesn't go out to the user.
It needs a way to focus attention though because you need a much larger potential context.
Is there any work on attention being limited to looking at a subset of embeddings defined by a parameterisable function because if a model could emit special tokens to change those parameters, it would potentially be able to scan it's own memory. Could be tricky to train but a training process that reduced the number of locations attended to over time might force it to compensate for the loss in scope by focusing.
Claude can definitely wait 50 seconds and then do a second thing. I see it all the time when running tools that never stop but iterate towards a goal. It can either run a bash `sleep 50` or use the ScheduleWakeup tool.
Have you thought about using llm-wiki? It's pretty powerful and works well.
I’ve been thinking about going that route.
I’ve seen some good research where xx% of the time, all LLMs essentially just get lazy and won’t actually look things up.
Been working on an informational retrieval system similar, but endless Nerfs combined with non deterministic and finicky behaviors make it hard to reliably depend on.
How’s your luck been with LLm-wiki?
looks like a nifty information retrieval approach. cool. temporal/versioned chronology is absolutely useful.
It may seem useful at first, but eventually you'll hit a wall where this doesn't work, and you have to add another memory technique. Eventually you end up with a complex multi-layered system, because what people want by "memory" is actually 10 different things which all need their own solution.