Coincidentally, I am working on the same thing right now. However, unlike storing conversations in an external retrieval system, I use a local LLM to store the conversation's KV cache and perform retrieval directly on that cache. The method involves running a prefill pass and, after obtaining the attention scores, filtering for the corpus segments that received attention.
This aligns with the "zero tokens" approach described in this paper. :)
I also tested it on the LoCoMo used in this paper, and also LongMemEval, both achieved SOTA results.
The useful contribution is not zero token cost; it is removing generative rewriting from memory. Preserving original traces avoids a subtle auditability failure: once an LLM compresses an interaction, retrieval is grounded in the summary's omissions rather than the evidence.
I would still want a harder benchmark around mutation and contradiction. If an entity changes attributes across sessions, can the graph and temporal hierarchy preserve both states, surface the conflict, and show which trace justified the answer? The 57.6% time reduction is compelling, but for production agents I would measure unsupported-answer rate and evidence recall under stale, conflicting, and adversarial traces. Encoder compute and index-maintenance cost should also sit beside token cost; otherwise "zero-token" risks being read as "free."
Coincidentally, I am working on the same thing right now. However, unlike storing conversations in an external retrieval system, I use a local LLM to store the conversation's KV cache and perform retrieval directly on that cache. The method involves running a prefill pass and, after obtaining the attention scores, filtering for the corpus segments that received attention.
This aligns with the "zero tokens" approach described in this paper. :)
I also tested it on the LoCoMo used in this paper, and also LongMemEval, both achieved SOTA results.
The useful contribution is not zero token cost; it is removing generative rewriting from memory. Preserving original traces avoids a subtle auditability failure: once an LLM compresses an interaction, retrieval is grounded in the summary's omissions rather than the evidence.
I would still want a harder benchmark around mutation and contradiction. If an entity changes attributes across sessions, can the graph and temporal hierarchy preserve both states, surface the conflict, and show which trace justified the answer? The 57.6% time reduction is compelling, but for production agents I would measure unsupported-answer rate and evidence recall under stale, conflicting, and adversarial traces. Encoder compute and index-maintenance cost should also sit beside token cost; otherwise "zero-token" risks being read as "free."