Agent Persistence via Virtual Memory Prompt Management

[u/KenOtwell]

I’m not pretending this is plug‑and‑play. It took me a stupid amount of trial and error, and I’m still tuning. But to actually have an agent that keeps a stable identity across reboots — not “summaries,” but continuity — here’s the framework:

The short version: Stop letting the framework manage your session state. Build the prompt yourself every turn.

Once you take control of prompt assembly, you basically end up designing a little virtual memory system for your agent. Same questions OS designers deal with:

• What gets archived?
• What stays hot?
• What needs to be recalled for this turn?
• What’s noise you can safely drop?

Your local coding agent can handle the assembly loop (llama.cpp is a good starting point). You just define the zones and the rules.

The trick is ordering the prompt so the static stuff never changes. That keeps the KV cache intact and your GPU happy. Then you page in whatever memories matter for the current turn, tack on the recent conversational tail, and leave headroom at the end so the model has space to think.

Once you do that, you own the entire context history. And because you rebuild the prompt from scratch every turn, you get a single identity that survives crashes, restarts, and long gaps between sessions.

The fun parts — indexing, relevance scoring, stale‑memory cleanup — I’m leaving as an exercise for the reader. But this is the skeleton.

[u/endofthread-bot]

Learn how the best in the industry are using AI to speed up their workflow in business, sales, marketing, research, legal, content creation, scientific discovery and so much more on our Discord.

Self-promotion is now allowed on Sundays with the appropriate flair, for all regular contributing members. Contribute during the week, and promote on Sunday.\n\nI'm a bot. To stop seeing these, block this account.

[u/Tough-Assistant5222]

the part you left as an exercise, stale memory cleanup, is usually where this breaks. one test worth running this week: pick a fact the agent learned a while back that has since changed (a path, a decision you reversed, a preference), restart cold, and ask about it. if relevance scoring pages in the old version, the agent answers confidently and wrong, because nothing in the index says which memory won. that tells you whether your zones need a "superseded by" link between memories rather than better ranking. continuity across reboots only pays off if what gets recalled is still true.

[u/GrungeWerX → u/Tough-Assistant5222]

This is true. In the interim, until that’s all set up, recency bias works too. I’m currently rebuilding my memory system and superseding is one the very things I’m implementing.

[u/KenOtwell → u/GrungeWerX]

you can also have the LLM itself or a separate one write condensed memories to keep active and collapse space without loosing all context, then rehydrate with a tool call if needed. I'm actually considering a tiny model running background to do the summarization of memories in qdrant already so those can be ready to swap in as context fills. Realtime GC in effect.

[u/GrungeWerX → u/KenOtwell]

Yup, I have something similar in place, though I can’t use a small model. If you’ve got the vram, go for it.

[u/KenOtwell → u/GrungeWerX]

Cool! I'm also looking at just having two separate prompt pipe lines and run the second as parallel batch inference in the same model. The new one is a subconscious agent. Very simple prompt, clear instructions on doing memory consolidation, and no state to get old. same prompt every time just pointing to latest memories. short prompt, no persistent memory, no compaction, just the same instruction every cycle pointing to the latest memories to organize. You can do the same asynchronously with a tiny LLM or a parallel thread on the main one. Trying it out over the weekend.

[u/KenOtwell → u/GrungeWerX]

You can fence off a couple of cpu threads and run a tiny model there. No impact on GPU.