2026-07-08 04:00 UTCOriginal source3 min readUpdated: 2026-07-08 08:05 UTC

Memory in the Loop: In-Process Retrieval as Extended Working Memory for Language Agents

This research proposes moving memory storage inside the language agent's reasoning loop, reading and writing at every step to overcome network latency. Experiments show that in-process storage (~100μs) reduces redundant actions from 7.2/12 to 0.0/12 and improves recall from 0/5 to 3.6–4.8/5. The bottleneck shifts to embedding generation rather than storage.

SourcearXiv AIAuthor: Yusuf Khan, Carlo Lipizzi

Article intelligence

EngineersAdvanced

Key points

Traditional language agents query memory outside the loop, limited by network latency (tens to hundreds of ms), inflating per-step delay up to 83x.
In-process storage responds in ~100μs, effectively becoming extended working memory rather than an external tool.
Experiments show in-process storage eliminates redundant actions (0.0/12), while cloud storage results in 7.2/12 redundancies.
Pairing with local embedder reduces complete operation to ~40μs, outperforming network embedding's 200–400ms.

Why it matters

This matters because traditional language agents query memory outside the loop, limited by network latency (tens to hundreds of ms), inflating per-step delay up to 83x.

Technical impact

May affect model selection, inference cost, product capability, and evaluation benchmarks.

This panel is AI-generated and reviewed for accuracy.

-->

[Submitted on 6 Jul 2026]

Title:Memory in the Loop: In-Process Retrieval as ExtendedWorking Memory for Language Agents

View a PDF of the paper titled Memory in the Loop: In-Process Retrieval as ExtendedWorking Memory for Language Agents, by Yusuf Khan and 1 other authors

View PDF HTML (experimental)

Abstract:Language agents run a loop - observe, reason, act - but the memory they reason over sits outside it: a store queried at most once per turn. We study the regime where memory moves inside the loop, read and written on every step. The obstacle has always been latency: networked stores answer in tens to hundreds of milliseconds, and in-loop retrieval can inflate end-to-end latency by up to 83x when retrieval is expensive. Prior work manages that cost rather than questioning it: serving-layer scheduling hides it, "memory-first" designs ration retrieval to once per turn. We argue latency is a property of where the store lives, not the in-loop pattern: an in-process store answers in ~100us, three orders of magnitude below the network regime, and at that speed the per-step tax collapses. By the extended-mind thesis's parity principle, a store fast enough to be constantly and directly available becomes extended working memory, not a tool the agent merely consults. The premise is causal: holding a fixed per-turn memory-latency budget and varying only the store's answer speed, redundant actions rise monotonically with latency - 0.0 of 12 at in-process speed, 7.2 of 12 at a 110ms cloud round trip (gpt-5-nano, gpt-5-mini; exact permutation p=0.0079). We demonstrate the regime end-to-end: across four GPT-5-class models under a bounded window, recall improves from 0/5 to 3.6-4.8/5 with in-loop memory, store ops at p50 80-165us - though an instructed restate-every-reply baseline also solves it perfectly, at a token cost that grows with the working set. The store never lost a fact in any run (244 of 244 writes kept); every miss traces to the agent's read policy, not the store. Our measurements also relocate the bottleneck: the dominant per-step cost is embedding (~200-400ms over the network); pairing the in-process store with a small local embedder returns the complete operation to a measured ~40us.

Subjects:

Artificial Intelligence (cs.AI); Computation and Language (cs.CL)

Cite as: arXiv:2607.05690 [cs.AI]

(or arXiv:2607.05690v1 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2607.05690

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Carlo Lipizzi [view email] [v1] Mon, 6 Jul 2026 23:16:11 UTC (589 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Memory in the Loop: In-Process Retrieval as ExtendedWorking Memory for Language Agents, by Yusuf Khan and 1 other authors

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.AI

new | recent | 2026-07

Change to browse by:

cs cs.CL

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)