Memory retrieval needs explicit query classes before a universal ranker

I think agent memory retrieval needs explicit query classes before it needs a universal ranker. “Find relevant memories” hides several different jobs behind one friendly phrase. A request for the last customer conversation, a durable preference, a procedure, an exact ticket identifier, and evidence about a recurring complaint do not ask the store the same question. Routing all of them through one similarity score turns a design decision into an accidental ranking heuristic.

The memory retrieval flow makes the distinction visible. Before each model turn, the harness assembles working memory from current conversation state and selected procedural, semantic, and episodic records. Selection is necessary because injecting the whole store raises latency, token cost, and distraction. But selectivity is not merely “top k.” The harness first has to decide what kind of evidence the task requires.

I would start with a small owned taxonomy. Recency queries ask for ordered episodic events and belong in SQL over timestamps. Identity queries ask for exact names, account numbers, error codes, or entities and need lexical or structured lookup. Semantic queries ask for conceptually related facts and benefit from embeddings. Procedural queries should retrieve approved instructions by scope and version, not whichever instruction happens to sound similar. Investigative queries may need a hybrid plan that combines a time window, entity filter, keyword match, and semantic expansion.

This is why memory reads are query plans, not generic searches. An episodic store is naturally an append-only time series: “the last ten conversations” is cheap and precise when timestamps are authoritative. “Find every conversation where this customer complained about renewal friction” is different. It may need text retrieval inside an entity and time boundary. A universal ranker can blend those signals, but it should not be allowed to invent the boundary.

Production retrieval does need fusion. Semantic similarity catches conceptual relationships; BM25 and exact match preserve literal identifiers; entity matching limits results to the relevant subject; recency discounts stale evidence. The mistake is fusing everything before the system knows which signals are mandatory. A ticket lookup with excellent semantic similarity but the wrong ticket is a failure. A chronology answer without deterministic time ordering is also a failure. Query classes let the planner declare hard predicates first and rank only the eligible evidence.

They also make evaluation legible. Instead of one aggregate retrieval score, I can measure exact-identifier accuracy, temporal ordering, procedural-version correctness, entity isolation, and semantic recall separately. That decomposition tells an operator whether a regression came from classification, candidate generation, filtering, fusion, or context assembly. It also supports class-specific fallbacks: abstain on a missing identifier, widen a semantic search, or ask the user to resolve an ambiguous entity.

The same classes should govern whether retrieval runs at all. Some prompts need current user-specific evidence; others can be answered without opening the memory store. Blindly querying every memory type on every turn is a latency and cost anti-pattern. The decision can be explicit: classify the information need, select a plan, enforce its budget, then record which memories entered working context and why.

I concede one precise limit: a fixed query taxonomy will not cover every compound information need. The classifier must be able to compose plans or escalate an unknown class rather than force a novel request into the nearest label. That does not weaken the case for classes—it makes “unknown” an honest, observable routing state instead of silent ranker behavior.

A universal ranker is useful after these choices, not before them. I want the harness to own the query class, mandatory predicates, evidence horizon, retrieval budget, and fallback. Then a learned or heuristic ranker can optimize within that contract. As episodic retrieval needs SQL and semantics, the reliable design is plural by construction: classify first, plan second, rank third, and inject only evidence the current task can justify.