Exact identifiers deserve lexical retrieval before semantic fusion

Exact identifiers deserve lexical retrieval before semantic fusion. When a query contains an invoice number, error code, API symbol, product SKU, legal clause, or quoted phrase, the literal token is not noise around the meaning. It is often the strongest evidence of user intent.

Dense retrieval is designed to place semantically similar text near each other. That is useful for paraphrases and concepts, but identifiers are intentionally arbitrary. The strings INV-10482 and INV-10428 may be visually close while referring to different obligations. An embedding can smooth that distinction because semantic neighbourhoods reward resemblance. An inverted index preserves it because the token either occurs or it does not.

I therefore classify the query before choosing the retrieval blend. High-entropy alphanumeric strings, code-shaped tokens, quoted text, names with version suffixes, and domain-specific keys receive an exact or normalized lexical pass. BM25 remains useful for ordinary words because term rarity carries signal; exact-match clauses handle identifiers; field-aware search keeps a ticket ID from competing with incidental mentions in body text.

Semantic retrieval then broadens the candidate pool for the surrounding question. A user may provide the correct incident number but describe the symptom loosely, or mention a class name while asking for architectural context. Dense candidates can recover paraphrases and related explanations that lexical search misses. Reciprocal-rank fusion is attractive here because it combines rank order without pretending that BM25 scores and cosine similarity share a calibrated scale.

The sequence matters. If exact evidence is diluted too early into a weighted average, a semantically polished document can outrank the record named by the user. I treat lexical hits on validated identifier fields as constraints or dominant features, then fuse within that boundary. Vector search remains one index among several, selected by query shape rather than vendor preference.

This also creates better evaluation. Build slices for exact identifiers, rare entities, paraphrases, misspellings, mixed queries, and queries where the identifier is wrong. Measure candidate recall separately for lexical and dense paths, then measure fusion and reranking. Retrieval failure must be separated from answer failure; hybrid systems need the same separation between candidate generators.

Normalization deserves restraint. Case folding, punctuation removal, and token splitting can help users who type identifiers inconsistently, but aggressive normalization can collapse distinct values. Preserve the raw query, emit typed variants according to domain rules, and record which variant matched. For a checksum, casing may matter; for a support ticket prefix, it may not. The schema should decide, not a generic text-cleaning pipeline.

There is one precise concession: some domains use identifiers that are routinely corrupted, OCR-damaged, or spoken aloud. Strict exact matching alone would sacrifice too much recall there. Character n-grams, edit-distance candidates, phonetic variants, or learned sparse models can help—but they should produce explicitly lower-confidence candidates that require corroboration before an effectful action.

RAG is a data pipeline, and query analysis is part of that pipeline. I do not ask dense retrieval to rediscover exactness it was built to abstract away. Let lexical search protect literal intent, let semantic search recover conceptual intent, and make fusion rules visible enough to test when the two disagree. Log the detected query type, candidate source, normalization path, fusion contribution, and final reranker decision. Those records turn a mysterious bad answer into a repairable retrieval defect instead of another round of prompt tuning.