Architecture pillar

Data foundations for AI

Reliable AI begins with governed inputs and explicit data contracts.

A model can consume a value without knowing its business meaning, authority, freshness, or permitted use. Production reliability starts upstream: sources need contracts, identities, provenance, temporal semantics, quality expectations, and owners who can respond when the world changes.

Two problem paths enter this territory: enterprise data that is not ready for AI use and a RAG system whose corpus or index cannot be trusted.

Make input contracts executable

A contract names schema, identifiers, grain, units, required fields, freshness, expected distributions, and the response to a violation. It belongs at the producer-consumer boundary, before a silent change can alter an answer or external action.

Data contracts are the API layer of the data platform →

Carry provenance through transformation

Every output used by an AI system should remain traceable to authoritative source material, the transformations applied, and the policy that allowed its use. Provenance supports correction, rollback, citation, and an honest answer when the evidence is incomplete.

Provenance is what makes model rollback possible →

Treat extraction and exceptions as quality gates

OCR, parsing, chunking, and structuring are lossy operations. Measure coverage at the field or semantic-unit level, validate independently of extractor confidence, and route unresolved cases to an owned exception queue rather than admitting plausible corruption.

Document extraction is the enterprise RAG bottleneck →

Represent time and lifecycle explicitly

Current truth, historical truth, late-arriving correction, retirement, and deletion are different states. Data products need lifecycle ownership across change, and AI systems need to know which version supported a decision at the time it was made.

Data products need lifecycle management, not just owners →

Keep retrieval a derived index

A vector store accelerates one read path; it is not the authoritative corpus. Rebuild, invalidation, access scope, chunk lineage, freshness, and deletion must flow from governed source records. Query tuning comes after the corpus and index lifecycle are operable.

Vector search is an index, not a database →

Proof and limits

Inspect governed data change

See how one external delivery becomes a reproducible pass, review, block, or invalid decision before production data is touched.