Architecture pillar

From demo to production

Diagnose the production gap before buying a better model.

A demonstration proves that a model can produce a useful result under selected conditions. An operated product must produce, constrain, evaluate, recover, and own that result against real inputs, dependencies, users, and failure modes. The gap is therefore a systems diagnosis, not another model comparison.

Start with the stalled-demo diagnostic path →

Define the workflow and outcome

Name the production decision, the user or operator who depends on it, the measurable outcome, and the smallest trajectory that creates value. Keep enumerable control flow deterministic and spend autonomy only where runtime discovery materially improves the result.

Choose workflows where determinism is cheap →

Test production-shaped inputs

Curated examples hide missing values, ambiguous identity, stale records, adversarial content, distribution shifts, and unavailable dependencies. Evaluation data should represent the conditions the system will actually face and the uncertainty it must surface.

The demo-to-production gap is an architecture problem →

Make integration and authority explicit

Every read and write needs an interface contract, service owner, identity, permission scope, timeout, and failure response. A demo can rely on manual setup and ambient credentials; a production slice must make those dependencies and boundaries reviewable.

Evaluate under real operating conditions

Evals should be executable product contracts: quality, containment, cost, latency, abstention, recovery, and side effects all belong in the release decision. A persuasive answer in a notebook is not evidence that the trajectory is safe to operate.

Evals are the contract, not the report card →

Assign end-to-end ownership

Someone must own exceptions, degraded dependencies, policy changes, rollback, and the final proceed decision. If the automation hands unresolved work to nobody, the happy path is not a product.

Own the exception queue before the happy path →

Choose the smallest credible production slice

The first release should bound users, data, actions, authority, and time while preserving enough evidence to decide what comes next. A supervised pilot is credible when it tests the riskiest seam, not when it merely reduces the number of users.

Proof and limits

Make one production decision

Bound one use case, reconstruct the as-built system, inspect the highest-risk seams, and decide whether to proceed, proceed with conditions, stop, or gather more evidence.