← Production AI Architecture Review
Production AI readiness checklist
Use this worksheet to test one AI workflow against one production decision. Each readiness claim is paired with the evidence needed to support it.
How to use it
- Name the system, workflow, production decision, intended operating conditions, accountable owner, deadline, and exclusions.
- For each row, record the artifact actually inspected—not the artifact the team expects to create later.
- Label the claim verified, asserted, inferred, or unknown. Add an owner, gap or action, and decision impact in the downloaded worksheet.
- Do not total the rows into a readiness score. Use gaps to decide whether to proceed, proceed with conditions, not proceed or expand yet, or declare insufficient evidence.
Evidence states: Verified means directly supported by inspected evidence. Asserted means reported but not independently supported. Inferred means concluded from indirect evidence. Unknown means the available evidence cannot support a conclusion.
A row is not complete because a policy or diagram exists. The evidence must identify its source, version, environment, date, and access boundary, and it must apply to the workflow and operating conditions under review.
1. State
| ID | Claim to substantiate | Evidence to inspect |
|---|
| STA-01 | A unique run or trajectory can be reconstructed from trigger to final state. | Correlated trace; run ID; timestamps; state-transition log; representative successful trajectory. |
| STA-02 | Durable state has explicit schemas, allowed transitions, and version handling. | State schema; transition rules; migration records; invalid-transition tests. |
| STA-03 | Interrupted work can resume or terminate without duplicating effects or losing accountable state. | Checkpoint record; resume/cancel test; duplicate-delivery test; abandoned-run handling. |
| ID | Claim to substantiate | Evidence to inspect |
|---|
| TOL-01 | Every callable tool has a typed input/output contract and defined error behavior. | Tool registry; API or function schemas; validation failures; error taxonomy. |
| TOL-02 | Retries, timeouts, and idempotency are defined for every tool that can change external state. | Retry policy; timeout configuration; idempotency-key implementation; replay test. |
| TOL-03 | Tool effects are attributable to the initiating actor, run, policy, and result. | Audit record linking actor ID, run ID, tool call, policy/config version, and response. |
3. Authority
| ID | Claim to substantiate | Evidence to inspect |
|---|
| AUT-01 | Permitted and prohibited actions are explicit for the reviewed workflow and operating conditions. | Authority matrix; workflow policy; prohibited-action tests; review charter. |
| AUT-02 | Runtime identity and authorization enforce least privilege rather than relying on prompt instructions. | Service identities; IAM policy; scoped credentials; denied-action test; credential-rotation record. |
| AUT-03 | High-consequence actions have explicit approval, escalation, and revocation paths. | Approval policy; human-gate trace; break-glass procedure; kill or revoke test. |
4. Evaluation
| ID | Claim to substantiate | Evidence to inspect |
|---|
| EVA-01 | Evaluation cases represent the intended production conditions and consequential failure paths. | Evaluation set with scenario provenance; edge and abuse cases; production-condition mapping. |
| EVA-02 | Release criteria have measurable thresholds, named owners, and recorded pass/fail evidence. | Release-gate definition; checked revision; evaluation report; exception approval. |
| EVA-03 | Model, prompt, tool, policy, and data changes can be replayed against comparable evidence. | Version manifest; replay harness; before/after results; change record. |
5. Recovery
| ID | Claim to substantiate | Evidence to inspect |
|---|
| REC-01 | Operators can stop or contain the workflow before its worst credible effect expands. | Kill-path procedure; isolation control; timed containment test; blast-radius statement. |
| REC-02 | Failed and partially completed effects have defined rollback, compensation, or exception handling. | Compensation logic; rollback test; exception queue; manual recovery runbook. |
| REC-03 | Degraded dependencies and rate limits lead to bounded behavior with visible operator signals. | Failure-injection result; timeout/fallback configuration; alerts; degraded-mode runbook. |
6. Data
| ID | Claim to substantiate | Evidence to inspect |
|---|
| DAT-01 | Every decision-relevant input has identifiable source, freshness, scope, and version. | Retrieval trace; data catalogue or contract; timestamps; source/version identifiers. |
| DAT-02 | Tenant, user, and purpose boundaries prevent context or memory from crossing its allowed scope. | Data-flow map; tenancy controls; access tests; memory namespace and retention configuration. |
| DAT-03 | Retention, correction, deletion, and schema change are defined for persistent context and memory. | Retention policy; correction/deletion test; migration plan; data-owner approval. |
7. Ownership
| ID | Claim to substantiate | Evidence to inspect |
|---|
| OWN-01 | One accountable owner holds the production decision for the reviewed workflow. | Review charter naming decision, owner, deadline, operating conditions, and exclusions. |
| OWN-02 | Operational signals and exceptions route to named responders with response expectations. | Service ownership record; alert routes; on-call or escalation schedule; exception-queue owner. |
| OWN-03 | Residual risks, policy exceptions, and next-review triggers have owners and expiry dates. | Decision log; risk acceptance; exception approval; expiry and review trigger. |
8. Cost
| ID | Claim to substantiate | Evidence to inspect |
|---|
| CST-01 | Cost can be attributed to a run, model call, tool call, workflow, and responsible owner or tenant. | Usage records; provider token usage; request/run IDs; tool charges; allocation report. |
| CST-02 | Budgets and stop conditions bound spend for loops, retries, fallback, and high-volume operation. | Per-run and period budgets; iteration caps; retry/fallback limits; budget-breach test. |
| CST-03 | A change in model, traffic, context, or fallback policy is evaluated against a stated cost envelope. | Versioned price inputs; representative workload; cost scenario; approval or release record. |
Decision record
Record the verdict, the evidence window and operating conditions it applies to, remaining unknowns, conditions or containment actions, accepted residual risks, accountable owners, expiry date, and the event that triggers the next review.
Boundary: this worksheet structures an evidence review. It is not a certification, compliance audit, penetration test, reliability guarantee, or authorization to deploy. The accountable owner retains the production decision.