Move AI from a useful demo to a defensible release decision

Distinguish PoC, pilot, readiness, implementation, and operations

These stages are often blurred, which creates scope and expectation problems.

Proof of concept

Tests technical feasibility in a controlled environment. It may use selected examples, synthetic data, or a simplified integration. Its output is evidence about feasibility, not a production recommendation.

Paid pilot

Tests a bounded use case with realistic users, data, workflow, and constraints. It should have a named decision owner, a defined period, client responsibilities, success and stop criteria, and an explicit next decision.

Readiness engagement

Examines whether the system has the architecture, evaluation, authority, data handling, documentation, and operational controls required for a production decision.

Implementation

Builds or integrates the accepted technical controls, evaluation fixtures, release gates, human-review workflow, and operational response.

Recurring oversight

Maintains evidence, reviews changes, reassesses risks, and supports repeated decisions when the system or context changes.

A team should not call a prototype "production-ready" because it completed a pilot. Readiness and implementation are separate claims.

Start with the use case and consequence

Model choice should not be the first decision. Define:

• Business problem and intended user
• Decision or action the output can influence
• Affected parties
• Consequence of a wrong, missing, late, or unauthorized result
• Required evidence
• Human authority
• Data and access boundaries
• Operational fallback
• Expected volume, latency, and cost constraints

This framing determines the evaluation plan. A low-consequence drafting aid may tolerate different failures than an eligibility, financial, safety, or account-access workflow.

Build a system inventory

A release review should identify the complete path:

• User interface and entry points
• Models and providers
• System prompts and templates
• Retrieval sources, chunking, embeddings, ranking, and filters
• Tools, APIs, and downstream actions
• Identity and authorization
• Data retention and logging
• Human review and escalation
• Evaluation code and datasets
• Deployment, monitoring, and rollback
• External dependencies and rate limits

The purpose is not paperwork. The inventory creates a stable description of what was evaluated. Without it, later test results cannot be reproduced or compared after a model, prompt, document corpus, access policy, or vendor changes.

Use representative case families

A small set of attractive examples proves little. Build case families that reflect real decision risk.

Golden cases

Common, important tasks with authoritative expected behavior.

Edge cases

Boundary values, ambiguous language, uncommon document structures, partial data, or unusual user context.

Adversarial cases

Prompt injection, misleading source content, conflicting instructions, malicious formatting, or attempts to expand tool authority.

Refusal and insufficient-evidence cases

Requests the system should decline, defer, or route to a human because sources are absent, authority is missing, or the task is outside scope.

Access cases

Users, tenants, roles, and sources that test whether relevant but unauthorized information remains excluded.

Operational cases

Timeouts, dependency failures, model unavailability, retry behavior, high latency, high cost, malformed tool responses, and reviewer backlog.

Change cases

Known regressions and representative fixtures used whenever the prompt, model, retrieval, code, or policy changes.

Evaluate the path, not one score

For retrieval-augmented generation and AI-assisted workflows, separate failure categories:

• Retrieval: the required source was not found
• Ranking: the source was found but not prioritized
• Access: the system retrieved or exposed unauthorized material
• Grounding: the answer was not supported by retrieved evidence
• Citation: the cited location did not support the claim
• Reasoning or synthesis: the sources were present but combined incorrectly
• Instruction following: the output violated workflow or formatting rules
• Tool use: the model chose, parameterized, or interpreted a tool incorrectly
• Human review: the interface or workload made informed review impractical
• Operational: latency, cost, timeout, capacity, or dependency behavior failed

A composite score can hide this distinction. A system with high average relevance and one serious cross-tenant access failure may be unacceptable.

Evaluation frameworks such as Pydantic Evals support code-first datasets, evaluators, and assertions. [S4] The tool is less important than the discipline: representative fixtures, explicit assertions, versioning, calibration, and review.

Calibrate automated evaluation against humans

LLM-as-judge or other automated evaluators can improve coverage, but they are themselves models with failure modes.

Use human-reviewed control sets to examine:

• Agreement by case type
• Systematic leniency or severity
• Sensitivity to answer style
• Ability to detect unsupported claims
• Stability across evaluator model versions
• Handling of partial correctness
• Treatment of refusals and unknowns

Do not claim that an evaluator is objective merely because it produces a number. Retain evaluator prompts, model versions, rubrics, and calibration results.

Version the evidence

Every meaningful result should be traceable to:

• Application build
• Model and provider version
• Prompt and policy version
• Retrieval corpus and index version
• Access-control configuration
• Tool and API versions
• Evaluation dataset version
• Evaluator version
• Date and environment

When any of those change materially, decide which tests must rerun. A release score from an earlier model or corpus is historical evidence, not current proof.

Create a failure taxonomy and response path

A failure taxonomy turns evaluation results into work. Each failure category should have:

• Definition
• Severity or consequence
• Owner
• Detection method
• Example cases
• Remediation options
• Release effect
• Monitoring signal
• Escalation or incident path

This prevents every problem from being described as "hallucination" and routed back to prompt tuning.

Release scorecards should support a decision

A useful scorecard is not a decorative dashboard. It should show:

• Scope and system state tested
• Case coverage
• Results by risk-relevant category
• Access or blocked-action failures
• Human calibration status
• Cost and latency ranges
• Open defects and unknowns
• Required remediation
• Exceptions accepted by named owners
• Rollback and monitoring conditions
• Recommendation: proceed, narrow, remediate, remain in pilot, or stop

NIST's AI RMF and its Generative AI Profile provide a voluntary structure for governing, mapping, measuring, and managing AI risk. [S1] [S2] The NIST Playbook offers suggested actions rather than a universal checklist. [S3] A release scorecard should similarly reflect the organization's use case, risk tolerance, and resources rather than claim a universal passing score.

Unknown can be the correct result

Unknown is appropriate when:

• The required source does not exist
• The dataset lacks representative cases
• Domain reviewers disagree
• Access behavior cannot be verified
• A dependency is opaque
• Production volume has not been tested
• The system state changed after evaluation
• A legal or policy question requires qualified review

Converting unknowns into optimistic assumptions creates false confidence. Assign an owner, consequence, and next action instead.

Procurement evidence should travel

Enterprise and public-sector reviewers often need concise artifacts:

• System card
• Architecture and data-flow diagram
• Model and vendor inventory
• Data use and retention summary
• Human-authority matrix
• Evaluation and release summary
• Known limitations
• Change and review dates
• Responsible contacts

These artifacts should be accurate enough for technical review and restrained enough not to imply certification or legal assurance.

Implementation checklist

• [ ] Define the use case, affected parties, consequence, and decision owner.
• [ ] Distinguish PoC, pilot, readiness, implementation, and recurring oversight.
• [ ] Inventory the full application, model, data, retrieval, tool, access, and review path.
• [ ] Build golden, edge, adversarial, refusal, access, operational, and change cases.
• [ ] Separate retrieval, grounding, citation, tool, access, human-review, and operational failures.
• [ ] Calibrate automated evaluators against human-reviewed controls.
• [ ] Version application, model, prompt, corpus, access, tool, and evaluator state.
• [ ] Define failure ownership and remediation paths.
• [ ] Create explicit proceed, narrow, remediate, pilot, stop, rollback, and blocked-action criteria.
• [ ] Retain unknowns with owners rather than converting them into pass results.
• [ ] Produce a concise technical evidence package for leadership and procurement.
• [ ] Rerun required evidence after material system change.

Decision connection

A defensible release is not a guarantee of perfect behavior. It is a documented decision based on representative evidence, named authority, accepted limitations, operational controls, and a plan for change.

The AI Production Readiness & Evidence Sprint establishes that decision package. The AI Evaluation & Release-Gate Implementation makes the evidence repeatable across future releases.