Before you fund an AI pilot, agree on what would make you stop it. A success target alone does not tell the team what to do when results are inconclusive, spending reaches its limit, or a serious risk emerges. We recommend a written decision contract that identifies the evidence, decision owner, review date, and action attached to each condition.
The most important distinction is between stopping user exposure now and stopping further funding at a scheduled review. A dangerous output or prohibited data exposure cannot wait for the investment committee. Equally, missing a business target does not automatically mean a system is unsafe or that the idea must be abandoned permanently. Those decisions need separate rules.
Separate value from permission to operate
AWS's guidance on designing a generative AI proof of concept connects measurable criteria to business goals and calls for exit thresholds covering quality, latency, and cost. It allows for pivoting or ending a PoC when those thresholds are not met. These are planning recommendations, not evidence that one numerical cutoff works for every organization.
Our proposed contract separates business value, technical quality, and hard boundaries. A value test asks whether the workflow improves enough over the existing approach to justify further investment. A quality test checks whether performance is adequate for the intended use. A hard boundary defines an event or condition that requires immediate containment, regardless of an acceptable average score.
Keep these tests independent. Staying within budget does not establish value. A strong aggregate quality score should not cancel a separately defined serious-harm condition. The sponsor should approve the relevant limits before the pilot begins, with the people responsible for operating and overseeing the workflow.
Define what counts as evidence
For each test, record the metric, how it will be measured, the comparison baseline, and the measurement window. Specify the population and workflow covered, as well as the evidence quality or minimum coverage needed to make the decision. Do not substitute an attractive percentage for an explanation of what was tested and what remains unknown.
NIST's AI RMF Playbook guidance on measurement recommends selecting risk metrics and documenting risks or trustworthiness characteristics that cannot be measured. Our recommendation is to carry those gaps into the review explicitly. An unmeasured risk is not a passed test, and an undersized or unrepresentative evidence set should not quietly become approval to expand. Appropriate thresholds and sample sizes require a decision for the specific use case.
Assign immediate authority separately from review authority
The NIST AI RMF Playbook's management guidance addresses whether to proceed with an AI system and mechanisms for disengaging or deactivating systems whose outcomes conflict with intended use. This is practice guidance, not a legal mandate. NIST notes that AI RMF 1.0 is being updated; these points refer to the currently displayed Playbook guidance.
For a pilot involving real users, we recommend naming who can disable access or stop the affected workflow, who acts if that person is unavailable, and where the decision is recorded. Define the fallback and how relevant evidence will be preserved. Test that fallback separately before relying on it. Disabling a system does not necessarily reverse actions already taken or restore information already disclosed.
The funding decision needs its own owner and date. Record who can authorize continuation, require revision, pause expansion, or end further spending. A successful demonstration in an isolated PoC is not, by itself, permission to expose real users or move into production.
Decide how to handle an inconclusive review
AWS's guidance on advancing a PoC to preproduction describes a decision gate comparing predefined objectives with observed results before further investment. We suggest agreeing in advance what happens when that comparison cannot support a decision.
One option is to pause expansion. Another is a bounded extension with a specific missing-evidence requirement, an accountable owner, a spending limit, and a new end date. Neither should count as success. A missed target may justify revising the experiment; repeated extensions should require an explicit new decision rather than continued spending by default.
Check the contract against a concrete scenario
Consider a hypothetical internal knowledge-search pilot. Its question is whether employees can find useful answers more effectively than with the existing search workflow. Before launch, the sponsor fills in [target], [budget], [review date], and [owner], then defines the evaluation method and coverage needed to compare the two approaches.
The contract separately specifies what exposure must stop immediately, such as access to information outside an approved boundary, and who activates the fallback. At the review date, insufficient evidence triggers the agreed pause or bounded-extension rule. No result is assumed here; the example shows which decisions need to be made before results arrive.
A decision contract does not prove that a pilot will succeed. It makes the next decision explicit enough to be reviewed against evidence.
Subscribe to Decision Crafters for evidence-backed AI and platform decisions.