AI evaluation economics: measure cost per accepted outcome

The cheapest model request can produce the most expensive usable result. A team saves a few cents on inference, then spends minutes checking an answer, repeating a call or escalating a failed case. A token dashboard sees the saving; the operating budget absorbs the difference.

The useful comparison is cost per accepted outcome, subject to quality and delivery requirements. Define the outcome first, then count everything needed to produce it. FinOps guidance explicitly encourages measures tied to the business task, rather than stopping at tokens or GPU hours. FinOps: AI tools and use-case economics.

This guide proposes an operating method using sources checked on October 7, 2026. Its experiment, prices and results are fictional; they illustrate a decision, not a provider benchmark.

Make acceptance a decision someone can reproduce

Choose a narrow task. For example: extract three required fields from an engineering change record, attach the supporting passages and send a reviewer an accurate summary. An accepted case contains all three correct fields, supports every material statement and needs no substantive correction after the permitted retry.

Set the operating conditions before comparing configurations. The example below requires at least 90% of cases to pass that rubric, no observed critical disclosure errors, and a 95th-percentile machine completion time below 15 seconds, including retries. Human review effort is measured separately. These thresholds are invented for this task, not universal standards; zero observed disclosure errors does not establish zero risk.

NIST's generative AI profile recommends measurement under conditions resembling deployment and cautions against generalizing from narrow or anecdotal assessments. It also calls for checking generated sources and citations. NIST AI 600-1, Measure 2.3 and 2.5.

Use the same representative cases for both configurations. Keep a held-out set for the final comparison, inspect results by task difficulty, and report sample size and uncertainty. Repeatedly tuning against the final set makes its apparent improvement less informative.

Count the whole attempt, including unsuccessful work

Give each business case a stable ID. Attach every model call, tool call, retry and review decision to that ID. A case with four calls remains one attempted outcome. Include abandoned cases and timeouts in the attempted population.

Use this calculation:

Cost per accepted outcome = total cost of processing the evaluated population ÷ accepted cases in that population.

The numerator includes costs incurred on failures. Keep separate columns for inference, other infrastructure, reviewer effort and additional failure handling. Record whether each amount is an invoice-backed charge, a usage estimate or a labor valuation. Handle one-time development and evaluation costs explicitly rather than hiding them in an arbitrary unit rate.

If nothing passes, report no accepted outcomes and the total cost incurred. A zero denominator is useful evidence that the workflow is not ready.

A worked comparison: four times the request price, lower delivery cost

Illustrative USD example. Two configurations process the same 1,000 short records. Each case receives one initial call and at most one retry. Fictional flat call prices stand in for measured usage costs; they are not current provider rates. Review includes judging both successful and unsuccessful cases. Terminal failure handling is additional triage, not already-counted review or eventual manual completion.

The exact unit calculations are $642 / 800 = $0.8025 and $357 / 950 ≈ $0.3758. Both configurations satisfy the assumed latency limit, but A fails the quality requirement. B's higher inference bill is outweighed by lower review and triage effort.

The $0.50 triage allowance covers a small administrative step. It does not price customer harm, lost revenue or regulatory consequences. Estimate those separately where relevant; serious failure modes can make a configuration unacceptable regardless of its average cost.

Do not extrapolate the result automatically to harder records or higher traffic. This table supports a bounded pilot decision, not a universal model ranking.

Give the experiment a budget and a stopping rule

In this example, testing both configurations costs $999 in modeled operating expense. Add $800 of setup and evaluation design: the pilot totals $1,799. A $2,000 budget leaves $201 for overruns. Keep that setup expense visible when deciding whether the expected production volume justifies further investment.

Before running, set maximum cases, retries and tool calls; assign someone to stop the experiment when cost or failure thresholds are crossed. Reserve budget for requests already in flight. Review time needs a budget too, even when salaried labor creates no new cash payment.

Retain the dataset version, rubric, model and prompt versions, run IDs, timing method, actual charges and reviewer decisions. This turns a cheaper-looking configuration into a testable engineering claim.

Test caching and batch processing against the same objective

Caching deserves its own experiment. Google Cloud's documentation distinguishes initial input processing from cached reuse and notes additional storage charges for explicit caches. Compare observed reuse savings with cache creation, storage and maintenance costs. Preserve permission boundaries and test behavior when source documents change. Google Cloud: context caching.

Batch processing changes the delivery window. Anthropic documents asynchronous batches that can take up to 24 hours and may expire incomplete. That may suit an overnight evaluation, while failing an interactive workflow's latency requirement. Include expired requests and reruns when assessing the saving. Anthropic: batch processing.

Neither optimization earns a pass simply because its invoice is smaller. Recheck accepted outcomes under the same task definition.

Connect efficiency to attributable value

A lower unit cost is evidence about efficiency. Revenue growth, avoided contractor expense and released staff capacity need their own support. FinOps distinguishes resource measures from business outcomes and calls for baseline, intervention and outcome evidence. FinOps: unit economics.

Tax credits and deductions require a separate assessment; an evaluation record does not determine eligibility. For an initial Proforma evidence review, teams can supply experiment records and cost exports. Preparation may be manual, and a configured Workspace agent can help investigate available Data. The useful deliverable is an inspectable link between what was attempted, what passed and what it cost. Explore the evidence workflow.

Sources and scope

• FinOps Foundation: AI tools and use-case economics
• NIST: Generative AI Profile, AI 600-1
• Google Cloud: context caching overview
• Anthropic: batch processing documentation
• FinOps Foundation: unit economics capability

All numerical results above are original illustrative calculations. Provider behavior and commercial terms should be checked for the services and contracts actually used.