OpenAI CFO Sarah Friar introduces an AI scorecard to measure ROI via task cost, dependability, and compute return.

What Happened

OpenAI CFO Sarah Friar has introduced a practical "AI scorecard" designed to help enterprises measure the actual return on investment (ROI) of generative AI deployments. The framework aims to standardize how organizations evaluate AI value, moving the conversation away from raw model benchmarks and toward tangible business and operational outcomes.

Technical Details

The scorecard introduces four core pillars for evaluation:

1. Useful Work: Quantifying the actual time saved or net-new capabilities unlocked by AI agents, rather than just measuring token throughput or API calls.
2. Cost Per Successful Task (CPST): A critical shift from raw API pricing (cost per 1k tokens) to the fully loaded cost of completing a specific workflow. This factors in the overhead of retries, multi-step prompt chaining, and orchestration.
3. Dependability: Measuring system reliability in production environments, encompassing uptime, error rates, hallucination frequency, and output consistency.
4. Return on Compute (RoC): Evaluating whether the compute expenditure translates directly to measurable business value, acting as a forcing function for efficient architecture.

Why It Matters

From an engineering and FinOps perspective, this is a necessary maturation of AI metrics. For the past two years, teams have struggled to translate theoretical benchmark scores (like MMLU or HumanEval) into defensible business cases. By establishing metrics like CPST and RoC, OpenAI acknowledges that inference costs are facing heavy enterprise scrutiny. This framework gives architecture teams a standardized vocabulary to justify model right-sizing—arguing for smaller, faster models for routine tasks, while justifying the premium of frontier models when high dependability is paramount. It effectively bridges the gap between AI engineering, DevOps, and finance.

What to Watch Next

Expect enterprise LLM observability platforms (such as LangSmith, Helicone, or Datadog) to natively integrate these specific scorecard metrics into their default dashboards. Additionally, monitor whether competing foundation model providers adopt this exact framework to compete head-to-head on "cost per successful task" rather than baseline token pricing.