Understand the tradeoffs
Start with the task your app performs. For a routing workflow, success might mean placing an email in the correct department. For a support assistant, it might mean resolving an issue without reopening it. These outcomes can require different model capabilities even when both apps process similar amounts of text. Cost per request is useful for engineering diagnostics, but it can hide failed attempts. Suppose, illustratively, one configuration spends 0.0111. A configuration that spends 0.0133 per success, before any human correction. Define both the numerator and the success denominator so lower spending cannot conceal worse outcomes. Latency also needs a task-specific definition. A user-facing chat may depend on time to first token and total response time. An asynchronous routing job may care more about finishing within a queue deadline. Choose the measure that reflects the user’s experience instead of adopting a model benchmark’s average latency.Define your objective to know the tradeoff
Write one primary objective and a small set of constraints. A hypothetical routing team might aim to reduce inference cost per correctly routed email by 20%, maintain at least 98% accuracy on its held-out evaluation set, and finish 95% of jobs within five seconds. Those values are example targets, not Narev recommendations or expected results. Select thresholds based on your current baseline and the consequences of errors. Specify which errors are unacceptable. Overall accuracy can conceal a serious regression in rare categories. If refund disputes require manual review, evaluate that category separately and preserve the escalation path. Decide how to score ambiguous messages before testing candidates; otherwise, you can unintentionally favor the model whose mistakes are easiest to overlook. Record the measurement window, included expenses, and workload boundary. If you include inference but exclude retrieval, database costs, and human review, say so. You can broaden the boundary later, but baseline and candidate must use the same one. Step 2 explains how to build the spending baseline that makes this objective measurable.Get everyone aligned
Include the feature owner, the engineer who can implement and roll back changes, the person responsible for evaluation, and the person tracking the budget. Small teams can assign several roles to one person. The important point is that each decision has an accountable owner. Agree on who makes the final call when results conflict. Engineering may find a candidate cheaper but less reliable; product may consider a small quality improvement worth a higher bill. Resolve that disagreement against the written objective and constraints. If the constraints need to change, record that as a new decision before evaluating a new run. Use a short brief to make the agreement reviewable:
You are ready to proceed when the owner can explain how a candidate passes or fails. Move to Step 2: know what you’re spending on to connect these decisions to actual usage.