Skip to main content
Switching models can reduce inference spend when a less expensive model meets your task’s requirements. This guide uses email routing to explain how to investigate that possibility. It separates an illustrative workflow from historical results retained in this repository, whose underlying evaluation data is not available here.

​
The scenario

Consider a support system that sends technical issues to engineering, payment questions to billing, and purchase inquiries to sales. The model needs to select a category and department reliably. General reasoning capability alone does not tell you which model can perform that task at the lowest cost. Begin with a concrete hypothesis: a cheaper model may preserve routing quality while changing cost and latency. Define how to handle mixed-topic messages and when to escalate to a person. Then assemble representative labeled emails, including ambiguous cases. These are steps for your evaluation, not a reconstruction of the historical experiment below.

​
Compare token rates

Use price search to identify candidates and their serving providers. For example, search gpt-4 for the baseline family and gpt-oss-20b for a candidate. Keep exact model/provider pairs, because broad queries return related models and several routes.
Inspect all result pages and compare pricing.prompt and pricing.completion in USD per token. Do not assume that the company named in a model family serves that model through its own API. A host returned by the catalog must also be one you can access and intend to use.

​
Price a representative request

For an initial rate comparison, price an illustrative request with 450 prompt tokens and 25 completion tokens. The baseline below uses GPT-4 on OpenAI; confirm that pair in the catalog before running it.
Repeat with a selected candidate pair and the same counts. This isolates rates, not workload behavior. In an actual routing test, record each model’s usage because tokenization and output length can differ. Measure accuracy and latency separately; the Pricing API does not run the task or supply those measurements.

​
Results

The previous version of this guide reported the following results. They are historical reported values, not current prices or independently reproduced benchmarks. The repository does not provide the complete dataset, sample count, scoring rubric, run date, model settings, or provider configuration needed to reproduce them. Costs are reported averages scaled to one million requests, not evidence that one million requests were executed. These numbers support a narrower interpretation than a universal model recommendation. GPT-4.1 nano had lower reported accuracy. GPT-5 nano matched the reported baseline accuracy but had substantially higher latency. It is therefore incorrect to say that both nano variants lost accuracy. Relative to the reported baseline, gpt-oss-20b’s cost was about 99.6% lower, while average latency was 954 ms higher. GPT-5 nano also met the reported accuracy result, at a higher cost and latency than gpt-oss-20b. Whether those differences matter depends on your task’s limits. Without the sample count and error distribution, 100% in this table cannot establish a production accuracy rate. The fixed 450/25-token example above does not reproduce these averages. Do not combine its token counts with the historical table to infer a methodology or explain unexplained cost differences. Use current rates and measured counts in your own evaluation.

​
Example in action

An email asking for enterprise plan pricing, volume discounts, and a demo is a plausible sales case. The historical guide reported that every listed model routed that example correctly. A shared success on an obvious case does little to distinguish candidates; the mixed billing-and-sales request or technical complaint mentioning a refund is more informative. Compare failures by category and inspect invalid outputs as well as wrong labels. If your application retries malformed JSON or falls back to another model, include those calls in the candidate’s cost. A low first-call price is not enough when downstream repairs are frequent.

​
The takeaway

Use the historical comparison to form hypotheses, then choose a model using your own evidence. Evaluate a held-out set, record actual token usage and serving latency, and compare cost per successful routing outcome. An asynchronous queue may tolerate a slower candidate that an interactive chat cannot. Follow FinOps step 3 for a gradual rollout with quality and latency guardrails. Keep the baseline available until production measurements confirm the change. If input verbosity remains a major cost afterward, evaluate prompt engineering as a separate experiment.