Compare cost between two models

Compare models in two stages. First, hold token counts fixed to see the effect of their rates. Then run representative tasks on each model to find the actual cost of an acceptable answer. The first stage is a useful screening calculation; the second supports a production decision.

Set a fixed usage block

Choose counts that resemble your workload. For a short classification task, you might start with 450 prompt tokens and 25 completion tokens. Keep those counts identical across the initial price requests so that the result reflects only pricing differences. Find both model/provider pairs with price search. A model family name is not enough: a dated variant or gateway route can have a different identifier and price. The following Node.js script accepts two pairs as command-line arguments and prices both using the same counts. It requires Node.js 22 or later and makes no inference calls.

Price the baseline and candidate

Save this as compare-costs.mjs. It rejects unsuccessful or invalid cost responses so a missing price cannot appear as a saving.

const [baselineModel, baselineProvider, candidateModel, candidateProvider] =
  process.argv.slice(2);
if (![baselineModel, baselineProvider, candidateModel, candidateProvider].every(Boolean)) {
  throw new Error('Pass baseline model/provider and candidate model/provider');
}
const usage = { prompt_tokens: 450, completion_tokens: 25 };

async function price(model_id, provider_id) {
  const response = await fetch('https://api.narev.ai/v1/traces/cost', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ model_id, provider_id, usage }),
  });
  if (!response.ok) throw new Error(await response.text());
  const result = await response.json();
  const total = result.cost_breakdown?.total;
  if (!Number.isFinite(total) || total < 0) throw new Error('Invalid cost total');
  return total;
}

const baseline = await price(baselineModel, baselineProvider);
const candidate = await price(candidateModel, candidateProvider);
console.table({ baseline: { usd: baseline }, candidate: { usd: candidate } });
console.log('USD saved per fixed-usage request:', baseline - candidate);
if (baseline > 0) {
  console.log('Percent saved:', (1 - candidate / baseline) * 100);
}

For example, after confirming both pairs in the catalog, run:

node compare-costs.mjs gpt-4o openai gpt-4o-mini openai

Compare totals

A positive saving means the candidate costs less at those token counts. A negative saving means it costs more. If an illustrative baseline costs

0.010 and a candidate costs

0.004, the difference is $0.006, or 60%. Those figures demonstrate the formula; they are not model prices or measured savings. Multiplying the difference by monthly volume gives a planning estimate only if the assumed usage mix holds. Try both input-heavy and output-heavy cases if your workload varies. Include cache or other usage dimensions when relevant instead of assuming all requests use standard text rates.

Measure the actual workload

Run the same task inputs through each candidate using the prompt and settings you intend to deploy. Record each model’s own token counts. Even identical text can tokenize differently, and outputs may vary in length or include reasoning. Also count retries, invalid responses, and fallback calls: a cheap first attempt can become an expensive successful outcome. Evaluate quality against a held-out task set and measure latency in your serving environment. Compare total cost divided by successful outcomes, alongside failure rates and latency percentiles. Keep rate snapshots with the run so later price changes do not obscure whether an improvement came from the model or its price. The model-switch case study shows how to interpret a reported result with these limits in mind. Use FinOps step 3 to define rollout and rollback criteria before changing production traffic.

Estimate costs

Track usage and bill customers

Reduce AI costs

Billing concepts

Compare cost between two models

Set a fixed usage block

Price the baseline and candidate

Compare totals

Measure the actual workload

​Set a fixed usage block

​Price the baseline and candidate

​Compare totals

​Measure the actual workload

Set a fixed usage block

Price the baseline and candidate

Compare totals

Measure the actual workload