Know your objective and who's in charge

Understand the tradeoffs

Everything is a tradeoff. Speed vs. accuracy. Cost vs. quality. Latency vs. intelligence.

LLM providers want you to believe their flagship models are one-size-fits-all solutions. They're not.

Think about cars. You wouldn't use the same car for every job:

You wouldn't drive a forklift to the grocery store. You wouldn't use a city car to traverse a mountain trail. The job determines the car you need.

LLMs work the same way. Different models for different tasks:

• Simple queries? Use a lightweight model (the "city car")
• Complex reasoning? Premium model justified (the "sports car")
• Batch processing? Optimize for throughput (the "bus")
• Real-time interaction? Optimize for speed (the "motorcycle")

Yet most teams default to the "sports car" for everything.

One size doesn't fit all

LLM providers invest heavily in marketing their flagship models as universal solutions. They showcase impressive benchmark scores (MMLU, HumanEval, GPQA) that measure "intelligence" in controlled environments.

But here's what those benchmarks ignore:

• Cost: Premium models can be 100x more expensive
• Latency: Premium models can be 2-3x slower—users abandon apps while waiting for the "perfect" answer
• Overkill: Most queries don't need maximum intelligence

The newest model always costs the same

Yes, GPT-3.5 is now 10x cheaper than it was at launch. However, the price of the frontier models stay constant.

And the best model? It always costs roughly the same—around $15-75 per million tokens—because that's what the edge of compute costs today.

The frontier model price is remarkably stable. What drops in price is yesterday's newspaper—models that are no longer state-of-the-art.

Hidden inflation no one talks about

Modern models consume exponentially more tokens through:

• Longer context windows (128K → 200K → 1M tokens)
• Test-time compute (models that "think" longer use more tokens)
• Agentic workflows (models that iterate, check their work, and refine)

A simple query that used to return 500 tokens might now return 50,000 tokens as the model plans, researches, writes, and refines. The per-token cost stays flat, but you're burning 100x more tokens.

Real-world example

Notice that GPT-5 has lower token prices than GPT-4 Turbo ($1.25/$10 vs $10/$30), yet it costs 80% more per request ($1,526 vs $850). Why? Because GPT-5 uses 7x more output tokens (150 vs 21).

Define your objective to know the tradeoff

So what are you actually optimizing for? You can't know until you ask the right people.

Assemble your the people who care about the outcome:

1. Product Manager - owns user experience and conversion metrics
2. Engineering Lead - owns performance, reliability, and technical architecture
3. Data Scientist/ML Engineer - owns model quality and evaluation
4. Finance/Operations - owns budget and unit economics
5. Executive Leadership - owns strategic priorities and resource allocation

Gather them and ask: What matters most to us?

Common answers:

• Conversion to paid features
• User retention and engagement
• Customer support ticket reduction
• Response speed (time to first token)
• Cost per interaction or per user
• Accuracy on mission-critical tasks
• Throughput (queries per second)

You can't optimize for everything. That's the point. Pick your top 2-3 priorities and accept the tradeoffs.

Get everyone aligned

For small teams, alignment might happen in a 30-minute conversation. For larger organizations, you need explicit buy-in. In a bigger team or complex product, decide who makes the final call when there's conflict? The product manager? Head of engineering? CEO? You?

Bring this person on the journey from day one. Keep them involved. When you need to choose between 10% better accuracy or 50% lower costs, they make the call.

Once you have clear answers, move to Step 2: Know what you're spending on.