​

Introduction to custom benchmarks

1. Send example prompts to a state-of-the-art model to establish a baseline with a Jupyter notebook.
2. Use Narev Cloud to find a model that performs like the baseline with lower cost and better latency.
3. Deploy the optimal model to production.

​

Why not use an LLM leaderboard instead?

• Academic benchmarks are a poor approximation of your app. The leaderboards show scores that LLMs get on standardized tests. The promise is that strong benchmark scores transfer to production, but this often fails. A 3-year-old model can match a current state-of-the-art model despite drastically different benchmark results.
• Academic benchmarks aren’t repeatable. Benchmark results vary depending on who runs them. For example, GPT-4’s score on the MMLU-Pro benchmark ranges from 72.6% to 85.7% depending on the leaderboard.
• People game academic benchmarks. For example, a major study by Singh et al. (2025) describes how participants manipulated Chatbot Arena. Data contamination runs rampant when models train on benchmark questions. Read more about it in our docs on benchmarking.

​

Before you begin

• A Narev account (sign up here) and a Narev API key (get one here)
• Python 3.8+ installed locally or access to Google Colab/Kaggle
• 50-100 example prompts representative of your production use case
• Basic familiarity with Python and Jupyter notebooks
• ~30 minutes to complete the setup and benchmarking

​

Setting up your benchmark

​

Part 1: Create an A/B test endpoint on Narev platform

https://www.narev.ai/api/applications/<YOUR_TEST_ID>/v1

​

Part 2: Open and prepare the environment

# Install required packages
import pandas as pd
import requests
from datasets import load_dataset
from kaggle_secrets import UserSecretsClient # or from google.colab import userdata

# Secrets
user_secrets = UserSecretsClient()
NAREV_API_KEY = user_secrets.get_secret("narev-api-key") # or userdata.get('narev-api-key')
NAREV_BASE_URL = "https://www.narev.ai/api/applications/cmj6nt7g40002i1qymlqvct4z/v1" # TODO: replace with your URL from Part 1

# Variables
SYSTEM_PROMPT="""Solve the following math problem. Only include the number in your answer.
Give your final as a number.
Do not include any reasoning"""

STATE_OF_ART_MODEL="openrouter:google/gemini-3-pro-preview"

dataset = load_dataset("openai/gsm8k", "main")
df_train = pd.DataFrame(dataset['train'])
df_test = pd.DataFrame(dataset['test'])

# take the first 50 prompts from the test
df_train = df_train.head(50)

# prepare expected output
df_train['expected_output'] = df_train['answer'].str.extract(r'####\s*(-?\d+(?:,\d+)*(?:\.\d+)?)')
df_train['expected_output'] = df_train['expected_output'].str.replace(',', '', regex=False).astype(float)

print(f"First question:\n\n{df_train.iloc[0]['question']}")

print(f"Expected output:\n\n{df_train.iloc[0]['expected_output']}")

def send_to_llm_with_expected_output(question: str, expected_output: str) -> dict:
    messages = []
    if SYSTEM_PROMPT:
        messages.append({"role": "system", "content": SYSTEM_PROMPT})

    messages.append({"role": "user", "content": question})

    try:
        response = requests.post(
            url=f"{NAREV_BASE_URL}/chat/completions",
            headers={
                "Authorization": f"Bearer {NAREV_API_KEY}",
                "Content-Type": "application/json"
            },
            json={
                "model": STATE_OF_ART_MODEL,
                "messages": messages,
                "metadata": {"expected_output": expected_output}
            },
            timeout=60
        )

        response.raise_for_status()
        data = response.json()
        return data
    except Exception as e:
        print(e)
        return {}

​

Part 3: Send the requests to Narev

for index, row in df_train.iterrows():
    question = row['question']
    expected_output = str(row['expected_output'])
    print(f"Processing question: {question}\n")
    print(f"With expected output: {expected_output}\n")

    response = send_to_llm_with_expected_output(question, expected_output)

    if response.get('choices'):
        print("Success\n\n")
    else:
        print("Failed to send a request\n\n")

​

Optimization in the Narev platform

​

Part 1: Select test Variants

• Gemini 3 Pro Preview - our first model, tagged as state of the art
• DeepSeek v3.2 Speciale - flagship model from the DeepSeek team
• GPT-5 Nano - recommended by Narev as a competitor performing in the benchmark similarly to the DeepSeek’s model
• Llama 3 70B Instruct - great open source model that I tried in many experiments in the past
• GPT OSS 120B - efficient open source model from OpenAI

​

Part 2: Select the Source of Truth model and the Quality Metric

​

Part 3: Run the experiment

​

Analyzing the results

​

Key insights from the benchmark

1. Perfect quality at 99.5% cost savings: GPT OSS 120B matches the baseline’s 100% quality while costing only $31.80 per million requests—saving$6,286.80 compared to Gemini 3 Pro Preview
2. Acceptable latency tradeoff: GPT OSS 120B is 72.2% slower than baseline (2,781ms vs 10,014ms), but still delivers responses in under 3 seconds, which is acceptable for most applications
3. GPT-5 Nano offers balanced performance: At $134.69/1M with 93.1% quality and 32.8% faster latency, it provides a good middle ground for latency-sensitive applications
4. Task alignment matters: Both DeepSeek v3.2 Speciale (46.4%) and Llama 3 70B Instruct (31.0%) showed poor performance on math reasoning despite being fast and cheap, demonstrating that model capabilities must match task complexity

• Quality - GPT OSS 120B matches the baseline quality at 100%, meaning we found an equally capable model at a fraction of the cost
• Cost - GPT OSS 120B provides the best cost efficiency at $31.80 per million requests—a 99.5% reduction while maintaining perfect quality
• Latency - For applications requiring the fastest response times with high quality, GPT-5 Nano offers 32.8% faster responses while maintaining 93.1% quality

​

Deploying to production

1. In the Narev dashboard, set your chosen variant (e.g., DeepSeek v3.2 Speciale) as the production variant
2. Continue using the same endpoint from Part 1: https://www.narev.ai/api/applications/<YOUR_TEST_ID>/v1
3. When you don’t specify a model in your requests, Narev automatically routes to your production variant

# No model specified - automatically uses your production variant
response = requests.post(
    url=f"{NAREV_BASE_URL}/chat/completions",
    headers={"Authorization": f"Bearer {NAREV_API_KEY}"},
    json={
        "messages": [{"role": "user", "content": "What is 25 * 17?"}]
    }
)

​

Resources

• Narev Platform Documentation
• A/B testing best practices
• Data Source Options
• Understanding evaluation metrics
• GSM8K Benchmark Dataset

​

Next steps

• Try with your own prompts and use case
• Experiment with different system prompts to optimize further
• Explore prompt optimization techniques