> ## Documentation Index
> Fetch the complete documentation index at: https://narev.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmark LLMs programmatically with API

> Use the Narev Cloud Applications API to run benchmark A/B tests programmatically, send requests through OpenAI-compatible endpoints, and pull results.

The Applications API provides an OpenAI-compatible endpoint that enables automatic benchmarking through your production code. Make requests with different configurations, and Narev automatically tracks and compares performance.

## Quick start

Replace your OpenAI base URL with your Narev A/B test endpoint:

```python theme={null}
from openai import OpenAI

client = OpenAI(
    api_key="YOUR_NAREV_API_KEY",
    base_url="https://narev.ai/api/applications/{benchmark_id}/v1"
)

response = client.chat.completions.create(
    model="openai:gpt-4",
    messages=[
        {"role": "user", "content": "What is the capital of France?"}
    ]
)
```

That's it! Narev now will add every prompt sent to this endpoint to your benchmark.

## Selecting a model provider or gateway

Narev supports multiple AI providers/gateways through **gateway prefixes**:

```
{gateway}:{model_name}
```

**Available Gateways:**

* `openai` - OpenAI direct
* `openrouter` - OpenRouter aggregator
* `nvidia` - NVIDIA NIM models
* `kilo` - Kilo Code models
* `github` - Github models

**Examples:**

* `openai:gpt-4` - OpenAI's GPT-4
* `openrouter:anthropic/claude-3-opus` - Claude via OpenRouter
* `anthropic:claude-3-sonnet-20240229` - Direct Anthropic
* `openrouter:meta-llama/llama-3.1-70b-instruct` - Llama via OpenRouter

The same model accessed through different gateways is treated as a **separate variant**. This lets you compare:

* **Latency** - Which gateway is faster?
* **Cost** - Which is more economical?
* **Reliability** - Which has better uptime?

```python theme={null}
# Test GPT-4 via OpenAI
response1 = client.chat.completions.create(
    model="openai:gpt-4",
    messages=[{"role": "user", "content": prompt}]
)

# Test GPT-4 via OpenRouter
response2 = client.chat.completions.create(
    model="openrouter:openai/gpt-4",
    messages=[{"role": "user", "content": prompt}]
)
```

<Note>
  **Default behavior:** If you omit the gateway prefix (for example, `gpt-4`), Narev defaults to the native provider.
  We recommend always using explicit gateway prefixes for clarity.
</Note>

<Tip>
  Include expected outputs for a subset of your production requests to continuously monitor quality and ensure model
  changes do not harm accuracy.
</Tip>

You can also include custom metadata for filtering and analysis:

```python theme={null}
response = client.chat.completions.create(
    model="openai:gpt-4",
    messages=[{"role": "user", "content": prompt}],
    extra_body={
        "metadata": {
            "expected_output": "Expected answer here",
            "user_id": "user_123",
            "session_id": "session_456",
            "category": "customer_support"
        }
    }
)
```