Skip to main content

Documentation Index

Fetch the complete documentation index at: https://narev.ai/docs/llms.txt

Use this file to discover all available pages before exploring further.

The Applications API provides an OpenAI-compatible endpoint that enables automatic benchmarking through your production code. Make requests with different configurations, and Narev automatically tracks and compares performance.

​
Quick start

Replace your OpenAI base URL with your Narev A/B test endpoint: 
from openai import OpenAI

client = OpenAI(
    api_key="YOUR_NAREV_API_KEY",
    base_url="https://narev.ai/api/applications/{benchmark_id}/v1"
)

response = client.chat.completions.create(
    model="openai:gpt-4",
    messages=[
        {"role": "user", "content": "What is the capital of France?"}
    ]
)
That’s it! Narev now will add every prompt sent to this endpoint to your benchmark.

​
Selecting a model provider or gateway

Narev supports multiple AI providers/gateways through gateway prefixes: 
{gateway}:{model_name}
Available Gateways:
  • openai - OpenAI direct
  • openrouter - OpenRouter aggregator
  • nvidia - NVIDIA NIM models
  • kilo - Kilo Code models
  • github - Github models
Examples:
  • openai:gpt-4 - OpenAI’s GPT-4
  • openrouter:anthropic/claude-3-opus - Claude via OpenRouter
  • anthropic:claude-3-sonnet-20240229 - Direct Anthropic
  • openrouter:meta-llama/llama-3.1-70b-instruct - Llama via OpenRouter
The same model accessed through different gateways is treated as a separate variant. This lets you compare:
  • Latency - Which gateway is faster?
  • Cost - Which is more economical?
  • Reliability - Which has better uptime?
# Test GPT-4 via OpenAI
response1 = client.chat.completions.create(
    model="openai:gpt-4",
    messages=[{"role": "user", "content": prompt}]
)

# Test GPT-4 via OpenRouter
response2 = client.chat.completions.create(
    model="openrouter:openai/gpt-4",
    messages=[{"role": "user", "content": prompt}]
)
Default behavior: If you omit the gateway prefix (for example, gpt-4), Narev defaults to the native provider. We recommend always using explicit gateway prefixes for clarity.
Include expected outputs for a subset of your production requests to continuously monitor quality and ensure model changes do not harm accuracy.
You can also include custom metadata for filtering and analysis: 
response = client.chat.completions.create(
    model="openai:gpt-4",
    messages=[{"role": "user", "content": prompt}],
    extra_body={
        "metadata": {
            "expected_output": "Expected answer here",
            "user_id": "user_123",
            "session_id": "session_456",
            "category": "customer_support"
        }
    }
)