Setting up Variants

Variants define your execution configuration—model, system prompt, and parameters. When setting up an A/B test, you can either choose from existing variants or create a new one.

What is a Variant?

A variant is a complete model configuration that includes:

• Model: The AI model to use (e.g., GPT-4, Claude, Llama)
• System Prompt: Instructions that guide the model's behavior (optional)
• Parameters: Fine-tuning controls like temperature, max_tokens, top_p, etc.

By defining multiple variants with different configurations, you can systematically compare their performance on the same dataset.

Choosing Existing Variants

When configuring an A/B test, you can select from your library of existing variants. This is useful when:

• You've already created variants for similar use cases
• You want to reuse a proven configuration across multiple experiments
• You're testing the same variant against new data sources

To choose an existing variant:

1. Navigate to your A/B test's Setup
2. Go to the Variants section
3. Click Add Variant and select from your existing variants
4. Choose at least 2 variants to enable comparison

Creating New Variants

You have two approaches to create new variants: from scratch or from a template.

Creating from Scratch

For complete control over your variant configuration:

1. Navigate to the Variants section in the sidebar
2. Click Create
3. Configure each section:

1. Basic Info

Give your variant a descriptive name to identify it later:

Name: "GPT-4 High Creativity"

2. Model Selection

Choose the AI model to use. You can search through available models from different providers:

• OpenAI (GPT-4, GPT-3.5)
• Anthropic (Claude 3.5, Claude 3)
• Open-source models (Llama, Mistral, etc.)

The model selection affects cost, latency, and capabilities.

3. System Prompt

Define the context and behavior for the AI model:

You are a helpful assistant specialized in customer support.
Always be polite, concise, and solution-oriented.
If you don't know something, admit it rather than guessing.

4. Parameters

Fine-tune the model's behavior with parameters:

Common Parameters

• Temperature (0.0 - 2.0): Controls randomness. Lower values make output more focused and deterministic, higher values make it more random and creative
• Max Tokens: Maximum number of tokens in the response. Controls the length of the output
• Top P (0.0 - 1.0): Nucleus sampling parameter. Lower values make output more focused by sampling from a smaller set of likely tokens
• Top K: Limits sampling to the K most likely next tokens. Helps control output diversity
• Frequency Penalty (-2.0 to 2.0): Penalizes tokens based on their frequency in the text so far. Positive values reduce repetition
• Presence Penalty (-2.0 to 2.0): Penalizes tokens that have appeared in the text so far, encouraging new topics
• Repetition Penalty (0.0 - 2.0): Penalizes repeated tokens to reduce redundancy in output

Advanced Parameters

• Min P (0.0 - 1.0): Minimum probability threshold for token selection. Alternative approach to controlling randomness
• Top A (0.0 - 1.0): Alternative sampling method for controlling output diversity
• Seed: Random seed for deterministic generation. Same seed with same inputs produces identical outputs
• Stop: Up to 4 sequences where generation will stop. Useful for controlling output format
• Response Format: Format of the response (e.g., {"type": "json_object"} for structured JSON output)
• Logprobs: Whether to return log probabilities of tokens. Useful for understanding model confidence
• Top Logprobs (0-20): Number of top log probabilities to return per token

Creating from Templates

For faster setup, use pre-built templates optimized for common use cases:

1. Navigate to the Variants section
2. Click Use Template
3. Browse two categories:

Common Use Cases

Pre-configured variants for typical scenarios:

• Content Generation: Optimized for blog posts, articles, and creative writing
• Code Assistant: Tuned for programming tasks and technical documentation
• Customer Support: Configured for helpful, professional customer interactions
• Data Analysis: Set up for processing and explaining data insights
• And more...

Brand Templates

Variants inspired by popular AI products:

• Lovable: Full-stack development assistant (React, Next.js, TypeScript)
• Base44: Backend-focused (API design, database architecture)
• Copilot: General-purpose coding assistant
• v0: UI/UX specialist for interface design

Each template includes:

• Pre-selected model
• Optimized system prompt
• Recommended parameter values

1. Attach them to an A/B test to start running experiments
2. Configure quality metrics to automatically evaluate outputs
3. Analyze the results to determine which variant performs best