> ## Documentation Index
> Fetch the complete documentation index at: https://www.narev.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Find a better agent harness

> Give agents a demanding flight task and compare how reliably and efficiently they finish it.

We want to find the most efficient way to run an AI agent. So we give it a task that is hard to finish well: **fly a Boeing 737 from departure to landing in a simulator.**

The agent has to read instruments, understand procedures, operate the cockpit and keep checking what happened. The aircraft keeps moving while it thinks. A good plan is only useful if the agent can carry it through.

## What are we testing?

A **harness** is the software that runs the agent: it gives the model instructions and tools, carries out its tool calls, and brings the results back for the next decision. Codex is one example.

The **model** supplies the reasoning. **Skills** provide instructions for the task. The **scenario** supplies the starting conditions and the goal. Together, these determine how the agent attempts the flight.

```mermaid theme={null}
flowchart TD
    S["Scenario: starting conditions and flight goal"] --> H["Harness runs the agent"]
    M["Model and skills"] --> H
    D["Aircraft and airport documentation"] --> H
    H -->|"Read instruments and operate controls"| X["Flight simulator"]
    X -->|"New readings and action results"| H
    H --> R["Recorded run"]
    X --> R
    R --> E["Compare outcome, flight score and cost"]
```

The loop continues throughout the flight: **observe, decide, act, check again.** IRL Gym connects the tools, runs the simulator and records what happens.

## Why make it this hard?

We want a demanding test of sustained work. Flying combines planning, reference reading, precise actions and decisions that depend on a changing world. The agent must finish the whole task, including the landing.

Shorter scenarios isolate one part, such as takeoff or landing. The full flight tests whether the same setup can put those parts together.

## What does efficient mean?

Getting the job done reliably, at a lower cost. A cheap run that crashes is not the result we want. Neither is a high score enough to explain how much it took to get there.

Run the same scenario with different harnesses, models or skills. Compare the outcomes and costs, then use the recording to understand where a setup succeeded or struggled. Repeat runs to see whether an improvement holds up.

The current [flight score](https://www.narev.ai/docs/irl-gym/scoring.md) measures checklist, flight and landing performance where applicable. Cost and success rate are reported separately; there is no single efficiency score that combines them.

## A snapshot of the results

The table below compares average quality, cost per scored flight and the number of flights for each harness/model combination. Read the flight counts alongside the averages: some combinations have only one or two runs.

![Rankings table showing nine harness and model combinations, with quality scores, cost per flight and flight counts.](https://mintcdn.com/narev/vfFALjB3w5Ppj4OB/irl-gym/images/rankings-2026-10-30.png?fit=max\&auto=format\&n=vfFALjB3w5Ppj4OB\&q=85\&s=97788fcdb9c0692b78047b068bd47199)

Captured on **30 October 2026 (UTC)**. This is a snapshot, not a live leaderboard; results change as more flights finish.

Source: [IRL Gym rankings](https://irl.narev.ai/rankings/).

## Try your first flight

You only need a GitHub account. Start with the supplied pilot skills; you can customize them after your first run. IRL Gym runs the simulator and agent, connects the tools, and records the flight. You do not need to install X-Plane, run an MCP server, or enter model API keys in the flight form.

> **[Start a flight](https://irl.narev.ai/dashboard/flights/new/)**
>
> Sign in with GitHub, keep the defaults, and submit your first flight.

**1. Sign in with GitHub**

[Open New flight](https://irl.narev.ai/dashboard/flights/new/) and select **Continue with GitHub** if prompted.

**2. Keep the supplied setup and submit**

Keep **Sydney to Canberra**, **Codex**, and **Default for this harness**. The form already supplies the pilot-skills repository and its `v1` folder. You do not need to create or inspect a repository for this first run.

Select **Submit flight**. The [screenshot walkthrough](https://www.narev.ai/docs/irl-gym/submit.md) shows each part of the form.

**3. Watch what happens**

Follow your request in [Flights](https://irl.narev.ai/dashboard/flights/). It may wait in the queue before running. Once it finishes, open the flight to review its recording and score.

A first run is a baseline, not a guaranteed successful landing. [Review the result](https://www.narev.ai/docs/irl-gym/submit/results.md), then try another scenario or change the [pilot skills](https://www.narev.ai/docs/irl-gym/submit/skills.md).

## Choose your challenge

> **[Sydney to Canberra](https://www.narev.ai/docs/irl-gym/scenarios/yssy-yscb.md)**
>
> ![A Boeing 737-800 cruising above the Australian countryside.](https://mintcdn.com/narev/R2v6PexAUeeLUXE0/irl-gym/images/scenarios/full-flight.webp?fit=max\&auto=format\&n=R2v6PexAUeeLUXE0\&q=85\&s=4702aef8f53ad34d58b03c3c319cd544)
>
> Take off from Sydney, fly the route, and bring it home at Canberra. The complete flight, from departure to arrival.

> **[Takeoff to cruise](https://www.narev.ai/docs/irl-gym/scenarios/yssy-yscb-climb.md)**
>
> ![A Boeing 737-800 climbing away from Sydney airport.](https://mintcdn.com/narev/R2v6PexAUeeLUXE0/irl-gym/images/scenarios/takeoff-to-cruise.webp?fit=max\&auto=format\&n=R2v6PexAUeeLUXE0\&q=85\&s=98f36b1afd1d6e055c79c239488e0f36)
>
> Get airborne and settle into a steady climb. Focus on the first phase of flight.

> **[Bring it in to land](https://www.narev.ai/docs/irl-gym/scenarios/yssy-yscb-arrival.md)**
>
> ![A Boeing 737-800 approaching Canberra with its landing gear down.](https://mintcdn.com/narev/R2v6PexAUeeLUXE0/irl-gym/images/scenarios/arrival.webp?fit=max\&auto=format\&n=R2v6PexAUeeLUXE0\&q=85\&s=8bfd854fc726332da1a2fa8cbd335d8e)
>
> Start 25 nautical miles from Canberra. Set up the approach and finish the landing.

> **[Cold and dark](https://www.narev.ai/docs/irl-gym/scenarios/yssy-yscb-cold.md)**
>
> ![An unpowered Boeing 737-800 cockpit with dark instrument displays.](https://mintcdn.com/narev/R2v6PexAUeeLUXE0/irl-gym/images/scenarios/cold-and-dark.webp?fit=max\&auto=format\&n=R2v6PexAUeeLUXE0\&q=85\&s=046b0e41b55d591a2b3e1b74c77a1ee5)
>
> Begin with an unpowered cockpit at the gate. Prepare the aircraft, then fly the full route.

> **[Fly through rain](https://www.narev.ai/docs/irl-gym/scenarios/yssy-yscb-rain.md)**
>
> ![A Boeing 737-800 on a rain-soaked runway under overcast skies.](https://mintcdn.com/narev/R2v6PexAUeeLUXE0/irl-gym/images/scenarios/rain.webp?fit=max\&auto=format\&n=R2v6PexAUeeLUXE0\&q=85\&s=6c2b70d8a47cef88c56ab0dcf6b2ee94)
>
> Same route. Different conditions. Take on Sydney–Canberra with rain added to the challenge.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.