A cheaper model is useful only if it still completes the task. Evaluate changes to models, skills, and harnesses against successful outcomes, then compare the cost of doing the work.
IRL Gym gives you repeatable flight scenarios for evaluating agent behavior. A harness runs the agent, skills provide instructions, and the scenario defines its starting conditions and goal. Evaluation provides evidence for optimization; it does not guarantee lower costs.
Run your first evaluation
You need a GitHub account. The hosted service runs the simulator and agent; you do not need to install X-Plane, start an MCP server, or supply model API keys in the flight form.
- Open New flight and select Continue with GitHub.
- Keep Sydney to Canberra, Codex, and Default for this harness. The form supplies the pilot skills.
- Select Submit flight and follow the request in Flights.
- Once it finishes, review the recording and score. Treat this run as your baseline, then change a skill, model, or harness and compare results.
A completed run does not guarantee a successful landing. Inspect failures as well as scores, and use relevant spending measurements before claiming savings.
Learn more
Explore IRL Gym for scenarios and setup. Follow the practical FinOps framework to evaluate cost, quality, and latency together. Last modified on October 30, 2026
