Tok/s vs cost benchmarks for running open models yourself, across GPUs and inference engines.

• gpt-oss-120b — OpenAI gpt-oss-120b reasoning model, native MXFP4 (~65GB). Fits a single H100 80GB. (6 setups)
• Step-3.7-Flash — StepFun Step-3.7-Flash — hybrid SWA/global-attention thinking model with MTP speculative decoding. Requires a full 8-GPU node (TP=8). (2 setups)
• Ling-3.0-flash — inclusionAI Ling-3.0-flash — 124B total / 5.1B active MoE, hybrid Mamba+MLA attention, 256K context. Requires a vendor vLLM fork (inclusionAI/vllm, branch ling_3_0). (3 setups)
• TinyLlama 1.1B Chat — TinyLlama 1.1B Chat — lightweight model for high-concurrency load testing on modest GPUs. (1 setup)