Methodology

Frontier models are getting surprisingly good at writing AI accelerator kernels — but no single model consistently dominates. The performance landscape is the cross product of hardware target, kernel type, precision format, tensor shapes, and more — a space large enough that vibes-based evaluation misses most of it. Proper measurement requires fault-tolerant agent harnesses, access to real hardware, and benchmarking strict enough to catch the many creative ways models reward-hack their way to inflated scores. Individual researchers shouldn't have to shoulder all of that just to pick a model.

KernelArena evaluates how well frontier language models can generate high-performance kernels. Each task specifies exact shapes, dimensions, data types, and operator configurations — the kind of narrow, hardware-specific detail that hand-tuned kernels are written for today. We believe LLMs will soon generate specialized kernels for any given configuration on demand, and KernelArena is built to measure progress toward that future.

We evaluate both the model and the harness it runs in. Each benchmark suite specifies its own harness and hardware target — the details are documented per suite below.

When we discover reward hacks — cases where models exploit the evaluation to inflate scores without genuine performance gains — we document and expose them publicly. Honest benchmarks require honest failure modes.

We're building toward a fully open-source evaluation ecosystem where the community can inspect, swap, and contribute entire harnesses. We plan to add support for additional harnesses, models, and hardware targets so results reflect the broadest possible picture of model capability.