cto bench – Benchmarking AI agents using real user tasks

Summary: cto bench benchmarks AI agents by measuring their performance on real tasks from cto.new users instead of hypothetical problems, providing practical data on model effectiveness in actual workflows.

What it does

It evaluates AI models based on real usage patterns and PR merge rates from cto.new, rather than custom test suites or imagined challenges.

Who it's for

Developers and researchers seeking practical benchmarks of AI agents aligned with real-world coding tasks.

Why it matters

It addresses the gap between theoretical benchmarks and actual agent performance on meaningful work queues.