Introducing Vision Arena

At OnDeck we're building the best measurement systems for vision.

Vision Index was the first of many steps and establishes a baseline for evaluating vision models across video question-answering tasks. We'll continue to improve these leaderboards with new models, metrics, and better methodologies. But benchmarks alone won't get us to where we want to be: defining the frontier of vision across all meaningful dimensions.

The landscape of vision evaluations

There's no single way to measure how good a model is at vision. A model might perform well on video question answering but struggle with object detection, spatial reasoning, or following instructions across a long video. Even within the same task, results can vary depending on the dataset, how questions are asked, and how answers are scored.

Vision evaluations vary depending on what capabilities they test, how they're scored, and what specific weaknesses they're designed to uncover. Public benchmarks give us a common baseline, but may be trained on and not represent actual performance on unseen footage. Private and enterprise evals help measure performance on real world tasks.

The challenge is understanding what each evaluation tells us, where models consistently fall short, and which capabilities we're still not measuring well.

Vision Arena

Today, we're launching Vision Arena as our next step. As models become increasingly optimized for existing benchmarks, we need other ways to understand how well they perform. Arenas are pretty simple: two anonymous models answer the same visual question, and users vote on which response is better. We aggregate these votes using Bradley-Terry scoring to rank models based on human preferences.

Your question

How many banners are in the arena, and what color is each?

A

Model A

Four banners stand in the arena. From left to right: green, orange, blue and purple.

B

Model B

There are three banners in the arena: a green one on the left, a red one in the middle and a blue one on the right.

Which answer is better?

A is betterB is betterBoth goodBoth bad

Vision Arena brings human preferences into how we evaluate vision models, alongside the structured benchmarks in Vision Index. It's another step toward a broader evaluation system that can measure everything from model capabilities to failures in real-world applications. For enterprises testing vision models, we also offer private benchmarks and arenas for your data.

We're just getting started. Try Vision Arena, cast a few votes, and help us understand what today's vision models can actually do.