Introducing Vision Arena
Our next step in measuring vision models for useful tasks
OnDeck AI, Justin Chan · November 3, 2026
Methodology
Benchmark Selection, Indexes, and Evaluation Protocol
Roadmap
Open benchmarks and indexes
- Now
Arena and ELO-based scores
Private benchmarks, real-world task simulation
Agent harnesses, world models, VLAs
We're starting with open benchmark evaluations as a baseline for visual intelligence. Each next step will fill a gap in current vision measurement. Ultimately, we seek to define the standard for measuring visual intelligence and push its frontier towards economically meaningful performance.
Thesis
Vision Index exists because there are no leaderboards or arenas that adequately evaluate vision model performance on real-world video understanding tasks.
LLMs have demonstrated success in automating open ended text/code based tasks. As they've become multimodal, attention has shifted to visual intelligence and physical AI. However, evaluation has not kept up.
It's a bit of a chicken and the egg problem: video is expensive and hard, so models and inference infrastructure do not prioritize it well. But because it's expensive and hard, it's increasingly necessary to be able to robustly evaluate these models on representative video understanding tasks.
For more on why we're building this, read the launch post.
Index aggregation and scoring
Indexes distill a model's performance into a single score, simplifying comparisons across benchmarks, categories and scoring methods. We still publish the benchmark scores for transparency and model comparison. We will soon also provide per-question results for added visibility and benchmark transparency.
The Visual Intelligence Index
The Visual Intelligence Index is our headline metric: one score that summarizes overall model performance on video understanding. It is comprised of six capability indexes that aggregate cross-benchmark performance.
(1)
We weigh capabilities based on our judgement of their importance for video understanding. New index versions may adjust these weights.
Capability indexes
To construct each capability index, we classify every question in all 7 benchmarks into one or more of the six capabilities below. A model's index score is a weighted mean across that particular suite of questions.
| Capability | What it asks |
|---|---|
| Perception | Asks to identify, count, read, compare, or determine a directly observable property of an entity or visual element. |
| Spatial Reasoning | Asks about position, direction, distance, layout, containment, relative location, or other spatial relationships. |
| Temporal Reasoning | Asks about time, order, duration, frequency, sequence, before/after relationships, or changes across time. |
| Action / Event Understanding | Asks what someone or something is doing, what action occurs, what event takes place, or what overall activity is happening. |
| Causal Reasoning | Asks why something happens, what causes or results from an event, what someone intends or is trying to accomplish, or requires commonsense inference about motivations or consequences. |
| Information Retrieval | Asks for information that must be located or combined across multiple moments, entities, events, or details in the video. |
Since different benchmarks have different numbers of questions, we weigh each benchmark's contribution to a capability by how central the capability is to that suite, adjusted by the absolute number of relevant questions.
(2)
Centrality is the share of a benchmark’s questions labelled for the capability, n is how many there are, and K = 150.
| Benchmark | Perception | Spatial Reasoning | Temporal Reasoning | Action / Event Understanding | Causal Reasoning | Information Retrieval |
|---|---|---|---|---|---|---|
| Video-MME v2 | 19.5% | 24.2% | 24.0% | 15.3% | 21.4% | 25.9% |
| LVBench | 19.8% | 18.6% | 19.6% | 14.6% | 8.0% | 19.4% |
| Perception Test | 16.0% | 22.2% | 16.1% | 12.1% | 5.2% | 14.3% |
| NExT-QA | 19.3% | 31.7% | 22.0% | 18.8% | 33.1% | 6.2% |
| Q-Bench Video | 20.7% | 2.8% | 5.1% | 1.9% | 1.6% | 10.0% |
| EgoSchema | 4.7% | 0.5% | 13.2% | 16.5% | 30.7% | 24.1% |
| UCF101-AD | — | — | — | 20.7% | — | — |
(3)
Evaluation methodology
We run all inference and evaluation ourselves. In the current vision research landscape, papers publish scores but they aren’t comparable or interpretable. They differ in many dimensions, including dataset split, frame budget, subtitle access and scoring code. Those cause differences that are often bigger than the gap between the models, preventing even less precise relative comparisons.
Our objective is to control the variables and increase visibility into the process. Wherever possible, we standardize framerate, prompting, scoring, model executions, and data splits within benchmarks and between models. Deviations from standard are explicitly noted on this page and wherever relevant on the Index.
For 3rd party hosted models, we respect the standards set by their systems. For self-hosted models, we push standardization even further since we host and manage the inference.
- Inputs
- Models that take video natively receive video. The image-based models get frames we decode. Our baseline is sampling every video at 1 frame per second (FPS) to remain fair across models. However, due to provider context limits, some models are capped. When that context limit is lower than a video's frames sampled at 1 FPS, we evenly space the maximum allowed frames across the whole clip. This results in a lower effective framerate. The caps are listed in the Per-model protocol table.
For example, if a video is 200 seconds and thus 200 frames at 1 FPS, but the provider only takes 100 frames, we will sample at the provider's limit of 100 frames, making the effective framerate 0.5 FPS.
When models support native video input, we provide the video as is, including the original audio. When models do not support native video input, there is no way to provide the audio. - Inference
- We leave sampling parameters at each provider’s defaults rather than tuning them per model, unless necessary to accomodate a datasets video format or content. No transcripts, no retrieval tools, and the prompt is fixed per benchmark. When a request fails, it is retried up to a fixed number of times, and is otherwise reported as missing coverage and marked as wrong. For content refusals, we also retry, and also re-encode the video. If refusals are deterministic, they are reported as missing coverage and marked as wrong.
- Scoring
- We don’t use the scoring methods used by benchmark authors. Most of them expect a bare option letter, and mark a correct answer wrong if it comes back with incorrect formatting, or as the restated option, or as a letter buried in an explanation. Our standard inference protocols are not intended to measure instruction-following (although we will do this separately in the future). Note that this affects some models more than others.
Instead we use our own multiple-choice evaluator: it reads the response, works out which option the model picked, and checks it against the answer key. Same evaluator for every model on every multiple-choice benchmark, so a formatting habit doesn’t get scored as a capability. - Coverage
- Every score also lists how much of the benchmark it covers. If a provider's minimum safety filter refuses to process an input, this is reported and the response is scored as incorrect.
Per-benchmark protocol
The table below lists the metric, evaluation split, and any benchmark-specific details that affect the scores.
| Benchmark | Primary metric | Notes |
|---|---|---|
| Video-MME v2 | Accuracy, no subtitles | We run this without subtitles to test what models can learn from the video alone. Videos last up to 35 minutes and exceed most models’ frame limits. When that happens, we sample fewer frames, evenly spaced across the full video. Each score reports the frame limit and how many videos exceeded it. |
| LVBench | Test accuracy | The median video is an hour long, and maxes out at two hours. Most videos exceed every model’s frame limit, so all models receive fewer than one frame per second, sampled evenly across the full video. We report each model’s limit beside its score. Differences in how many frames models receive matter when comparing these results. |
| Perception Test | Overall accuracy | We use the 454-question validation split from 2025. All clips fit within every model’s frame limit at one frame per second, so these scores can be compared without accounting for different sampling rates. |
| NExT-QA | Accuracy (hard split) | We use the multiple-choice questions from the 570-video validation split. We exclude the open-ended questions because they require a judge and a different scoring method. With an average of nine questions per video, this benchmark benefits most from prompt caching. |
| Q-Bench Video | Overall accuracy | We use the benchmark’s own evaluator on the combined dev and test splits. Q-Bench-Video has no dedicated multiple-choice answer extractor, so its answers are scored differently from those of the other six benchmarks. Keep that difference in mind when comparing scores across benchmarks. |
| EgoSchema | Accuracy (fullset) | We use the 500-question scored subset. Each video is exactly three minutes long and has one question. There are no repeat questions to offset the cost of caching a video, making this the most expensive benchmark per question for providers that charge for cache writes. |
| UCF101-AD | Accuracy | Each pair of questions asks the model to identify an action that occurs and reject a plausible action that does not. We report overall accuracy across both question types, rather than their harmonic mean. All models score close to perfect on actions that occur; most of the ranking differences come from correctly rejecting actions that do not. |
Per-model protocol
Different providers implement image and video transport and processing differently. We leave these differences in our results, rather than knee-capping a model's performance to standardize the protocol because another doesn't have as much video infrastructure. This means our results measure actual inference processes that you would use for real tasks, not a perfectly equal evaluation environment.
Hosted models were called at each provider’s public endpoint from Virginia, USA. Self-hosted models ran under vLLM in bfloat16 on NVIDIA A100-SXM4-80GB GPUs.
| Model | Sent | Frame ceiling | Endpoint | Region | Notes |
|---|---|---|---|---|---|
| Gemini 4 Argon | Video, 1 fps | 5,000 | Vertex AI | Global | Workaround for a caching issue: each request adds a placeholder model turn instead of sending a single user part. |
| GPT-6 Astra | Frames, JPEG | 1,500 | OpenAI API | Not exposed | JPEG at quality 85 to meet API image size limits. |
| Gemini 3.8 Flash | Video, 1 fps | 5,000 | Vertex AI | Global | Default media resolution. |
| Claude Fable 5.1 | Frames, JPEG | 550 | Anthropic API | Not exposed | JPEG at quality 85 to meet API image size limits. |
| Qwen3.8-Max | Video, 1 fps | 768 | Qwen Cloud (DashScope) | International (Singapore) | A content filter refuses a significant portion of videos deterministically; those items are marked as wrong and the reduced count is published. |
| Kimi K3 | Frames, JPEG | 250 | Qwen Cloud (DashScope) | International (Singapore) | — |
| Qwen3.8-27B | Frames | 768 | Self-hosted, vLLM: 2× A100 80GB, 262,144-token context | — | — |
| Qwen3.5-35B-A3B | Frames | 768 | Self-hosted, vLLM: 2× A100 80GB, 262,144-token context | — | — |
| Nemotron 3 Nano Omni 30B-A3B | Frames | 727 | Self-hosted, vLLM: 2× A100 80GB, 131,072-token context | — | — |
| Molmo 2 8B | Frames | 338 | Self-hosted, vLLM: 1× A100 80GB, 36,864-token context | — | Smallest model in the field, at 8B parameters. |
1 FPS is used until the frame ceiling is reached, after which we sample at the maximum rate the frame ceiling allows. This is recorded in the score’s setting. On the longest benchmarks nearly every video is restricted by the frame ceiling.
Model availability
Technical issues can delay a model’s evaluation. We document those blockers here while results are pending.
| Model | Notes |
|---|---|
| We are currently expanding our Visual Intelligence Index to evaluate video agents and broader video systems alongside traditional VLMs. |
Cost, latency and tokens
Video understanding tasks often require uploading and processing large video files. Hosted model providers can vary in their implementation of image and video processing, and cloud endpoints may have changing latencies and performance. Local model performance also changes based on hardware, serving framework, and configuration. We provide a real-world measurement of the cost and speed of running vision models on such tasks, testing the actual limits we observe in providers.
What a task is
A task is one question about one video. The model receives the video, or frames from it, and a single question, and answers.
Some benchmarks ask several questions about the same video. We send the video again with each one, so every question is answered on its own and in any order. Providers that cache prompts serve the repeated video from their cache, and we share the observered cache hit rates we find. However, our reported cost per task does not consider cache discounts.
Similarly for local models, vLLM prefix caching can drastically increase the latency of subsequent requests. For these models, we only consider the first question's latency on the respective hardware.
| Benchmark | Questions | Videos | Per video |
|---|---|---|---|
| Video-MME v2 | 3,200 | 800 | 4.0 |
| LVBench | 1,549 | 103 | 15.0 |
| Perception Test | 454 | 189 | 2.4 |
| NExT-QA | 4,996 | 570 | 8.8 |
| Q-Bench Video | 868 | 862 | 1.0 |
| EgoSchema | 500 | 500 | 1.0 |
| UCF101-AD | 4,224 | 4,224 | 1.0 |
Tokens
- Input
- The input token count each model call returned: the video or its frames, and the prompt.
- Output
- The output token count each model call returned, including any reasoning before the answer.
Cost per task
(4)
Cost is calculated at the API price at the time of the evaluation run, and does not include any cache discounts for benchmarks with multiple questions per video. For local models, we do not currently report a price, but we may standardize a pricing formular for self-hosted GPU inference in the future.
A model’s cost per task across benchmarks is the mean of its per-benchmark results.
Latency
The time from sending a request to receiving the full answer. A model’s latency per task is the mean of it's median request latencies across benchmarks.
Hosted model requests initiate in Virginia, USA, with maximizing requests in flight before ratelimits. Their latency includes uploading and processing the video or frames according to the model’s protocol.
Self-hosted models ran one request at a time on NVIDIA A100-SXM4-80GB GPUs. We ignore vLLM prefix caching latency by only considering the first question's latency on the respective hardware. See the per-model protocol for each model’s endpoint and hardware.
Arena ranking
Benchmarks measure models on fixed footage and fixed questions. The arena measures them on yours: people upload a clip, ask a question, and judge two anonymous answers side by side. Rankings will be published once they have enough votes to be stable.
Battles and votes
Each battle draws two models at random from the arena pool, favouring models with fewer votes. Both see the same clip, question and generation settings where possible. Model identities stay hidden until a vote is cast, and the side each model appears on is randomised. A vote can prefer either answer or call them equal, as both good or both bad.
Bradley–Terry ratings
Votes are fitted with a Bradley–Terry model: each model has a strength, and the probability that one beats another depends only on the difference between their strengths. Strengths are estimated by maximum likelihood over all votes, then reported on an Elo-like scale.
(5)
A 100-point gap means the higher-rated model is preferred about 64% of the time.
A tie, whether both good or both bad, counts as half a win for each model. Ratings are anchored so the average model scores 1000.
Latest posts
Changelog
- Added cost and tokens/latency metrics with updated methodology.
- Added comparison page for models.
- Added details pages for per model and benchmark evaluation runs.
- Added Gemini 4 Argon to the Visual Intelligence Index, after a delay due to caching issues in the Vertex AI API.
- Public access to Vision Arena.