Introducing Vision Index
Why build an index?
OnDeck AI, Justin Chan · September 10, 2026
Methodology
Benchmark Selection, Indexes, and Evaluation Protocol
Roadmap
- Now
Open benchmarks and indexes
Private benchmarks, real-world task simulation
Arena and ELO-based scores
Agent harnesses, world models, VLAs
We're starting with open benchmark evaluations as a baseline for visual intelligence. Each next step will fill a gap in current vision measurement. Ultimately, we seek to define the standard for measuring visual intelligence and push its frontier towards economically meaningful performance.
Thesis
Vision Index exists because there are no leaderboards or arenas that adequately evaluate vision model performance on real-world video understanding tasks.
LLMs have demonstrated success in automating open ended text/code based tasks. As they've become multimodal, attention has shifted to visual intelligence and physical AI. However, evaluation has not kept up.
It's a bit of a chicken and the egg problem: video is expensive and hard, so models and inference infrastructure do not prioritize it well. But because it's expensive and hard, it's increasingly necessary to be able to robustly evaluate these models on representative video understanding tasks.
For more on why we're building this, read the launch post.
Index aggregation and scoring
Indexes distill a model's performance into a single score, simplifying comparisons across benchmarks, categories and scoring methods. We still publish the benchmark scores for transparency and model comparison. We will soon also provide per-question results for added visibility and benchmark transparency.
The Visual Intelligence Index
The Visual Intelligence Index is our headline metric: one score that summarizes overall model performance on video understanding. It is comprised of six capability indexes that aggregate cross-benchmark performance.
(1)
We weigh capabilities based on our judgement of their importance for video understanding. New index versions may adjust these weights.
Capability indexes
To construct each capability index, we classify every question in all 7 benchmarks into one or more of the six capabilities below. A model's index score is a weighted mean across that particular suite of questions.
| Capability | What it asks |
|---|---|
| Perception | Asks to identify, count, read, compare, or determine a directly observable property of an entity or visual element. |
| Spatial Reasoning | Asks about position, direction, distance, layout, containment, relative location, or other spatial relationships. |
| Temporal Reasoning | Asks about time, order, duration, frequency, sequence, before/after relationships, or changes across time. |
| Action / Event Understanding | Asks what someone or something is doing, what action occurs, what event takes place, or what overall activity is happening. |
| Causal Reasoning | Asks why something happens, what causes or results from an event, what someone intends or is trying to accomplish, or requires commonsense inference about motivations or consequences. |
| Information Retrieval | Asks for information that must be located or combined across multiple moments, entities, events, or details in the video. |
Since different benchmarks have different numbers of questions, we weigh each benchmark's contribution to a capability by how central the capability is to that suite, adjusted by the absolute number of relevant questions.
(2)
Centrality is the share of a benchmark’s questions labelled for the capability, n is how many there are, and K = 150.
| Benchmark | Perception | Spatial Reasoning | Temporal Reasoning | Action / Event Understanding | Causal Reasoning | Information Retrieval |
|---|---|---|---|---|---|---|
| Video-MME v2 | 19.5% | 24.2% | 24.0% | 15.3% | 21.4% | 25.9% |
| LVBench | 19.8% | 18.6% | 19.6% | 14.6% | 8.0% | 19.4% |
| Perception Test | 16.0% | 22.2% | 16.1% | 12.1% | 5.2% | 14.3% |
| NExT-QA | 19.3% | 31.7% | 22.0% | 18.8% | 33.1% | 6.2% |
| Q-Bench Video | 20.7% | 2.8% | 5.1% | 1.9% | 1.6% | 10.0% |
| EgoSchema | 4.7% | 0.5% | 13.2% | 16.5% | 30.7% | 24.1% |
| UCF101-AD | — | — | — | 20.7% | — | — |
(3)
Evaluation methodology
We run all inference and evaluation ourselves. In the current vision research landscape, papers publish scores but they aren’t comparable or interpretable. They differ in many dimensions, including dataset split, frame budget, subtitle access and scoring code. Those cause differences that are often bigger than the gap between the models, preventing even less precise relative comparisons.
Our objective is to control the variables and increase visibility into the process. Wherever possible, we standardize framerate, prompting, scoring, model executions, and data splits within benchmarks and between models. Deviations from standard are explicitly noted on this page and wherever relevant on the Index.
For 3rd party hosted models, we respect the standards set by their systems. For self-hosted models, we push standardization even further since we host and manage the inference.
- Inputs
- Models that take video natively receive video. The image-based models get frames we decode. Our baseline is sampling every video at 1 frame per second (FPS) to remain fair across models. However, due to provider context limits, some models are capped. When that context limit is lower than a video's frames sampled at 1 FPS, we evenly space the maximum allowed frames across the whole clip. This results in a lower effective framerate. The caps are listed in the Per-model protocol table.
For example, if a video is 200 seconds and thus 200 frames at 1 FPS, but the provider only takes 100 frames, we will sample at the provider's limit of 100 frames, making the effective framerate 0.5 FPS.
When models support native video input, we provide the video as is, including the original audio. When models do not support native video input, there is no way to provide the audio. - Inference
- We leave sampling parameters at each provider’s defaults rather than tuning them per model. No transcripts, no retrieval tools, no second attempts, and the prompt is fixed per benchmark.
- Scoring
- We don’t use the scoring code that ships with each benchmark. Most of them expect a bare option letter, and mark a correct answer wrong if it comes back as prose, or as the restated option, or as a letter buried in an explanation. Our standard inference protocols are not intended to measure instruction-following (although we will do this separately in the future). Note that this affects some models more than others.
Instead we use our own multiple-choice evaluator: it reads the response, works out which option the model picked, and checks it against the answer key. Same evaluator for every model on every multiple-choice benchmark, so a formatting habit doesn’t get scored as a capability. One benchmark has no multiple-choice form and keeps its own scorer — its row below says so. - Coverage
- Every score also lists how much of the benchmark it covers. If a provider's minimum safety filter refuses to process an input, this is reported and the response is scored as incorrect.
The benchmarks themselves are in the directory. We add one when it’s well built, widely cited, big enough for a difference between models to mean something, and covers a length or a capability we don’t already have.
Per-benchmark protocol
The table below lists the metric, evaluation split, and any benchmark-specific details that affect the scores.
| Benchmark | Primary metric | Notes |
|---|---|---|
| Video-MME v2 | Accuracy, no subtitles | We run this without subtitles to test what models can learn from the video alone. Videos last up to 35 minutes and exceed most models’ frame limits. When that happens, we sample fewer frames, evenly spaced across the full video. Each score reports the frame limit and how many videos exceeded it. |
| LVBench | Test accuracy | The median video is an hour long, and maxes out at two hours. Most videos exceed every model’s frame limit, so all models receive fewer than one frame per second, sampled evenly across the full video. We report each model’s limit beside its score. Differences in how many frames models receive matter when comparing these results. |
| Perception Test | Overall accuracy | We use the 454-question validation split from 2025. All clips fit within every model’s frame limit at one frame per second, so these scores can be compared without accounting for different sampling rates. |
| NExT-QA | Accuracy (hard split) | We use the multiple-choice questions from the 570-video validation split. We exclude the open-ended questions because they require a judge and a different scoring method. With an average of nine questions per video, this benchmark benefits most from prompt caching. |
| Q-Bench Video | Overall accuracy | We use the benchmark’s own evaluator on the combined dev and test splits. Q-Bench-Video has no dedicated multiple-choice answer extractor, so its answers are scored differently from those of the other six benchmarks. Keep that difference in mind when comparing scores across benchmarks. |
| EgoSchema | Accuracy (fullset) | We use the 500-question scored subset. Each video is exactly three minutes long and has one question. There are no repeat questions to offset the cost of caching a video, making this the most expensive benchmark per question for providers that charge for cache writes. |
| UCF101-AD | Accuracy | Each pair of questions asks the model to identify an action that occurs and reject a plausible action that does not. We report overall accuracy across both question types, rather than their harmonic mean. All models score close to perfect on actions that occur; most of the ranking differences come from correctly rejecting actions that do not. |
Per-model protocol
One rate and one metric across the board still leaves the transport up to each provider. What a model gets sent, and how much of a video it can hold at once, is set by its API and not by us.
| Model | Sent | Frame ceiling | Endpoint | Notes |
|---|---|---|---|---|
| Qwen3.8-Max | Video, 1 fps | 768 | Qwen Cloud (DashScope) | A content filter refuses a significant portion of videos deterministically; those items are marked as wrong and the reduced count is published. |
| GPT-6 Astra | Frames, JPEG | 1,500 | OpenAI API | — |
| Gemini 3.8 Flash | Video, 1 fps | 5,889 | Vertex AI | Sent video rather than frames, at the vendor default media resolution. |
| Claude Fable 5.1 | Frames, JPEG | 550 | Anthropic API | JPEG rather than PNG: a 32 MB request cap that PNG reaches at roughly 135 frames. The 600-image API limit binds before the context does. |
| Kimi K3 | Frames, JPEG | 250 | Qwen Cloud (DashScope) | — |
| Qwen3.8-27B | Frames | 768 | Self-hosted, vLLM | — |
| Qwen3.5-35B-A3B | Frames | 768 | Self-hosted, vLLM | — |
| Nemotron 3 Nano Omni 30B-A3B | Frames | 727 | Self-hosted, vLLM | — |
| Molmo 2 8B | Frames | 338 | Self-hosted, vLLM | Smallest model in the field, at 8B parameters. |
1 FPS is used until the frame ceiling is reached, after which we sample at the maximum rate the frame ceiling allows. This is recorded in the score’s setting. On the longest benchmarks nearly every video is restricted by the frame ceiling.