Introducing Vision Index

Pretext

Vision Index is created by OnDeck AI, a visual intelligence research company. We are both an applied AI lab that serves a proprietary vision model and a deployment company working with customers to solve real-world tasks.

This means we inherently need to evaluate vision models and systems a lot. We see a growing need for robust evaluation, but few readily available solutions.

OnDeck's primary goal is to build recursive self-improving mechanisms for vision. That requires a hill to climb: measurable objectives we can repeatedly improve against. This hill exists for language-based tasks thanks to arenas and independent benchmark companies. A comparable well-defined hill for video understanding tasks does not exist.

Vision Index is a first step towards solving this problem. We are sharing lessons, results, and evaluation standards with the community, that we've developed through working on hard frontier vision problems.

We are excited for everyone to use it when choosing and deploying vision models for real world tasks.

Feedback & Inquiries

Video is hard

How do we evaluate a model's visual intelligence? Video understanding is an active research area and existing evaluations range from action recognition and temporal grounding to long-video reasoning and question answering, often under very different settings.

The harder question is how well any of this represents visual intelligence in the real world. Humans rarely watch video just to answer a question about it. We watch with a goal, use outside context, decide where to focus, connect evidence across time, and ultimately use what we understand to do something economically valuable.

Evaluating this is difficult and may explain why current evaluation providerscurrent evaluation providersEvaluation Landscapearena.aiLimited to image input and categories like OCR, captioning, and entity recognition.Artificial AnalysisVisual Reasoning Index is just single image input MMMU Pro benchmark.RoboflowEvaluates VLMs on image tasks like object detection, counting, OCR, and QA.Andon LabsReal world evals that indirectly incorporate vision capabilities.RobocurveReal world evals for physical AI.Benchmark AggregatorsLLM Stats, OpenCompass, BenchLMCaptures some video benchmarks but vision is not the main focusNotable MentionsDesign Arena, Vals AI, DatacurveNo vision focus largely don't address them. Not to mention:

• Video is orders of magnitude larger than text.
• Inference is expensive and video infrastructure is fragmented.
• Hallucinations are difficult to detect.
• Ground truth is harder to annotate and may be subjective.
• Public benchmarks are prone to contamination and language priors.

Reducing visual intelligence to video question answering (often multiple choice) is not representative of real tasks. Real tasks are less constrained: systems need to decide what matters, find the relevant evidence, bring in context, and determine what to do with it efficiently.

What we're building

While public benchmarks won't get us all the way there, they provide useful signal on the core capabilities behind visual intelligence. We aggregate them into capability indexes and leaderboards, informed by what we've learned evaluating models internally. This is where Vision Index starts today.

Our view is that evaluation should get much closer to the real tasks these systems are used for. That means private benchmarks beyond question answering, from closed real-world tasks to full harnesses that simulate production systems around a vision model. It also means an open arena, where crowdsourced voting can provide an independent signal.

Evaluating the model alone also won't be enough. With agentic methods like our own Perception-0, we'll want to understand the reasoning traces and trajectories of video agents, and expand to world models, VLAs, and other new approaches.