Are AI Models Really Understanding Videos, or Just Getting Better at Multiple Choice?

Disentangling video understanding from option exploitation.

OnDeck AI, Elliott Piercy · November 6, 2026

Illustration: a red line runs from a scene seen through a porthole to answer B on a multiple-choice sheet, past a question mark

Vision-language models are becoming increasingly capable of answering questions about videos. But how much of their performance comes from actually understanding what they see?

Our research, What Does Multiple Choice Actually Measure? Disentangling Video Understanding from Option-ExploitationNeurIPS 2026, investigates a fundamental problem with how video AI systems are evaluated.

The Problem with Multiple Choice

Most video understanding benchmarks use multiple-choice questions. A model receives a video, a question, and several possible answers, and must select the correct one.

Crucially, these answer choices are typically provided at the very beginning of inference, before the model has attempted to solve the question. This applies both to direct question-answering systems, which process the video and generate an answer in a single inference, and agentic systems, which iteratively inspect video segments, gather evidence, and reason before producing a final answer.

Six frames of three musicians, with the question 'why is the man standing on the right moving his left hand up and down the guitar' and five options. The answer, strum guitar, is found with the video and from the question alone. — Figure 2. NExT-QA, 3270490566.mp4, question 5 (frames sampled evenly across the whole video). The stem already names the instrument and the hand motion. Only strum guitar is compatible with both; the rest are ruled out before a frame is seen.

Six frames of a diplomatic meeting, with the question 'What color of coat does Melania Trump wear for the meeting that is hosted at Helsinki, Finland?' and four options. The answer, yellow, is found with the video and from the question alone. — Figure 5. LVBench, aJI8XTa_DII.mp4, question 10 (frames sampled inside the annotated time_reference window). The 2018 Helsinki summit is a heavily reported public event. The coat colour is recoverable from pretraining text, so the video is redundant.

Six frames of a dog in a swimming pool, with the question 'what is the dog doing' and five options. The answer, swimming, is found only when the video is shown. — Figure 12. NExT-QA, 2793806282.mp4, question 1 (frames sampled evenly across the whole video). A four-word stem carrying no constraint at all, against five ordinary dog activities. The answer lives only in the frames.

1 / 3

For agentic systems in particular, this means the answer choices can influence the entire reasoning process: what the agent searches for, which evidence it considers relevant, and when it decides it has enough information to answer.

This creates an opportunity for option exploitation. Rather than independently understanding the video, models can use the provided answers to narrow their search, eliminate unlikely candidates, or identify the most plausible response.

To investigate this, we evaluated video understanding systems on six established benchmarks: NExT-QA, EgoSchema, LVBench, QBench-Video, Perception Test and Video-MME v2.

On every benchmark, accuracy stayed above chance even with the video removed entirely.

Bar chart of accuracy on six benchmarks (NExT-QA, EgoSchema, LVBench, QBench-Video, Perception and Video-MME) under full multiple choice, no video, a black video, no question, random choice and a model-free heuristic. Full multiple choice scores highest on every benchmark, and the no-video and black-video bars stay well above random on all six. — Figure 1. The core context variations against direct QA, drawn against each benchmark’s chance baseline and its model-free heuristic. The gap between the blind bars and chance is the accuracy the question and options carry on their own.

This is a phenomenon known as benchmark shortcutting, where models exploit patterns in questions, answer choices, or dataset construction rather than solving the intended visual task. For example, a model might identify the most plausible answer based on common real-world scenarios, recognize linguistic similarities between questions and correct answers, or exploit recurring patterns in how distractors are written.

Public benchmarks also introduce the possibility of training-data contamination, where models may have encountered benchmark examples during training. While our experiments cannot establish memorization directly, they demonstrate how substantial accuracy can be achieved without visual evidence.

This raises an important question: are we measuring visual understanding, a model's ability to exploit the structure of the evaluation, or memorization?

Does Open-Ended Evaluation Solve This?

Real-world video understanding is open-ended. Users rarely give AI systems a list of possible answers. They expect the model to work out what happened. Multiple-choice benchmarks therefore risk measuring capabilities that don't reflect real-world deployment.

A natural alternative is open-ended (OE) evaluation, where models generate answers without seeing predefined options. Across our experiments, this consistently produced lower accuracy than multiple-choice evaluation.

However, open-ended evaluation introduces its own challenge: how do we reliably score free-form answers? We used two independent LLM judges and found substantial differences in their assessments of identical responses, highlighting the sensitivity of results to evaluation methodology.

To investigate whether this scoring uncertainty explains the performance gap, we also tested open-ended-to-multiple-choice matching (OE→MC). A separate evaluator maps generated answers to the closest multiple-choice option without seeing the video. While this recovered some of the performance gap under direct QA, a substantial difference remained, suggesting that access to answer choices influences more than just how responses are scored.

Introducing Option-Delayed Replay

We introduce Option-Delayed Replay (ODR), a technique designed for agentic video understanding systems.

Rather than exposing answer choices from the beginning, ODR allows an agent to investigate the video independently, gathering evidence and reasoning about the question without knowing the possible answers.

We then replay its reasoning trajectory, revealing the multiple-choice options at controlled points: halfway through the investigation or only at the final answer.

Algorithm 4, option-delayed replay. Given an open-ended trajectory, the options and a cut index k: keep the trajectory up to step k without re-running tools, append the options to the question, then either select an option from the completed trajectory (when k is the last step) or resume the agent from the cut. — Algorithm 4. Option-delayed replay. The options enter the agent’s context only at step k + 1; everything before the cut is the open-ended trajectory, unchanged.

Revealing answer choices only at the final step produced performance close to conventional multiple-choice evaluation. Across 18 agentic evaluations, accuracy remained within 3.4 percentage points of native multiple-choice performance, although it was slightly lower in most cases.

This small but consistent gap suggests that early access to answer choices helps beyond picking the final answer: the options may shape what evidence agents look for and how they reason about the video.

The blind results complicate this. Language priors and benchmark shortcuts already answer a large share of questions, so only a smaller subset actually needs the video. Further work is needed to tell how much of the remaining gap is visual reasoning and how much is option exploitation.

Why This Matters

As VLMs move into real-world applications, benchmark accuracy alone is insufficient.

We need evaluations that distinguish visual perception, reasoning, language shortcuts, and scoring effects.

Our work provides a framework for investigating these differences and highlights a simple principle: getting the right answer does not necessarily mean understanding the video.