What I learned re-evaluating VLADBench on the latest VLMs

TLDR

I re-evaluated 15 of today's top VLMs on VLADBench, a driving scene VQA benchmark from 2025, and here is what I learned:

• VLMs have gotten a lot better, the field mean improved across all tasks with one exception (see below).
• Frontier capability is going to cost you (Overall Score, Cost of all Tasks)
  • GPT-6-Astra (83.6, $542.16) and Claude-Opus-5.5 (82.7, $205.28) sit at the frontier of visual understanding.
  • Gemini-3.8-Flash (75.9, $16.93) is a goldilocks pick on cost-performance but fails to lead the field on any task.
• Also, building benchmarks for VLMs is hard.

We're releasing the results of this VLADBench re-evaluation with the full code and dataset on Hugging Face. Every figure below is a live embedded visualization from the companion results explorer, which was used to analyze the results.

How well do frontier VLMs understand driving scenes?

Turns out, a lot better than last year.

Internally we've seen VLMs struggle to understand robotics video. We're constantly evaluating new models so I figured it would be a good idea to run a fresh model survey on a domain-specific benchmark. I wanted to see if general improvements in visual understanding transferred to domains like Autonomous Driving, so I revisited a driving scene understanding benchmark from a paper repo I had recently starred.

VLADBench (Li et al., ICCV 2025) is a closed-form VQA benchmark made up of 28 scored tasks across 5 domains. It evaluates how well VLMs understand driving scenes across a variety of situations and dimensions. The paper's core contributions were its fine-grained task suite and domain-adapted model training. Most other AV benchmarks bucket tasks into broad categories like perception, planning, and prediction. By breaking tasks into smaller concerns, the authors were able to measure whether visual understanding capability in general-purpose VLMs transferred or if specialized models were needed for driving scene understanding.

This 2026 model survey evaluates zero-shot VQA accuracy of frontier VLMs. It is not a comprehensive AV benchmark of driving scene understanding nor a domain transfer training experiment. In other words, I'm only examining how general-purpose VLMs perform on specialized driving tasks, using VLADBench's scores as a practical indicator of how broadly their visual capabilities extend to AV settings. The original 2025 survey included general and specialized models, where five of the paper's 25 models were driving-specialized or domain-trained (Dolphins 9B, DriveLM 4B, DriveMM 7B, DriveLM-B 4B, and the paper's own 4B model).

A few notes on how I ran this re-evaluation: I used the original questions, frames, gold references, scorer, and per-task weights to survey frontier models across all tasks. We didn't explore domain transfer training. Not all of the models supported video inputs, so image frames were sent otherwise. Additionally, reasoning was turned off where providers allowed it, and the lowest setting otherwise.

What the survey says

Overall, the field mean improved across all but one task, with most of today's models comfortably leading last year's top score.

OpenAI's GPT-6-Astra now leads the benchmark with a combined score of 83.6 which is a +25.6 point improvement over Qwen2.5-VL-72B's score of 58.0 from 2025. GPT-6-Astra leads 14 of the 28 tasks tested, with Claude-Opus-5.5 leading 7 tasks, GPT-5.6-Sol 3, Qwen-3.8-Max 2, GPT-6-Sol 1, and GPT-6-Luna 1 splitting the other task high scores.

It comes as no surprise that the largest, most expensive model performs the best. This survey had no shortage of big expensive frontier models. Clearly size matters, but the leaderboard tells us which models were better than others under fixed scoring criteria. What it doesn't tell us is how well each model fared across tasks, or whether the task itself was graded fairly.

There were a few problems I ran into when reviewing the results. Some tasks had questions with goldens that weren't part of the choices in the prompt, and others like the weather and light tasks presented images at dusk or in otherwise ambiguous scenes without complementary disambiguating labels. How are you going to show an intersection at dusk with a bright sky and expect a model to say nighttime? I guess the lights were on... The companion has a feature for you to be able to take the benchmark yourself for this very reason.

The vehicle cut-in task was the weirdest. It was the only task where the field got worse (36.6 to 35.6), and when I dug in, 172 of the 174 yes/no questions had true positive goldens. Just answering yes every time would get you about 99% on those questions. The prompt also asks whether the vehicle has "the intention to cross the road", which is a pretty strange way to ask about a cut-in. I'll dive into all of this in a follow-up. If you want to look now, the Caveats tab has every question and every model's answer.

For now, trust that designing VLM benchmarks is hard.

Back in 2025, none of the tasks really showed risk of saturation. This model survey now shows eight tasks averaging above 75 and top scores well into the mid 90s. The original benchmark didn't have a single task's field mean exceed 68.

That being said, I don't expect AV teams to be burning GPT-6-Astra credits on fleet-scale video. Clearly there are still gaps in visual understanding capabilities across the field, but the question isn't simply which model got the highest score, but also at what cost? Tracking the cost-performance frontier is how you find out which model is necessary given a task and a budget. The figure below shows which model leads the cost-performance frontier for this model survey. In a separate follow-up I'll dive into what running each model costs per video hour and how that changes the frontier. (It stays mostly the same.)

There were a few results that surprised me. I've included several expensive frontier models in the survey so the leading scores were expected. What was surprising was GPT-6-Luna and GPT-6-Sol scoring worse or about even with their predecessors. If I had to choose, would I run GPT-6-Luna, GPT-5.6-Luna, or Gemini-3.8-Flash?

Also, I haven't seen a ton of benchmarks with Meta's Muse-Spark-1.3 in the mix. Given the competition, a score of 73.1, seventh of 14, for $64 a sweep is no headline. It was also the second-slowest model to answer, at a 10.2-second median. It also failed to lead on any of the tasks, with a spatial-temporal reasoning of 75.1 which was right around the field mean.

Lastly, I hate to say I'm pretty disappointed with the smaller open-source models. I'm a big advocate for open-source, but unfortunately it looks like size really does matter. If it really takes a 2T+ parameter model to achieve frontier visual understanding in self-driving, self-hosting isn't going to be economically feasible.

What I learned

A few things.

First, VLMs should be judged against the task and the budget. VQA only works when it's grounded in the task you're looking to evaluate. Whether or not you plan on using GPT-6-Astra, Gemini-3.8-Flash, or Qwen-3.6-35B-A3B for that task depends on how much video you need to process and how big your pocketbook is. This is separate from the difficult problem of actually knowing what you want to learn from a particular video segment. Open-loop VQA is a particularly brittle means of understanding long horizon tasks.

Second, spatial-temporal reasoning has come a long way, even for models that don't accept video. VLADBench is not a perfect measure of spatial-temporal reasoning, but given how broad the spread is in results, it's clear that the task is measuring capability, not just bias. GPT-6-Astra (95.1), Qwen-3.8-Max (89.9), Claude-Opus-5.5 (87.8), and GPT-6-Sol (87.6) all score above 87, and none of them used video inputs on this task. Only 1 of the task's 161 questions failed to receive a correct answer from any model, so low scores are not from a bad-reference artifact.

Finally, evals are hard. At the end of the day, model surveys on benchmarks like VLADBench help teams calibrate their intuition against how the frontier is evolving. Every time a new model comes out, the cost-performance frontier has the chance to get updated. Whether or not your use case requires frontier capability is a separate, more expensive question. This 2026 survey serves as a practical comparison of different VLMs' visual understanding capabilities in a specific domain. That grounding helps practitioners like me tell whether or not a specific model actually serves their use case and whether or not it's worth the arm and leg after all.

Resources: data, code, and requests

• Interactive results: eventual-inc.github.io/VLADBench.
• Dataset: Eventual-Inc/VLADBench-reeval
• Code: Eventual-Inc/VLADBench.
• Protocol and differences from the paper: docs/PROTOCOL.md.
• The benchmark: VLADBench paper, original repository, original dataset.
• Take the benchmark yourself: answer the questions in your browser.
• Request a model to be added: email me with the provider slug.