Example question
Accuracy on this category
Abstract
For embodied agents in urban streets, visual perception must support spatial reasoning not only relative to the camera but also relative to objects in the scene. Previous studies have focused either on object-centric reasoning in indoor scenes and general images, or on urban scenes from vehicular, aerial, or satellite viewpoints, leaving the pedestrian viewpoint largely unexplored. To address this gap, we propose UrbanStreetVQA, a pedestrian-viewpoint VQA benchmark covering perception as well as camera- and object-centric spatial reasoning. Our benchmark consists of 1,731 human-verified questions across seven cities and seven categories. We evaluate nine representative VLMs under a circular-evaluation protocol and find that object-centric direction estimation collapses consistently, often below chance rate, whereas perception and camera-centric reasoning are handled comparatively well.
Human-verified, in seven categories
Frames filmed at eye level along a pedestrian’s path
Across Asia, the Middle East, Europe, and the Americas
Proprietary, open, and specialized for spatial perception or driving
Seven cities, at eye level.
35 walking videos, about 31 hours in total, filmed along a pedestrian’s path. Every city has at least 20 questions in every category.
Tokyo259 questions, 4.0 h of video
Shanghai248 questions, 4.4 h of video
Mumbai155 questions, 1.7 h of video
Dubai205 questions, 3.9 h of video
London292 questions, 5.7 h of video
New York243 questions, 5.2 h of video
Mexico City329 questions, 6.6 h of videoWhat we found.
Accuracy (%) under circular evaluation: a question counts only if the model picks the right answer under every order of the options.
| From the camera | From a person | |||||||
|---|---|---|---|---|---|---|---|---|
| Model | Obj. recog. | Count | Self-loc. | Dir. | Dist. | Dir. | Dist. | All |
| Proprietary | ||||||||
| GPT-5.4 | 82.2 | 61.3 | 61.4 | 78.6 | 84.7 | 17.8 | 78.9 | 67.9 |
| Gemini 2.5 Flash | 85.5 | 49.3 | 72.9 | 66.9 | 84.7 | 19.0 | 62.5 | 63.1 |
| Open | ||||||||
| InternVL3.5-8B | 83.6 | 64.0 | 72.1 | 47.2 | 83.6 | 22.7 | 66.4 | 60.2 |
| LLaVA-OV-1.5-8B | 84.9 | 38.2 | 59.3 | 35.9 | 75.1 | 17.0 | 55.9 | 49.5 |
| Gemma 4-12B | 83.6 | 56.0 | 75.7 | 64.9 | 85.0 | 22.3 | 75.0 | 65.2 |
| Qwen3.5-9B | 86.2 | 70.2 | 70.7 | 71.2 | 82.7 | 32.0 | 75.7 | 69.5 |
| Spatial perception | ||||||||
| PerceptionLM-8B | 88.2 | 52.0 | 68.6 | 75.3 | 87.8 | 31.2 | 61.2 | 67.9 |
| SpatialThinker-7B | 78.9 | 60.9 | 63.6 | 66.0 | 85.0 | 27.9 | 62.5 | 64.4 |
| Autonomous driving | ||||||||
| DriveLMM-o1-8B | 72.4 | 49.3 | 51.4 | 66.5 | 83.0 | 22.7 | 70.4 | 61.0 |
| Reference | ||||||||
| Human (5 annotators) | 94.3 | 93.3 | 93.3 | 98.1 | 91.4 | 88.6 | 96.2 | 93.6 |
Bold: best model in each column. Shaded: direction from a person, where every model collapses.
Direction collapses once the viewpoint changes.
No model exceeds 32% on direction from a person’s view, and six of nine fall below the 25% chance rate. Humans reach 89%. Models built for spatial perception or driving collapse too.
The errors follow a pattern.
Asked about front or behind, models mostly answer left or right. Asked about left or right, they pick the correct side about half the time, no better than a coin flip. Their answers rarely change with the option order, so the errors are systematic.
Front/Behind questions answered as front or behind (%)
Left/Right answers on the correct side (%)
Dashed line: chance (50%).
Some cities are harder for every model.
Averaged over nine models, accuracy ranges from 69.6% in London to 58.1% in Tokyo, and London is easier than Tokyo for every model. Stronger models also vary less from city to city (r = −0.72).
Mean accuracy across nine models (%)
Fine-tuning helps, but does not close the gap.
Fine-tuning Qwen3.5 lifts direction from the camera above 90% without touching the vision encoder. Direction from a person improves only when the encoder is trained too, and even then reaches just 40%.
Direction from the camera (%)
Direction from a person (%)
Each step trains one more module: the projector, then the language model, then the vision encoder.
Data release
We do not redistribute YouTube frames. The repository provides video IDs, timestamps, frame-extraction scripts, all annotations, and evaluation code, so the benchmark can be rebuilt in full. For research use only; annotations about people are limited to coarse geometry.
Citation
@inproceedings{huang2026urbanstreetvqa,
title = {UrbanStreetVQA: A Benchmark for Visual
Understanding of Urban Street Views},
author = {Ziliang Huang and Kosuke Kurihara and
Kiyoharu Aizawa and Takayuki Hamamoto},
booktitle = {Proceedings of the Asian Conference on
Computer Vision (ACCV)},
year = {2026}
}