A Dataset-Centric Survey of Video Understanding:
Benchmark Characteristics, Architectural Evolution, and Future Directions
Abstract
Research in video understanding has advanced rapidly, driven by increasingly diverse datasets and more powerful model architectures. While existing surveys often organize progress by tasks, benchmarks, or model families, they provide limited insight into why particular architectures emerged and succeeded.
This survey adopts a dataset-centric perspective: dataset structure shapes model design. Motion complexity, temporal span, compositional hierarchy, multi-agent interaction, and multimodal richness impose distinct learning challenges. These pressures naturally give rise to inductive biases for viewpoint robustness, temporal ordering, long-range dependency modeling, relational reasoning, and cross-modal alignment.
Dataset Properties
Motion, duration, interaction, composition, and modality define the learning signal.
Inductive Biases
Models favor different invariances and evidence patterns depending on the data regime.
Architectural Response
Model families can be understood as responses to evolving dataset requirements.
Five-Dimension Taxonomy Explorer
Every benchmark is decomposed into five structural dimensions, each coded from its own sub-variables under a fixed rubric. Pick an anchor benchmark to see where it lands.
Each axis sums its sub-variables, normalised to 0–4. The codes are reproduced verbatim from the survey’s coding; only the weighting into a scalar is ours.
What the structure looks like in the raw data
Magnitude vs. fine-grained dependence
UCF101 is recognisable from global spatiotemporal patterns. Diving48 categories differ only in body rotation, posture and execution order.
Egocentric hand-object interaction
EPIC-KITCHENS-100: a moving first-person camera and continuous manipulation make human-object relations task-critical.
Paired ego / third-person
Charades-Ego records the same overlapping everyday actions from both viewpoints, making viewpoint variability an explicit variable.
Semantic hierarchy
Kinetics-400 organises labels taxonomically. The hierarchy structures the label space, not the timeline.
Procedural hierarchy
Breakfast decomposes one activity into ordered sub-actions. This one introduces temporal dependencies between constituents.
Long procedural span
MPII Cooking 2 holds the camera still and lets the activity run long, isolating temporal span from viewpoint change.
The hook
C/w/s: coarse alignment, weak pairing, sparse
grounding. Ego4D is coded D/v/d: dense alignment,
verified pairing, dense grounding. Ten million weakly captioned clips
score 1.3 / 4 on the multimodal axis;
Ego4D scores 4.0. Modality availability is not
multimodal structure — which is exactly why a single
“multimodal” label cannot predict what a model has to learn.
Structure Leaves a Trace in Attention
If dataset structure really is the driver, then holding the architecture fixed and changing only the pretraining corpus should change what the model looks at. It does. The attention maps below are from identical architectures that differ only in the dataset they were pretrained on.
MotionFormer
Dataset-induced representational biases (final block)
Original Frames
Kinetics-400
Kinetics-600
Something-Something V2
EPIC-KITCHENS-100
Dataset-induced biases across network depth (pre-training on Something-Something V2)
TimeSformer
Dataset-induced representational biases (final block)
Original Frames
Kinetics-400
Kinetics-600
Something-Something V2
Dataset-induced biases across network depth (pre-training on Something-Something V2)
Playground
Choose two model/pretrain/video groups. Every block is shown in order.
Benchmark ↔ Architecture Co-evolution
Datasets above the axis, architecture families below. Hover or tap a benchmark to trace what it is associated with — and how that association is supported.
Not just a narrative: matched comparisons
Reported accuracies aggregate publications that differ in pretraining corpus, capacity and evaluation protocol, so they can only be read descriptively. These two cases hold one group of factors roughly fixed while another varies. They are associational, not controlled; the residual confounds are stated.