Video understanding models have made significant progress, but there's a problem that hasn't been solved well: what's the actual difference between a model watching a video and looking at a pile of photos?
Most video understanding models essentially treat video as "a sequence of images." The time dimension is added, but not deeply enough. Human actions, pose changes, motion trajectories—these video-specific signals get flattened into frame differences, losing a lot of structured motion semantics.
A research team from NYU and others submitted Cambrian-P on May 21, bringing pose information directly into video understanding models as a first-class citizen. Accepted at CVPR 2026.
Pose Is Not an Add-on, It's the Key to Understanding Video
Cambrian-P's core judgment is direct: human pose changes in video are the most direct clues for understanding action intent, interaction relationships, and scene semantics.
When you watch a video, you understand "one person teaching another to box" not because each frame is especially clear, but because you capture the pose change relationship between two people—one demonstrates, the other imitates. This kind of understanding is hard to build from frame-level visual features alone.
Cambrian-P puts pose estimation and video understanding in a unified framework. Not a pipeline of "run a pose estimation model first, then feed results to the video model," but joint learning within the same model.
Why Now
Pose estimation itself is already mature. OpenPose, MMPose—these tools can be precise down to the joint level. But effectively integrating pose information into large video understanding models has lacked a validated paradigm.
On one hand, alignment between pose information and visual features isn't something simple concatenation can solve. On the other hand, pose data itself is noisy—occlusion, fast motion, low lighting all cause estimation errors. The model needs to learn when to "fall back" to pure visual mode when pose is unreliable.
Paper Details Are Limited
The paper was just submitted. Detailed method descriptions and experimental results need to wait for the full PDF. The project page should have more visualization results.
But from the direction itself, if pose-grounded video understanding can be validated on large-scale datasets, it would directly impact several applications: video content moderation, sports analysis, human-computer interaction, and even pedestrian behavior prediction in autonomous driving.
A Notable Point
The author list includes Saining Xie (NYU) and Bingyi Kang—both names with solid track records in vision and robotics. This isn't a "chase hot topics and publish one paper then disappear" team. Follow-up work is worth watching.
Primary sources:
- Cambrian-P paper (arXiv:2605.22819, May 21, 2026)
- Accepted at CVPR 2026
- Project page: https://cambrian-mllm.github.io/