Learning the 4D World through Point Trajectory Forecasting

可靠的多视角预测要求不同相机的视频描述同一个演化中的三维场景。本文提出视频—点模型 VPM，以多视角视频历史和语言指令为条件，联合预测任意视点的未来视频，以及具有持续身份的场景点的三维轨迹。作者为预训练世界模型加入相机条件化的多视角处理，整合各视角互补观测，并加入预测物理点运动的轨迹头。同一条轨迹在全部视角中对应同一物理点，因此联合监督轨迹与视频，可把各相机的外观变化锚定到共同的三维运动描述。为支持训练与评估，作者构建配对同步视频与持续点轨迹的仿真语料 VPM-Data，以及覆盖仿真和真实环境的 VP-WorldBench。进一步把机器人动作作为生成器的 token 模态，动作与未来视频联合去噪，并共享经过点监督的骨干。在该基准及现有四维生成评测中，相比纯视频版本，点监督改善视频保真度和三维运动准确率；在仿真与真机实验中，也提高加入动作模态后的操作表现。

Reliable multi-view forecasting requires videos from different cameras to describe the same evolving 3D scene. To this end, we introduce Video-Point Model (VPM), a world model that conditions on multi-view video histories and a language instruction to jointly forecast future videos from arbitrary viewpoints and the 3D trajectories of persistent scene points. We extend a pretrained world model with camera-conditioned multi-view processing that integrates complementary observations across cameras, and add a trajectory head that predicts how physical points move through time. Each trajectory tracks a single point across all views, and supervising these trajectories alongside video generation therefore anchors appearance changes in every camera to a shared description of 3D motion. To support training and evaluation, we build VPM-Data, a simulation corpus pairing synchronized multi-view videos with persistent point trajectories, and VP-WorldBench, a forecasting benchmark spanning simulated and real environments. We further extend VPM to robot control by adding robot actions as a token modality in the generator. Action tokens share the point-supervised backbone and are denoised jointly with future videos, thereby grounding the policy in the same 3D-aware representation. On VP-WorldBench and existing 4D generation benchmarks, point supervision improves both video fidelity and 3D motion accuracy over a video-only counterpart, and in simulated and real-robot experiments, it also raises the manipulation success of the action-enabled model.