TouchScale: 500 Hours of Human Vision and Touch for Visual-Tactile Learning

大规模第一视角人类交互数据正成为具身学习的重要物理监督来源，但视频无法记录物理交互的接触与压力。近期视觉触觉数据集补充了这些信号，但同步触觉规模远小于人类视频；较大资源常混合不同传感器或标注流程，难以单独研究规模的影响。一致传感流程下记录数百小时视觉与触觉的数据仍然稀缺。因此作者提出 TouchScale：以统一可穿戴设备记录的 500 小时接触密集人类交互数据集。约 2K 预定义任务描述覆盖日常活动和结构化操作，每段记录将第一视角 RGB-D、腕部 RGB 和稠密双手全手触觉按时间对齐。作者研究扩大人类视觉触觉数据能为感知和机器人学习带来什么。相较既有数据，完整 TouchScale 将未见传感器上的零样本接触 IoU 从 0.134 提至 0.383。预训练视觉编码器在三个动作识别基准上均优于所比较的数据集。作为机器人策略视觉触觉中训练数据，TouchScale 将四项真机接触密集任务平均成功率从 22.5% 提升至 57.5%。固定传感器和采集协议后，随着数据增加，零样本触觉预测与机器人成功率总体上升。这说明一致传感下规模化采集的人类视觉触觉数据有益于感知和机器人操作。作者计划公开所有同步记录和重建物体模型，支持可扩展视觉触觉学习研究。

Large-scale egocentric human interaction data is becoming an important source of physical supervision for embodied learning, yet video alone leaves the contact and pressure that characterize physical interaction unrecorded. Recent visual–tactile datasets provide this missing supervision, but their synchronized tactile data remain far smaller in volume than human video. Moreover, the largest resources often merge recordings from different sensors or annotation procedures, which makes the effect of data scale difficult to isolate. In short, datasets that capture hundreds of hours of human vision and touch through one consistent sensing pipeline remain scarce. We therefore introduce TouchScale, a 500-hour dataset of contact-rich human interaction recorded with a single unified wearable setup. Its approximately 2K predefined task descriptions span everyday activities and structured manipulation, and each recording temporally aligns egocentric RGB-D video with wrist RGB video and dense full-hand bimanual tactile measurements. With TouchScale, we ask what scaling human visual–tactile data unlocks for perception and robot learning. Compared with prior tactile data, training on the full TouchScale raises zero-shot contact IoU on data from an unseen tactile sensor from 0.134 to 0.383. Pretraining a visual encoder on TouchScale also yields the highest action recognition accuracy on three benchmarks among the compared visual–tactile datasets. Used for visual–tactile mid-training of a robot policy, TouchScale improves the average real-world success rate across four contact-rich manipulation tasks from 22.5% to 57.5%. With the sensor and collection protocol held fixed, both zero-shot tactile prediction and robot success show an overall upward trend as more TouchScale data is used. These results suggest that human visual–tactile data collected at scale with consistent sensing benefits both perception and robot manipulation. We will publicly release TouchScale, including all synchronized visual–tactile recordings and reconstructed object models, to support future research on scalable visual–tactile learning.