STEAM: Self-Supervised Temporal Ensemble Advantage Modeling for Real-World Robot Learning

真实世界机器人学习越来越依赖异质数据，但示范和 rollout 往往混合了有用进展、停滞、纠正和次优行为。因此，有效的策略学习需要逐帧 advantage 来区分可靠的局部进展与失败或回退。本文提出 Self-supervised Temporal Ensemble Advantage Modeling（STEAM），一种无需标签的方法，从专家示范中学习这样的 advantage。STEAM 在专家轨迹内部的帧对上训练 temporal-offset predictor ensemble，以两帧之间的归一化时间偏移作为自监督信号。每个 predictor 将帧对映射到时间偏移分布，再转换为标量 advantage。随后 STEAM 在 ensemble 中取最小 advantage，以保守方式给混合质量 rollout 数据打分。在真实双臂叠毛巾、薯片结账、可乐补货以及单臂 pick-and-place 任务中，STEAM 能识别停滞、失败和恢复。与 CFGRL 结合后，STEAM 相比基线进一步将策略成功率分别提升 59%、54.3%、23% 和 16.2%。

Real-world robot learning increasingly relies on heterogeneous data, but demonstrations and rollouts often mix useful progress with stalls, corrections, and suboptimal behavior. Effective policy learning therefore requires frame-level advantages that distinguish reliable local progress from failures and regressions. We propose Self-supervised Temporal Ensemble Advantage Modeling (STEAM), a label-free method that learns such advantages from expert demonstrations. STEAM trains an ensemble of temporal-offset predictors on frame pairs within expert trajectories, using the normalized temporal offset between two frames as a self-supervised signal. Each predictor maps a frame pair to a distribution over temporal offsets, which is converted into a scalar advantage. STEAM then takes the minimum advantage across the ensemble to score mixed-quality rollout data conservatively. Across real-world bimanual towel folding, chip checkout, cola restocking, and single-arm pick-and-place tasks, STEAM identifies stalls, failures, and recoveries. When combined with CFGRL, STEAM further improves policy success rate by 59%, 54.3%, 23% and 16.2% over baselines, respectively.