Recently, world-action models (WAMs) have emerged as a powerful framework for robot manipulation. By jointly modeling action generation and future observation prediction, WAMs can act and anticipate the outcomes of the actions. However, existing WAMs typically rely on frozen pretrained encoders trained with reconstruction, which may not produce representations well suited for robot control.
In this paper, we tackle the problem of learning physically-grounded, control-relevant representations for WAMs. We propose MDF++, a visuo-tactile world-action model that learns latent representations of multisensory observations end-to-end with a diffusion backbone. While naive joint training can lead to feature collapse, MDF++ introduces a multi-task learning strategy that leverages cross-modal correlations to learn task-relevant representations. We evaluate MDF++ on a suite of contact-rich manipulation tasks, including real-world bolt screwing and nut mating, and show that it outperforms WAMs built on frozen pretrained representations by up to 50%.
We evaluate MDF++ on a Franka Research 3 robot with a WSG-50 gripper on two real-world contact-rich threading tasks on a car engine: Bolt Screwing and Nut Mating. The evaluation uses 20 trials per policy-task pair, with training data collected from 300 Bolt Screwing demonstrations and 255 Nut Mating demonstrations using a UMI equipped with GelSight sensors.
MDF++
UWM
Misaligns and fails to recover.
Inserted but not engaged.
MDF++
UWM
Falls off after gripper release.
Not engaged.
MDF++
MDF++ (frozen encoder)
OOD and stays still.
Fails to approach.
MDF++
MDF++ (frozen encoder)
Inserted but not engaged.
Fails to approach.
MDF++ represents each training query as a time-modality noise matrix: observed history tokens are given zero noise, while target tokens over the prediction horizon are corrupted and denoised. Instead of relying only on random masks, MDF++ samples five control-relevant patterns with equal probability: policy learning, world-action modeling, forward dynamics, inverse dynamics, and random masking. These structured masks focus the diffusion backbone on conditionals that are directly useful for robot control while still preserving temporal and cross-modal prediction.
