Cylinder Insertion
Four cylinders of different sizes are randomly arranged in front of their slots. The robot must identify each cylinder by size and insert them in ascending order until all four are placed.
Building on recent advances in world models, World Action Models (WAMs) jointly model video prediction and action generation. However, they typically represent videos in 2D pixel space, creating a representation gap with the 3D space in which robotic actions are executed. Recent 3D approaches introduce 3D information, but fail to fully exploit the dynamics of 3D structures.
In this work, we propose 4D-WAM, a model-agnostic training strategy that injects spatiotemporal knowledge from 3D trajectory fields into WAMs through representation alignment. To this end, we introduce two complementary objectives: (1) motion alignment, which aligns temporal feature variations across adjacent frames and encourages the model to build local 4D awareness during training, and (2) destination alignment, which guides the model to infer the final destination from the source frame by minimizing the gap between their attention-like similarity distributions. Together, these objectives provide both local motion supervision and long-horizon goal guidance, enabling WAMs to learn trajectory-level spatiotemporal representations.
Extensive in-distribution and out-of-distribution experiments across different base models demonstrate improvements in spatial understanding, execution precision, robustness, generalization, and versatility.
Directly matching WAM features to 3D/4D foundation-model features — the recipe that works for VLAs — consistently hurts WAMs. The two representation spaces are simply too different. 4D-WAM instead aligns their temporal differences and their source-to-destination correspondence, which describe the same underlying motion regardless of feature space.
Figure 3. Alignment process. Multi-frame video is encoded by the (trainable) WAM and the (frozen) Trace Anything teacher. Lightweight projectors map both into a shared space; (1) token-wise latent differences between adjacent frames are matched with cosine similarity, and (2) source→destination similarity distributions are matched with KL divergence. Everything added is removed after training.
Figure 2. Why naive alignment fails. Across five RoboTwin 2.0 tasks, (a) a VLA's (π0.5) hidden states are consistently positively similar to 3D features from VGGT, whereas (b) a WAM's (Lingbot-VA) hidden states are consistently negatively similar to 4D features from Trace Anything. (c) Their frame-to-frame temporal differences, however, share consistent motion patterns — the signal 4D-WAM aligns.
For each token, the change between adjacent frames in the WAM's projected DiT features is pulled toward the corresponding change in the teacher's 4D trajectory features. Absolute features may live in different spaces, but their frame-to-frame deltas describe one consistent motion process — object displacement, contact, and the physics of manipulation.
The first frame is the source, the last frame the destination. For every source token, a temperature-scaled softmax over its similarities to destination tokens gives an attention-like distribution of "where this ends up". The WAM's distribution is matched to the teacher's with KL divergence, teaching goal-directed correspondence beyond local motion.
Overall objective: L = Llatent + Laction + λmotion Lmotion + λdest Ldest, with both λ < 0.1. Alignment is applied at DiT layer 20. Pre-extracting the teacher's trajectory features keeps training cost within ~2% of the baseline (7.3 s → 7.5 s per step; 51 → 52 eight-GPU hours), and the projectors add only 56 MiB of parameters that are discarded after training.
4D-WAM is evaluated on top of two different base models: FastWAM-Joint (RoboTwin 2.0, LIBERO, LIBERO-Plus) and Lingbot-VA (RoboTwin Clean2Rand, real world). Success rates in %.
| RoboTwin 2.0 | LIBERO | |||||||
|---|---|---|---|---|---|---|---|---|
| Model | Clean | Random | Avg. | Object | Goal | Spatial | Long | Avg. |
| π0.5 | 42.98 | 43.84 | 43.41 | 98.2 | 98.0 | 98.8 | 92.4 | 96.9 |
| WorldVLA | 42.50 | 32.20 | 37.35 | 96.2 | 83.4 | 87.6 | 60.0 | 81.8 |
| X-VLA | 72.80 | 72.84 | 72.82 | 98.6 | 97.8 | 98.2 | 97.6 | 98.1 |
| Motus | 88.66 | 87.02 | 87.84 | 99.8 | 96.6 | 96.8 | 97.6 | 97.7 |
| X-WAM | 89.80 | 90.70 | 90.25 | – | – | – | – | – |
| FastWAM-Joint (base) | 90.84 | 90.32 | 90.58 | 99.0 | 98.4 | 99.6 | 95.8 | 98.2 |
| 4D-WAM (ours) | 91.92 | 90.76 | 91.34 | 99.5 | 98.6 | 98.8 | 97.4 | 98.6 |
| Perturbation | FastWAM-Joint | 4D-WAM | Δ |
|---|---|---|---|
| Camera | 27.89 | 45.15 | +17.3 |
| Robot | 60.90 | 64.26 | +3.4 |
| Language | 89.26 | 90.63 | +1.4 |
| Light | 86.73 | 94.29 | +7.6 |
| Background | 49.26 | 57.71 | +8.5 |
| Noise | 51.09 | 69.08 | +18.0 |
| Layout | 74.82 | 79.21 | +4.4 |
| Average | 62.21 | 71.01 | +8.8 |
| Model | Clean | Rand. | Avg. |
|---|---|---|---|
| StarVLA | 58.1 | 10.6 | 34.3 |
| GR00T-N1.7 | 43.6 | 20.7 | 32.2 |
| π0.5 | 68.5 | 46.0 | 57.3 |
| Abot-M0 | 70.7 | 36.0 | 53.4 |
| FastWAM | 70.2 | 1.2 | 35.7 |
| Lingbot-VA (base) | 80.7 | 34.6 | 57.7 |
| 4D-WAM (ours) | 81.5 | 41.8 | 61.7 |
| Setting | Object | Goal | Spatial | Long | Avg. |
|---|---|---|---|---|---|
| Layer 16 | 99.8 | 97.0 | 98.2 | 96.8 | 97.9 |
| Layer 18 | 99.2 | 97.6 | 98.0 | 94.8 | 97.4 |
| Layer 20 (ours) | 99.6 | 98.6 | 98.8 | 97.4 | 98.6 |
| Layer 22 | 100.0 | 98.6 | 98.8 | 96.6 | 98.5 |
| Layer 24 | 99.0 | 98.4 | 98.0 | 95.8 | 97.8 |
| w/o Destination | 99.4 | 97.6 | 98.4 | 96.6 | 98.0 |
| w/o Motion | 99.8 | 97.2 | 99.0 | 96.0 | 98.0 |
| Metric | Baseline | 4D-WAM | 4D-WAM (Online) |
|---|---|---|---|
| Time per step (s) | 7.3 | 7.5 | 8.4 |
| Peak GPU memory (%) | 77.3 | 82.7 | 96.0 |
| Training time (8 GPUs, h) | 51 | 52 | 59 |
| Extra trainable params (MiB) | – | 56 | 56 |
| Extra inference cost | – | none | none |
Executing policies in simulation and comparing predicted future frames with ground truth on LIBERO-Plus, 4D-WAM improves PSNR 19.22 → 21.44 and SSIM 0.75 → 0.83 over FastWAM-Joint (Table 4, 100 episodes per perturbation type).
Three LIBERO rollouts, each showing the main view and the wrist view generated by the world model. 4D-WAM keeps the arm, the manipulated object and the target region coherent across time.
Figure 4. Four consecutive predicted frames on LIBERO-Plus. Red boxes mark poor predictions; green boxes mark high-quality ones. Row 1: FastWAM-Joint loses the arm from the second frame under camera blur. Row 2: 4D-WAM keeps tracking the microwave door despite occlusion. Row 3: while grasping the book, 4D-WAM preserves the book's position from its initial frame to its destination.
Two 6-DoF arms with 1-DoF grippers, one primary camera between the arms and two wrist cameras. 550 demonstrations across four tasks; Lingbot-VA as base model; 50 trials per task. Hover (or tap) a marker to see which task each object belongs to.
Hover a task to highlight its objects on the platform.
Four cylinders of different sizes are randomly arranged in front of their slots. The robot must identify each cylinder by size and insert them in ascending order until all four are placed.
A crumpled towel is placed on the table. The two arms grasp opposite ends, pull it open, and lay it flat on the tabletop.
A tray sits next to a closed toolbox. The right arm opens the toolbox, inspects the items inside one by one, locates the wrench, and places it in the tray.
Several lights sit on a running turntable. While it rotates, the robot must identify the illuminated red light, grasp it, and place it on the table.
| Lingbot-VA | 4D-WAM | |||
|---|---|---|---|---|
| Task | Prog. | SR | Prog. | SR |
| (a) Cylinder | 28.1 | 0.0 | 50.6 | 2.0 |
| (b) Towel | 2.5 | 0.0 | 7.5 | 4.0 |
| (c) Wrench | 6.7 | 0.0 | 8.3 | 12.0 |
| (d) Light | 25.0 | 4.0 | 23.8 | 4.0 |
| Average | 20.3 | 1.0 | 31.8 | 5.5 |
@article{yang2026fourdwam,
title = {4D-WAM: Infusing Spatiotemporal Awareness into World Action Models through Trajectory Fields},
author = {Yang, Lishan and Song, Wenxuan and Wang, Xi and Sheng, Pingyue and Fang, Zheng and
Zhou, Ziyang and He, Junjie and Yan, Haodong and Chen, Jiayi and Sun, Nan and Sun, Qiao and
Liu, Lingqiao and Wang, Yan and Gao, Yuxiang and Dayoub, Feras and Li, Haoang},
journal = {arXiv preprint arXiv:2608.08023},
year = {2026}
}