arXiv 2608.08023 · cs.RO · 2026

4D-WAM: Infusing Spatiotemporal Awareness into World Action Models through Trajectory Fields

1University of Adelaide    2HKUST (Guangzhou)    3COCO Matrix    4Tsinghua University    5Fudan University
*Equal contribution  ·  †Project leader  ·  ‡Corresponding authors
Overview of 4D-WAM: (a) training recipe aligning World Action Model features with Trace Anything 4D features through destination and motion alignment; (b) success-rate gains over baselines in-distribution and out-of-distribution; (c) video prediction comparison.

Figure 1. (a) The 4D-WAM training recipe: intermediate DiT features of a World Action Model are aligned with 4D trajectory-field features from a frozen Trace Anything teacher through motion alignment and destination alignment. (b) 4D-WAM consistently outperforms its base model, with particularly large gains under out-of-distribution shifts. (c) It also yields more faithful future-frame prediction.

+8.8pts
LIBERO-Plus (OOD)
62.2 → 71.0 avg. success, 7 perturbations
+7.2pts
RoboTwin Clean2Rand
34.6 → 41.8 under domain randomization
+11.5pts
Real-world progress
4 long-horizon tasks on ARX LIFT2
0overhead
At inference time
Alignment modules are removed after training
Abstract

Videos are 2D projections of a dynamic 3D world

Building on recent advances in world models, World Action Models (WAMs) jointly model video prediction and action generation. However, they typically represent videos in 2D pixel space, creating a representation gap with the 3D space in which robotic actions are executed. Recent 3D approaches introduce 3D information, but fail to fully exploit the dynamics of 3D structures.

In this work, we propose 4D-WAM, a model-agnostic training strategy that injects spatiotemporal knowledge from 3D trajectory fields into WAMs through representation alignment. To this end, we introduce two complementary objectives: (1) motion alignment, which aligns temporal feature variations across adjacent frames and encourages the model to build local 4D awareness during training, and (2) destination alignment, which guides the model to infer the final destination from the source frame by minimizing the gap between their attention-like similarity distributions. Together, these objectives provide both local motion supervision and long-horizon goal guidance, enabling WAMs to learn trajectory-level spatiotemporal representations.

Extensive in-distribution and out-of-distribution experiments across different base models demonstrate improvements in spatial understanding, execution precision, robustness, generalization, and versatility.

Method

Align what changes, not what is

Directly matching WAM features to 3D/4D foundation-model features — the recipe that works for VLAs — consistently hurts WAMs. The two representation spaces are simply too different. 4D-WAM instead aligns their temporal differences and their source-to-destination correspondence, which describe the same underlying motion regardless of feature space.

Alignment process of 4D-WAM: WAM DiT tokens and Trace Anything tokens are projected, latent differences are matched with cosine similarity (motion alignment), and source-to-destination softmax distributions are matched with KL divergence (destination alignment).

Figure 3. Alignment process. Multi-frame video is encoded by the (trainable) WAM and the (frozen) Trace Anything teacher. Lightweight projectors map both into a shared space; (1) token-wise latent differences between adjacent frames are matched with cosine similarity, and (2) source→destination similarity distributions are matched with KL divergence. Everything added is removed after training.

Cosine similarity between representation spaces across five RoboTwin 2.0 tasks: pi0.5 vs. VGGT 3D features are consistently positive; Lingbot-VA vs. Trace Anything 4D features are consistently negative; their temporal differences are mixed.

Figure 2. Why naive alignment fails. Across five RoboTwin 2.0 tasks, (a) a VLA's (π0.5) hidden states are consistently positively similar to 3D features from VGGT, whereas (b) a WAM's (Lingbot-VA) hidden states are consistently negatively similar to 4D features from Trace Anything. (c) Their frame-to-frame temporal differences, however, share consistent motion patterns — the signal 4D-WAM aligns.

Objective 1

Motion Alignment

For each token, the change between adjacent frames in the WAM's projected DiT features is pulled toward the corresponding change in the teacher's 4D trajectory features. Absolute features may live in different spaces, but their frame-to-frame deltas describe one consistent motion process — object displacement, contact, and the physics of manipulation.

Lmotion = 1 − cos( Δhf , Δzf ),  Δhf = hf+1 − hf
Objective 2

Destination Alignment

The first frame is the source, the last frame the destination. For every source token, a temperature-scaled softmax over its similarities to destination tokens gives an attention-like distribution of "where this ends up". The WAM's distribution is matched to the teacher's with KL divergence, teaching goal-directed correspondence beyond local motion.

Ldest = KL( qz(·|i) ‖ ph(·|i) ),  ph(·|i) = softmax(⟨hsrc,i, zdest⟩ / τ)

Overall objective: L = Llatent + Laction + λmotion Lmotion + λdest Ldest, with both λ < 0.1. Alignment is applied at DiT layer 20. Pre-extracting the teacher's trajectory features keeps training cost within ~2% of the baseline (7.3 s → 7.5 s per step; 51 → 52 eight-GPU hours), and the projectors add only 56 MiB of parameters that are discarded after training.

Experiments

Small gains where benchmarks saturate, large gains where they don't

4D-WAM is evaluated on top of two different base models: FastWAM-Joint (RoboTwin 2.0, LIBERO, LIBERO-Plus) and Lingbot-VA (RoboTwin Clean2Rand, real world). Success rates in %.

In-distribution: RoboTwin 2.0 & LIBERO

Table 1 · Base model FastWAM-Joint · 50 RoboTwin tasks (500 randomized + 50 clean demos each); LIBERO four suites, 40 tasks.
RoboTwin 2.0LIBERO
ModelCleanRandomAvg.ObjectGoalSpatialLongAvg.
π0.542.9843.8443.4198.298.098.892.496.9
WorldVLA42.5032.2037.3596.283.487.660.081.8
X-VLA72.8072.8472.8298.697.898.297.698.1
Motus88.6687.0287.8499.896.696.897.697.7
X-WAM89.8090.7090.25–––––
FastWAM-Joint (base)90.8490.3290.5899.098.499.695.898.2
4D-WAM (ours)91.9290.7691.3499.598.698.897.498.6

OOD: LIBERO-Plus

Table 2 · Trained on LIBERO only, evaluated under seven unseen perturbations · base FastWAM-Joint.
PerturbationFastWAM-Joint4D-WAMΔ
Camera27.8945.15+17.3
Robot60.9064.26+3.4
Language89.2690.63+1.4
Light86.7394.29+7.6
Background49.2657.71+8.5
Noise51.0969.08+18.0
Layout74.8279.21+4.4
Average62.2171.01+8.8

OOD: RoboTwin Clean2Rand

Table 3 · Trained on 50 clean demos per task, evaluated with domain randomization · base Lingbot-VA. Best, second best.
ModelCleanRand.Avg.
StarVLA58.110.634.3
GR00T-N1.743.620.732.2
π0.568.546.057.3
Abot-M070.736.053.4
FastWAM70.21.235.7
Lingbot-VA (base)80.734.657.7
4D-WAM (ours)81.541.861.7

Ablation on LIBERO

Table 5 · Alignment layer and training objectives · base FastWAM-Joint.
SettingObjectGoalSpatialLongAvg.
Layer 1699.897.098.296.897.9
Layer 1899.297.698.094.897.4
Layer 20 (ours)99.698.698.897.498.6
Layer 22100.098.698.896.698.5
Layer 2499.098.498.095.897.8
w/o Destination99.497.698.496.698.0
w/o Motion99.897.299.096.098.0

Training cost

Table 7 · LIBERO, 25k steps, batch 512, 8× H100 80 GiB. Cached vs. online trajectory extraction.
MetricBaseline4D-WAM4D-WAM (Online)
Time per step (s)7.37.58.4
Peak GPU memory (%)77.382.796.0
Training time (8 GPUs, h)515259
Extra trainable params (MiB)–5656
Extra inference cost–nonenone
Video Prediction

Better futures, not just better actions

Executing policies in simulation and comparing predicted future frames with ground truth on LIBERO-Plus, 4D-WAM improves PSNR 19.22 → 21.44 and SSIM 0.75 → 0.83 over FastWAM-Joint (Table 4, 100 episodes per perturbation type).

Predicted future frames: 4D-WAM (top) vs. baseline (bottom)

Three LIBERO rollouts, each showing the main view and the wrist view generated by the world model. 4D-WAM keeps the arm, the manipulated object and the target region coherent across time.

Side-by-side future prediction frames from FastWAM-Joint (left, errors in red boxes) and 4D-WAM (right, correct regions in green boxes) on three LIBERO-Plus scenes.

Figure 4. Four consecutive predicted frames on LIBERO-Plus. Red boxes mark poor predictions; green boxes mark high-quality ones. Row 1: FastWAM-Joint loses the arm from the second frame under camera blur. Row 2: 4D-WAM keeps tracking the microwave door despite occlusion. Row 3: while grasping the book, 4D-WAM preserves the book's position from its initial frame to its destination.

Real-World Experiments

Four long-horizon tasks on a bimanual ARX LIFT2

Two 6-DoF arms with 1-DoF grippers, one primary camera between the arms and two wrist cameras. 550 demonstrations across four tasks; Lingbot-VA as base model; 50 trials per task. Hover (or tap) a marker to see which task each object belongs to.

Real-world platform: ARX LIFT2 bimanual robot with three cameras; on the table a red base with four yellow cylinders, an open toolbox, a rotating turntable with lights (one red), and a folded blue towel.
task objects robot & sensors· hover or tap a marker

Tasks

Hover a task to highlight its objects on the platform.

a Bimanual · 4D-WAM rollout

Cylinder Insertion

Four cylinders of different sizes are randomly arranged in front of their slots. The robot must identify each cylinder by size and insert them in ascending order until all four are placed.

50.6%progress (vs. 28.1)
2.0%success (vs. 0.0)
b Bimanual · 4D-WAM rollout

Towel Flattening

A crumpled towel is placed on the table. The two arms grasp opposite ends, pull it open, and lay it flat on the tabletop.

7.5%progress (vs. 2.5)
4.0%success (vs. 0.0)
c Single arm · 4D-WAM rollout

Toolbox Wrench Localization

A tray sits next to a closed toolbox. The right arm opens the toolbox, inspects the items inside one by one, locates the wrench, and places it in the tray.

8.3%progress (vs. 6.7)
12.0%success (vs. 0.0)
d Single arm · 4D-WAM rollout

Rotating Light Pick-and-Place

Several lights sit on a running turntable. While it rotates, the robot must identify the illuminated red light, grasp it, and place it on the table.

23.8%progress (vs. 25.0)
4.0%success (vs. 4.0)

Real-world results

Table 6 · 50 trials per task · Progress = fraction of sub-tasks completed · SR = success rate.
Lingbot-VA4D-WAM
TaskProg.SRProg.SR
(a) Cylinder28.10.050.62.0
(b) Towel2.50.07.54.0
(c) Wrench6.70.08.312.0
(d) Light25.04.023.84.0
Average20.31.031.85.5
Citation

BibTeX

@article{yang2026fourdwam,
  title   = {4D-WAM: Infusing Spatiotemporal Awareness into World Action Models through Trajectory Fields},
  author  = {Yang, Lishan and Song, Wenxuan and Wang, Xi and Sheng, Pingyue and Fang, Zheng and
             Zhou, Ziyang and He, Junjie and Yan, Haodong and Chen, Jiayi and Sun, Nan and Sun, Qiao and
             Liu, Lingqiao and Wang, Yan and Gao, Yuxiang and Dayoub, Feras and Li, Haoang},
  journal = {arXiv preprint arXiv:2608.08023},
  year    = {2026}
}