The claim
No simulator. No mocap. No demonstrations to imitate. Policies are trained from scratch inside a learned world model — in image space, from raw pixels — and deployed zero-shot. No markers, nothing clamped in place, and no reset that puts things back where they started.
Three real robots
Same recipe, three embodiments
Roadmap
Outline
- Motivation
- Method
- Does the world model hold up?
- Experiments
- What this buys, and what it costs
The one-line thesis: forward accuracy and backward tractability are different jobs —
stop asking one model to do both.
01
Motivation
RL works beautifully — as long as you have a simulator you trust.
01 · Motivation
Simulated RL has been extraordinarily good to locomotion
What works
- Massively parallel GPU simulators → billions of samples are cheap.
- Extreme parkour, ANYmal parkour, robot parkour learning, CaT, SoloParkour.
- Rigid-body dynamics + actuator models transfer well enough for legged locomotion.
Why it stops there
- Locomotion is mostly ego-body dynamics — the environment barely changes.
- Manipulation changes the environment with every action: an infinite extra degree of variability.
The sim2real recipe is not general — it is a recipe that happens to fit the physics
of walking.
01 · Motivation
Two walls — and they trap you between themThe first pushes you into a simulator. The second says the simulator won't get you contact-rich manipulation.
Wall 1 — the real world doesn't parallelise
- Model-free RL wants tens of millions of interactions.
- Real-world RL exists — it just doesn't parallelise: one actor per robot, each costing another machine and another patch of floor.
- And each one is slow: hardware wear, a human resetting after every attempt, someone watching throughout.
→ so you have to train in a simulator.
Wall 2 — soft bodies and multi-contact don't sim2real
- Soft bodies and rich multi-contact are hard to simulate at all.
- What you can simulate transfers poorly — the parameters are too hard to identify.
- Granular gravel. A can pulled by its tab. A T-shirt. The semi-soft handle on our G1's box.
→ and a hand-crafted physics simulator won't get you this task.
The way out is to stop writing the environment down:
learn it instead. Train a world model on real robot data,
and do the RL inside that.
01 · Motivation
World models are not a free lunchOnce your simulator is a 1.4B-parameter diffusion transformer, the RL algorithm has to pay for every single sample.
Zeroth-order RL (PPO, SAC…)
- Treats dynamics as a black box — robust to model error.
- Needs vast numbers of rollouts.
- Every rollout step is now a diffusion sample. Cost explodes.
First-order (FoG / analytic gradients)
- Backprops through the dynamics — far lower gradient variance, high data efficiency.
- Requires ∂f/∂s, ∂f/∂a of the model.
- Backprop through a 1.4B diffusion transformer, unrolled for a horizon? Intractable.
The bind: the model that is accurate enough to replace the simulator is
exactly the model you cannot afford to differentiate — or to sample enough times.
01 · Motivation
The observation the paper turns on
The forward pass needs…
- Long-horizon accuracy
- Visual fidelity, contact detail
- Object permanence under occlusion
- Generalisation off the data manifold
→ a big model.
The backward pass needs…
- Only local derivatives
- Accuracy over a single step
- Low dimensionality
- Speed, and nothing else
→ a small model.
These are different specifications. Use two models.
02
Method
Decoupled forward–backward model-based policy optimisation, in image space, without a simulator.
02 · Method · background
What first-order model-based RL isLearn a model of the dynamics from data, roll the policy forward through it, then differentiate the total reward back through that same rollout.
- The dynamics model is learned from data — that is the “model-based” part. Being a neural network, it is differentiable, so the whole rollout is one differentiable graph.
- The catch: every backward step needs the model's derivatives, i.e. how the next state moves when you nudge the state or the action.
Fine for a small model. Very impractical for a 1.4 B-parameter
diffusion transformer unrolled over a horizon — which is where this talk starts.
02 · Method · background
DMO: forward and backward need not be the same modelFirst Order Model-Based RL through Decoupled Backpropagation — Amigo, Khorrambakht, Chane-Sane, Mansard, Righetti, CoRL 2025.
- In DMO the forward pass was a high-fidelity simulator, which generates the trajectory. A small learned model was used only for the backward pass — and only at the states the simulator produced.
- Because it is re-anchored every step, the learned model is never trusted to say where you are, only how things move right here — so its own drift never accumulates.
Forward accuracy and backward tractability are now optimised independently.
02 · Method · this paper
Our extension: kill the simulator, work in image space
Global model — \(f^{\rightarrow}\)
- A large diffusion / flow-matching world model in pixel space, trained offline on real robot data.
- DIAMOND for Push-T; a DreamerV4-style latent-diffusion transformer for the ego-centric tasks.
- Frozen during RL — it only ever runs forward.
Local model — \(f^{\leftarrow}_\phi\)
- A lightweight DreamerV3-style RSSM in a low-dimensional latent space.
- Acts as both the local dynamics and a local reward model.
- It supplies every gradient, so nothing is ever backpropagated through the 1.4 B-parameter diffusion model.
Prior FoG work relied on a simulator for the forward pass. Prior diffusion world
models relied on zeroth-order optimisation. We remove both crutches at once:
large-scale pixel-space world models become usable for first-order MBRL.
02 · Method · pipeline
The whole pipeline, four steps
02 · Method · step 1
Step 1 — play data, not demonstrations
- An unstructured, high-entropy action–image play dataset, teleoperated on the real robot.
- No clean temporal segmentation into tasks. No frequent environment resets.
- Cheaper to collect than demonstrations, and reusable across tasks.
- Task intent is labelled on that data afterwards — that's what the reward model learns from.
4 h
Push-T
12 h
Go2 push cube
20 h
G1 grasp & lift
Everything downstream — dynamics, rewards, the policy — comes out of these hours of
unstructured real-robot play. No simulator is ever used.
02 · Method · step 2
Step 2 — the global world model
Push-T · DIAMOND
- params
- 4 M
- gen. res
- 64 × 64
- diff. steps
- 3 (4 conditioning steps)
- trained on
- 1 × RTX 4090
Chosen for long-horizon prediction accuracy and fast inference on an easier-to-model task. Push-T needs none of the machinery on the right.
Push Cube & G1 · DreamerV4-style
- params
- 1.4 B
- arch
- transformer latent diffusion
- gen. res
- 128 × 128 (Go2) · 256 × 256 (G1)
- context
- 96 steps, patch size 16
- diff. steps
- 4
- trained on
- 24 × H200 (Go2) · 64 × H100 (G1)
And what makes this one affordable to roll out
- Shortcut learning — few diffusion steps, so a rollout step is cheap.
- Causal diffusion forcing — KV-cache-friendly real-time recurrent generation.
- Proprioception tokenized alongside pixels — G1 only; the Go2 model is pixels alone.
02 · Method · step 2b
Rewards in pixel space
Push-T (DIAMOND) — contrastive reward
- A goal-conditioned energy, scoring a frame by how close it is to the goal.
- Trained on random play and, optionally, demonstration videos.
- Following Rank2Reward.
→ no manual labelling. The supervision is how far a frame sits from the goal in time, which the play data already tells you.
Ego-centric tasks — demonstration-intent head
- A lightweight head on the world model itself.
- Classifies whether a frame belongs to the task-demonstration distribution.
- We hand-labelled the play data to train it.
→ needs manually labelled data — the one the paper lists as a limitation.
Both rewards are learned — the difference is whether a human had to label the
data. That's one of several places we're still in the loop: the play data is
teleoperated, and patching round-trips through the real robot.
A VLM-labelled reward would remove this one.
02 · Method · step 3
Step 3 — the local RSSM
- DreamerV3-style RSSM acting as both the local latent dynamics and a local reward model — distinct from the global reward model of the previous slide.
- Pre-trained offline on the play dataset, with the global reward model providing ground truth.
- An image encoder projects observations to a compact latent — policy gradients never flow through pixels or the encoder.
- Fine-tuned online during RL on trajectories from the global model, so it stays accurate under the current policy's action distribution.
The requirement that changes everything
Unlike DreamerV3 — where the RSSM must imagine entire trajectories — here it only has to be accurate over a single forward step. Its Jacobians are always evaluated at encodings of precise images from the global model.
02 · Method · step 4
Step 4 — the DMO loop
for epoch = 1 … N:
# 1 · backward-model learning
\((o,a,o') \sim \mathcal{B}\)replay buffer
\(\phi \leftarrow \phi + \alpha_\phi \nabla_\phi \mathcal{L}_{f^{\leftarrow}}(\phi)\)DreamerV3 model loss
# 2 · imagined rollout, horizon H
for h = 1 … H:
\(l_h \leftarrow \text{encoder}(o_h)\)latent
\(a_h \leftarrow \pi_\theta(l_h)\)
\(o_{h+1} \leftarrow f^{\rightarrow}(o_h,a_h)\)FORWARD — global diffusion model, in pixel space
\(r_h \leftarrow r(o_{h+1},a_h)\)global reward model — supervises \(\hat R_\phi\) and \(V_\psi\)
\(\hat R_h \leftarrow \hat R_\phi(l_h,a_h)\)local reward head
\(\text{total\_reward} \leftarrow \text{total\_reward} - \hat R_h\)
# 3 · losses
\(\mathcal{L}^{\text{DMO-SAPO}}_\pi(\theta)\) from \(\text{total\_reward},\; V^{\pi_\theta}_\psi(l_{H+1})\)Eq. 1 — entropy-regularised return
\(\mathcal{L}_V(\psi)\) from \(l_1 \ldots l_H\)Eq. 2–3 — TD against \(\lambda\)-returns
# 4 · backward pass — the DMO substitution
\(\partial f^{\leftarrow}_\phi/\partial l,\; \partial f^{\leftarrow}_\phi/\partial a\) at \((l_{h+1},a_{h+1})\)stand in for \(\partial f/\partial o,\; \partial f/\partial a\)
\(\theta \leftarrow \theta + \alpha_\theta \nabla_\theta \mathcal{L}^{\text{DMO-SAPO}}_\pi(\theta)\)
\(\psi \leftarrow \psi + \alpha_\psi \nabla_\psi \mathcal{L}_V(\psi)\)
The global model is frozen and only ever runs forward. The local model is the
only thing anyone differentiates.
03
Does the world model actually hold up?
How good is it, where does it break, and what we do about it.
03 · Fidelity
Real vs local vs global, six seconds of G1 manipulationBoth models are initialised from the same first real frame and driven by the same recorded actions — 60 steps at 10 Hz. They diverge from there.
Real camera
Local · DreamerV3 RSSM
Global · DreamerV4 diffusion
The local model renders at 64×64 natively and is upscaled here without smoothing, so what you see is what the gradient would see. Watch the box: its shape and the detail of the contact dissolve, while the global model keeps box, hand and contact coherent to the end.
This is the empirical justification for the split: the local model cannot carry
the forward pass — and it doesn't have to.
03 · Fidelity
Inside the world modelClips from a single long interactive teleoperation episode run entirely inside the learned G1 world model. Native 256×256. The overlaid scalar is the model's own predicted reward.
Each clip is seeded from a different starting frame of the same episode. Nothing here is a real camera — it is all generated, in real time, conditioned on the operator's actions.
03 · Keeping it honest
The policy will find your world model's bugs
Refinement 1 — immersive probing
- A human explores the world model in VR, driving it with the same action space the policy uses.
- Find the poorly modelled regions, collect real data there, fine-tune.
Quest 3 · Go2
Pico + IMUs · G1
Only possible because the model's action space is one a human can reproduce.
Refinement 2 — exploit patching
- Deliberately train policies that exploit the model for artificially high return.
- Deploy those exploiters on the real robot to harvest targeted data.
- Fine-tune the world model on it — the hallucination disappears.
The RL agent becomes an automatic bug-finder for the simulator.
03 · Keeping it honest
Physical hallucinations, before and after patchingThe same exploit policy and the same actions, rolled out in the world model before and after it was fine-tuned on real data — collected by deploying that very policy on the robot.
Exploit 1 — the box teleports into the grasp
original
patched
Exploit 2 — the box is dragged by an invisible force
original
patched
In the original model the policy found physics it could break for reward. After patching, the hand comes up empty and the box stays where it is.
04
Experiments
Three tasks, two baselines, two ablations — and real hardware for every number that matters.
04 · Setup
Fewer, harder, realWe deliberately favour a small number of representative real-robot deployments over a large suite of simulated tests that would miss exactly the complexity we're trying to capture.
1 · Push-T
- robot
- Flexiv Rizon-10S, 7-DoF
- action
- tool vertical + horizontal velocity
- state
- stack of 4 frames, 5 Hz
- data
- 4 h play
- global WM
- DIAMOND, 4 M
Easier to model, non-trivial motion planning. Full observability.
2 · Ego-centric push cube
- robot
- Unitree Go2
- task
- push a cube into a soccer goal with its body
- action
- body velocity cmds to a low-level RL locomotion policy, 5 Hz
- data
- 12 h play
- global WM
- 1.4 B transformer diffusion
Adds walking dynamics, a large room, and partial observability.
3 · Humanoid grasp & lift
- robot
- Unitree G1 + BrainCo dexterous hand
- task
- grasp a box by its flexible handle, lift it off the table
- WM action
- 19-DoF upper body + 3-DoF locomotion
- policy
- 11-DoF (right arm + hip yaw/roll/pitch) at 10 Hz
- low level
- NVIDIA Sonic body tracking at 50 Hz
- data
- 20 h play
Adds a dexterous hand, a semi-soft handle no rigid-body simulator can model, and a moving base — the torso pitches, rolls and yaws while the hand reaches.
04 · Setup
What we compare against
Baselines
- PPO — model-free RL trained directly on image observations, inside the same world model.
- ACT — Action Chunking Transformer, a strong behaviour-cloning representative, trained on the demonstration subset.
Ablations
- No Diffusion — use the DreamerV3 RSSM for the forward pass too, so one model does the forward rollout and the backward gradients.
- No RSSM Finetuning — freeze the local model during RL instead of tracking the policy.
The two ablations isolate the two halves of the claim: do you need a big
forward model, and do you need the local model to keep up with the
policy?
04 · Results
Sample efficiency and wall-clock efficiency
The right-hand panel is the one that matters. First-order gradients don't just save
samples — they save the diffusion model's compute, which is where the money goes.
04 · Results · hardware
Real-robot success, out of 10 trials
Push-T · real robot
| Method | Successes |
|---|---|
| DMO (ours) | 9 / 10 |
| PPO | 1 / 10 |
| No Diffusion | 0 / 10 |
Under a relaxed criterion — counting runs where the T briefly passes through the correct pose without stopping — PPO reaches 4/10.
Go2 push cube · real robot
| Method | Successes |
|---|---|
| DMO (ours) | 9 / 10 |
| BC (ACT) | 6 / 10 |
| PPO | 4 / 10 |
| No Diffusion | 0 / 10 |
BC's 6/10 counts reaching the immediate pre-goal vicinity — it never fully pushes the cube in. Trajectory quality on the next slide.
G1 grab box · real robot
| Method | Successes |
|---|---|
| DMO (ours) | 8 / 10 |
| BC (ACT) | 7 / 10 |
| PPO | 1 / 10 |
| No Diffusion | 0 / 10 |
BC is genuinely competitive here — the task is well covered by demonstrations. On Push Cube, to its left, it comes apart.
04 · Results · rollouts
Go2 push cube — DMO vs PPO vs BC
DMO (ours)
PPO
BC (ACT)
Scene 1 — the cube starts in view; the policy goes straight to the goal.
04 · Results · hardware
Push cube: it's not just success rate, it's how
Trajectory quality · N = 10 real-robot episodes · metrics over successful episodes only
| Method | Success ↑ | Steps to success ↓ | Straightness ↑ | Curvature (rad/m) ↓ |
|---|---|---|---|---|
| DMO (ours) | 90 % | 177.2 ± 52.2 | 0.520 ± 0.266 | 0.692 ± 0.206 |
| BC (ACT) | 60 % | 371.0 ± 94.0 | 0.330 ± 0.147 | 0.761 ± 0.151 |
| PPO | 40 % | 173.5 ± 20.3 | 0.233 ± 0.182 | 1.222 ± 0.306 |
| No Diffusion | 0 % | — | — | — |
Straightness = displacement / path length (1.0 is optimal). Curvature = yaw change per metre.
DMO is both the most reliable and the smoothest; PPO's successes are erratic and jerky.
BC's "successes" count reaching the immediate pre-goal vicinity — it never fully enters the goal.
04 · Results · rollouts
Go2 — when the cube starts out of view
DMO (ours)
PPO
BC (ACT)
Scene 2 — discovery behaviour. This is the case that separates the methods.
04 · Results · emergent behaviour
RL discovers active search; BC just backs up
Fig. 11 of the paper: the ACT policy executes a naive backward retreat — brittle, and it
drags the cube further away if the cube is behind it. The DMO policy retreats to the
goal area and performs a full 360° yaw scan. Nobody demonstrated that.
04 · Results · rollouts
Push-T on hardware
Policy trained entirely inside a 4 M-parameter DIAMOND world model built from 4 hours of play. Deployed with no fine-tuning.
04 · Results · rollouts
G1 grasp & lift — DMO vs ACT vs PPO
DMO (ours)
ACT
PPO
Scene 1. Same initial condition, three policies, all deployed zero-shot from the world model.
04 · Results · rollouts
G1 grasp & lift — a second initial condition
DMO (ours)
ACT
PPO
Scene 2.
04 · Results · negative result
We tried planning with the world model directly. It doesn't work.
- CEM model-predictive control on the Go2 task, using the same global world model and reward model.
- Aggressively optimised: diffusion steps cut 4 → 1; a single outer-loop CEM refinement per executed action; chunk-diffusion mode enabled by diffusion forcing.
- Still prohibitive for real-time control.
≥ 16
H200 GPUs required for real-time inference with 128 parallel rollouts
Large world models give accurate forward predictions, but test-time planning with
them is out of reach — so you have to memorise the behaviour into a policy.
Which is what DMO does.
05
What this buys, and what it costs
Limitations, compute, and where it goes next.
05 · Wrap
What the framework actually delivers
- Simulator-free RL from real robot data — visually realistic dynamics learned directly from play, so there is no sim2real gap to bridge.
- Tractable first-order MBRL at scale — decoupling gives high-fidelity forward prediction and cheap analytic gradients, despite pixel-space inputs.
- Zero-shot transfer with high success rates on three real platforms, trained from scratch with no BC guidance or regularisation.
- A safe proxy for real-robot training — the policy does its exploring inside the model, not on the hardware. No damage, no unsafe behaviour during learning.
- Behaviour beyond the demonstrations — active search, full goal completion, robustness to a controller swap.
05 · Wrap
Limitations
- Global world models are trained from scratch. The play data must cover a wide range of physical interactions in image space.
- The policy is trained from scratch too — deliberately, so that what you see is genuine learning inside the world model, not a pretrained policy being nudged. It is not the efficient route: initialising from a BC-pretrained policy — a VLA, say — and fine-tuning it with our method should reach competence with far less exploration.
- Rewards are hand-labelled — we labelled the play data ourselves for Push Cube and Grab Box.