Diffusion policies are increasingly built by composing \(K\) components and summing their score predictions into a single composed score to create modular and adaptable systems. In practice, however, the components are not supervised individually during training, so they end up redundant. As such, we introduce a drop-in score-decorrelation loss, added to the diffusion objective with weight \(\lambda\), that penalizes overlap between the components' noise predictions \(\epsilon_{\theta_i}\):
\[ \mathcal{L}_{\text{decorr}} = \mathbb{E}\Big[\tfrac{2}{K(K-1)} \sum_{i<j} \big(q_i^\top q_j\big)^2\Big], \qquad q_i = \frac{|\epsilon_{\theta_i}|}{\|\epsilon_{\theta_i}\|_2 + \delta} \]In the 2D navigation task on the right, as the slider is pushed and \(\lambda\) increases, the two factors align with the coordinate axes (up/down - left/right) and become interpretable. They can then be recombined, with three copies of Factor 1, one per axis, solving out-of-distribution 3D navigation tasks zero-shot.
success
Loading the interactive figure…
When a factorized policy is trained on multi-stage tasks, adding decorrelation lets it complete more of each composite's sub-goals, even in rollouts that fail to finish the full sequence. This indicates the gains come from reusable factorized structure rather than memorized composite trajectories.
On a real ARX R5 arm task, DP succeeds in 10/25 rollouts, FDP without decorrelation in 15/25, and FDP with decorrelation in 19/25.
Overhead
Wrist camera
@article{laprevotte2026learning,
author = {Laprevotte, Alec and Chen, Haonan and Han, Xiaoshen and Du, Yilun},
title = {Learning to be Different: Component Specialization for Compositional Diffusion Policies},
journal = {arXiv preprint},
year = {2026},
}