CLOUD: Contrastive Learning of Unsupervised Dynamics

Jianren Wang
Yujie Lu
Hang Zhao
Tsinghua University
CORL 2020
[Download Paper]
[GitHub Code]

Developing agents that can perform complex control tasks from high dimensional observations such as pixels is challenging due to difficulties in learning dynamics efficiently. In this work, we propose to learn forward and inverse dynamics in a fully unsupervised manner via contrastive estimation. Specifically, we train a forward dynamics model and an inverse dynamics model in the feature space of states and actions with data collected from random exploration. Unlike most existing deterministic models, our energy-based model takes into account the stochastic nature of agent-environment interactions. We demonstrate the efficacy of our approach across a variety of tasks including goal-directed planning and imitation from observations.

CLOUD Architecture

Our framework consists of four learnable functions, including a forward dynamics model F(·), an inverse dynamics model I(·), a state representation model g(·) and an action representation model q(·). We propose to learn these four models jointly via contrastive estimation.

Paper and Bibtex

Jianren Wang, Yujie Lu, Hang Zhao.
CLOUD: Contrastive Learning of Unsupervised Dynamics
In CORL 2020.

[Bibtex] [Paper] [ArXiv]
    Author = {Wang, Jianren and Lu, Yujie and Zhao, Hang},
    Title = {CLOUD: Contrastive Learning of Unsupervised Dynamics},
    Booktitle = {CORL},
    Year = {2020}


We would like to thank members of the CMU Rpad and MIT CSAIL for fruitful discussions. The work was carried out when JW/FZ was at CMU and HZ was at MIT. This work was supported by PanGU Young Investigator Award to JW.