Introduction: Exploring an alternative, supervision free source of training data
Vision policies and Vision Language Action Models rely on human demonstration to learn to solve a task. While having its merits, this seems like a finite pool of data. This work aims to explore an alternative source of data and way of learning.
Unsupervised reinforcement learning, specifically Goal-conditioned RL methods such as Contrastive RL, has demonstrated some capabilities to learn through interacting with the world. We adopt Contrastive RL (CRL) and its variants, applying them on embodiments and tasks unseen in prior literature, with the aim to understand its limitations.
How well does Online-CRL handle different object poses?
For simple tasks, such as push or pick and place, CRL worked very well.
However, training one policy for just one scene is pretty expensive. What if we can randomise some parts of the scene, provide that information (e.g. the object pose of a fixed object) to the model, and let it learn to solve variations of the same task? Specifically for the square insertion task, we want to randomise not just the pose of the square peg (which can be moved by the arm), but also the rods (yellow and white) for each episode. Note that these rods cannot be moved by force; they are mounted to the table.
The critic loss panel is where the shortcut shows. The baseline drives its loss down to 0.46, lower than either CRTR run reaches, while its success stays at 0.004 — the objective is being minimised by something other than solving the task. Both CRTR runs settle at a higher loss, 1.6 and 1.9, and turn it into 32% and 50% success.
Turns out, these features that only change across each batch and do not change within an episode provide a shortcut to the contrastive objective. We found prior work (2025, Ziarko, CRTR) to face a similar issue in the offline setting; adopting their method mitigates the problem and improved performance, but does not fully solve the task. Scaling the model with CRTR = 2 seems to help.
Does CRL scale to multi-arm manipulation?
We started the work with a Franka arm, then tried it using an OpenArm (bimanual), and then we made a very long table with three SO-101s placed to cover each section of the table. When there is just one single object, CRL seems to work pretty well at passing that object to a given position.
Notice that while CRL can discover solutions to these problems, it seems to include lots of redundant movements in the robot's motion. This is exacerbated when we work with the SO-101.
What if we have more than one object?
This is where we start seeing CRL struggle. See the following tasks:
CRL was only able to handle the square task. For mug cleanup, it fails to explore opening the drawer and then picking and placing the mug. We include all free objects (objects that can move) as state input.
Another interesting finding is that for a single-arm robot, CRL could not learn to handle two objects. We recorded contact frequency to each object and found that it would prefer one over another. We later read (2024, Park, OGBench) and found that even in an offline setting, when multiple objects (i.e. the goal space consists of multiple objects' poses) are involved, CRL gives worse performance the more objects there are.
Our intuition is that CRL might be bad at exploring the environment and lack the combinatorial prior. More objects lead to a combinatorially more complex search space. In addition to that, the horizon of state-action transitions to a goal is also longer, as while the robot moves, its goal state remains stationary (e.g. reaching to open a drawer while the mug remains untouched).
Three pieces of evidence support the exploration and horizon hypothesis. First, when the goal space is the end-effector pose of the OpenArm, the robot thus receives dense goal space variations, and reached near perfect success online. Second, when the OpenArm is given two objects to push, unlike the Franka which has to prioritise one over another, it can simply move both at the same time, reaching a high success rate too. Finally, another work (2026, Yan, SHCRL) studied this more thoroughly and published on 21 July 2026, that, when the actor can simply move the cube through "telekinesis", i.e. random action leads to dense goal variation, the model learned to handle multiple objects (though still capping at 4 to 5 objects).
We found that CRL could discover many interesting behaviours
Without the human prior, CRL can learn interesting behaviours such as nudging with its own wrist, using a part of the robot that isn't the end-effector. Running with a different seed also leads to the policy converging at different behaviours. For instance, some runs lead to tossing behaviour via grasping, while some learned reliable tossing without even grasping the object!
The quality of the simulation also plays a significant role in the learned policy. The actor would discover physically unrealistic movements when the simulation permits it.
Limitation and future work
I haven't had a chance to port over more results, so expect this page to be updated in the future.
There are a few issues that stand between CRL and real world deployment, as a demonstrator should manage:
- While being able to produce novel behaviours, some movements are too aggressive or redundant, which adds strain on real machines.
- The observation space is misaligned between CRL and the downstream Vision Action Model; some skills don't seem learnable for the VA.
- The current range of tasks CRL can do is still limited to short horizon, i.e. it can't yet compete with human demos, or LLM driven discovery methods (2026, Lu, ASPIRE).