Your critic may learn to give up before your robot learns to pick up. Make an RL curriculum too hard, too early, and repeated failures can teach the critic the same answer across a broad region: nothing good happens here. The difficulty is the lack of distinction between “closer” and “nowhere near.”
The curriculum determines what the critic gets to see
In a sparse-reward task, a policy that almost never succeeds provides little contrast in observed returns. The critic can assign similarly low values to many states the policy cannot yet solve. Those estimates may be reasonable for the current policy. But when rewards and bootstrapped targets offer little contrast, the learner gets less useful evidence about what to do differently.
The video compares two start-state arrangements: “all stages at once” and “frontier first.” At training iteration 2,000, the displayed full-task success rates are 0% and 92% in the runs shown. Watch the value maps beneath the robot: a broad region before grasping stays low on the left, while richer value structure develops on the right. This is a comparison of the displayed runs, not a multi-seed statistical result; the video does not separately measure recovery from unsuccessful training.
Train the next thing the agent can learn
A useful curriculum stays near the edge of current competence: enough success to reveal what works, enough failure to expose the next gap. Expand the training distribution as the policy improves, and keep evaluating on the full task so that solving the curriculum does not become the endpoint.
Starting too hard and making things easier later can carry an extra interaction cost. The agent has to discover better outcomes while the critic revises its old expectations. How much recovery costs depends on the algorithm, rewards, exploration and data distribution; a flat value map alone cannot establish it.
Let the agent learn what it can learn next. Save the existential crisis for the researcher.
Further reading
Reverse Curriculum Generation for Reinforcement Learning · Spinning Up: Policy Optimization