Learning Social Navigation from Internet Videos in the Policy State Space

Overview video

Three minutes, end to end

Video pipeline, replay simulator, Arena transfer, and the Go2 deployment.

Motivation

Two bottlenecks, both made of hand-built content

Social-navigation policies are trained in simulation, because learning them on real robots is costly and potentially unsafe. But the simulators themselves are expensive in two independent ways, and each caps a different kind of diversity:

The obvious fix — reconstruct real scenes from video — replaces the artist with a reconstruction, but keeps the expensive part. Gaussian splats, generated assets and neural renderers all have to render counterfactual observations: the moment the robot steps off the recorded camera path, the simulator must synthesise what it would have seen. Dynamic social scenes make this worse, since people now need appearance models and motion models too.

Simulation in pixel space

hours per scene · renders every step

Simulation in state space

4.5 min per 20 s clip · never renders

Source

LLMdesignervideo

Layouts, terrain and 3D assets are authored, or reconstructed into something renderable.

An ordinary monocular walking video. Nothing else.

World

geometryassetshuman model

A 3D scene plus scripted people, whose behavior is whatever the chosen model produces.

A metric traversability map and the trajectories people actually walked.

Observation

Rendered pixels — re-synthesised for every counterfactual robot pose.

The robot-frame state itself — a rigid transform away, at any pose.

Policy acts on

pixels

state

traversable non-traversable unobserved people camera path robot

Key idea

A navigation policy does not need pixels

Local social navigation depends on two things: where the robot can traverse and how nearby pedestrians move. Both are geometric. Neither needs appearance. So instead of reconstructing a scene that can be rendered, we recover only those two quantities and define the simulator's forward dynamics directly on them.

That choice is what makes counterfactual robot motion cheap, because each layer transforms in a way that needs no model:

World time follows the recording; the robot pose evolves independently from the policy's velocity commands. The two only interact when the recorded state is transformed into the robot's current frame — which is defined for any pose. So the robot can stop, turn, or leave the demonstrator's path entirely and still observe a consistent world state.

Method

The pipeline, stage by stage

Every stage below runs on ordinary 4K walking-tour footage with pretrained perception models. One 20 s clip (200 frames at 10 fps) takes 4.5 minutes on a single A100.

a Walking video

First-person 4K walking tours. The only input.

b Data processing

People (YOLO11 + BoT-SORT), walkability (SAM-TP) and metric depth + camera poses (Depth Anything 3), fused into a gravity-aligned world frame.

c Metric replay step

World time follows the recording; the robot pose integrates from the policy's command with a unicycle model. Map and people are transformed into the robot frame each step.

0.1 m cells±10 m10 Hz ~1,000 steps/s

d Policy

zB_t h_t^1:Kg_t^R a_t−1

transformer · 4 blocks
203 tokens · d = 192
1.6 M parameters

actor → Beta (v, ω)critic V

180 angular cone tokens + up to 20 pedestrian tokens + goal + previous action. RoPE on the cone tokens only; pedestrian tokens stay permutation-invariant.

Trained with PPO from scratch in the replay simulator — 32 parallel environments, reward on goal progress with collision termination.

e Deployment

The same state is rebuilt online from RGB-D and odometry, so the video-trained policy transfers with no modification.

What the simulator actually checks

The map is accumulated causally: the map the robot sees at video time t contains only cells the camera had observed by t, never ground revealed later. On top of that state the simulator enforces the usual robot constraints —

• Static collision when the 20th percentile of map values under the 0.7 × 0.5 m footprint falls below 0.1.
• Pedestrian collision against 0.25 m discs at each person's replayed position.
• Episode start jittered in a 2 m forward half-disc, goal at least 15 m ahead — so the policy cannot simply replay the camera trajectory.

Training is PPO from scratch, 32 parallel environments, at roughly 1,000 environment steps per second on one RTX 5090 — a 100 M-step run takes 26 hours.

Corpus

44 walking tours, 29 cities, no overlap between splits

Result 1 — unseen videos

92.6% on 2,155 episodes from cities it has never seen

Closed-loop in the state-space simulator. The robot starts at the demonstrator's pose at t = 2 s and must reach a point at least 15 m ahead, among the recorded people.

For scale: replaying the recorded camera trajectory itself — a real person walking that street — satisfies the same collision criteria in only 72% of episodes, with 21% pedestrian and 7% static collisions. The walker brushes past people far closer than the robot's rules allow.

Result 2 — the central claim

Replaying people beats modelling them

The simulator's main approximation is that pedestrians follow their recorded trajectories instead of reacting to the robot. If that were a serious limitation, a reactive model should train a better policy. It does not.

One held-out episode, five policies

A street in Belgrade, from a video none of these policies has seen. Every clip is the same episode with the same deterministic start; only the training setup differs. Left panel is the policy's egocentric observation, right panel the world map with the path it took.

The cross-evaluation that settles it

Training on replay could simply be tuning a policy to a quirk of replay. So each policy is scored under all three pedestrian dynamics — the recorded replay, a social-force crowd, and a Trajectron++ crowd. A policy that only works in its own training dynamics would show a strong diagonal. The replay-trained policy is the best or tied-best in every column:

Trajectron++ matches replay without improving on it, and costs more to get there: it is an extra model that has to be trained (here from scratch on our own tracks, with no test video touching it) and it drops the simulator from about 1,000 steps/s to 800. Social force is clearly worse. Direct replay is the simplest of the three and the most robust.

More video is what helps

If the recorded crowds are the source of the policy's competence, then adding videos should buy competence — and specifically it should buy pedestrian safety rather than general path quality. It does:

Result 3 — independent benchmark

Transfer to Arena with no fine-tuning

Arena / Isaac Sim, four UrbanVerse street scenes, 160 scripted scenarios with reactive social-force pedestrians. A Clearpath Jackal with an OAK-D camera navigates between goals 18–22 m apart. Our policy is dropped in at 10 Hz, unchanged.

Against HEIGHT, the strongest baseline, success improves by 6.2 points and collisions fall from 25.0% to 15.0% — both pedestrian contacts (13.8% vs. 19.4%) and wall contacts (1.2% vs. 5.6%). And the gap is not bought by timidity: our policy spends less time in the personal and intimate zones than HEIGHT while succeeding more often. It maintains progress through dense crowds, including the brief close passes that are normal in the recorded videos.

The advantage grows with crowd density

Result 4 — physical robot

19 out of 20 trials on a Unitree Go2

No policy fine-tuning, no global map or planner. Traversability is built online from RGB-D and odometry; pedestrians are detected and tracked from LiDAR with a YOLO11 prior.

The failure mode of the baseline is instructive. MPPI treats a detected person as an occupied costmap cell, so it holds its course until someone is already close and then avoids late. Both learned policies receive pedestrian velocities, so they anticipate the encounter and slow or step aside before the crossing point. The replay-trained policy's single failure is indoors, where the 0.5 m/s hardware speed limit — far below the speeds it selects during training — leaves it too little time to clear an approaching person's path.

Limitations

What this representation cannot do

• Planar ground. The representation assumes approximately planar ground within each 20 × 20 m local window; substantial curvature or elevation change distorts the BEV projection.
• Only people move. Everything else in the scene is treated as static, which suits pedestrian paths but makes moving vehicles and bicycles a source of map error — and a contributor to failures in mixed-traffic scenes.
• Perception is the floor. Pedestrian states depend on human-pose detection and depth; detection errors and sensor noise degrade velocity estimates and hence collision avoidance.
• Partial coverage. The robot can move into regions the source video never observed. We measure this rather than assume it away: the policy's observed-cell coverage is 21% against the recorded walker's 23%, and it travels 15.2% of its path on unsupported space against the walker's 11.2%.