What CROSS does
A robot that comes back should know where it is: CROSS builds a map from a camera stream, saves it, and relocalizes later sessions in it.
- Session 1 Map The robot drives once. CROSS stores keyframes, their poses and how they connect.
-
On disk
Saved map
map.pklandmap.pkl.store/: the graph and its keyframe images, which a VLM or an agent can read too. - Every later session Relocalize and extend Starts with no pose, joins the map, reports poses in the map's coordinates, adds new places.
Between sessions the world changes: day turns to night, furniture moves, people walk by. CROSS keeps competing guesses of the pose until the evidence separates them and checks every loop closure, so one look-alike view cannot teleport the robot.
Install
One script installs everything into a local environment: PyTorch, CROSS, GTSAM and the model weights.
It needs Linux, an NVIDIA GPU with CUDA and about 10 GB of disk, and puts the environment in .venv
(Python 3.11, via uv, which it fetches if missing).
$ git clone https://github.com/jiaming-ai/CROSS.git
$ cd CROSS
$ bash install.sh --stereo
$ source .venv/bin/activate
Install options
| Option | Adds |
|---|---|
| (none) | the RGB-D modes with PnP (rgbd_wheel, rgbd_basalt) |
--stereo | VGGT-Omega and our fine-tuned checkpoint (Jamie-nus/VGGT-Omega-CROSS): stereo_wheel, stereo_basalt, stereo_imu, rgbd_ff_*, mono_wheel, mono_basalt |
--mono | --stereo + DPVO (built from source; needs nvcc matching PyTorch's CUDA), Depth Anything 3 and LightGlue: every mode |
bash install.sh --help lists the rest. Check the installation with python -m pytest -q
(a few minutes).
models/...) are relative to it. A few models download on first use, so the first run
needs internet; after that, export CROSS_HUB_OFFLINE=1 uses the cached copies.Quick start
Map once, relocalize later: the whole workflow on two recordings of the same office.
The recordings come from OpenLORIS-Scene,
a public dataset recorded by a wheeled robot with an RGB-D camera and wheel odometry. The
quick start uses rgbd_wheel, which every install includes.
-
Get two sessions
Download the office package from Hugging Face (seven sessions, 10.6 GB; unpacking needs about 22 GB free) with the benchmark's script, then convert two sessions to CROSS's folder layout:
Terminal# py7zr unpacks the archives; the PATH prefix finds uv if install.sh has just installed it $ PATH="$HOME/.local/bin:$PATH" uv pip install py7zr $ bash benchmark/datasets/download_openloris.sh data/openloris office1-1_7 $ python scripts/datasets/convert_openloris.py data/openloris/office1-1 data/posed/office1-1 $ python scripts/datasets/convert_openloris.py data/openloris/office1-2 data/posed/office1-2 -
Session 1: build the map and save it
Terminal$ python run.py data/posed/office1-1 --mode rgbd_wheel --save-map maps/office/map.pklA Rerun window shows the map as it grows (on a server, add
--no-viz). The map ismap.pkland the foldermap.pkl.store/beside it: copy or move them together. -
Session 2: relocalize in the saved map
Terminal$ python run.py data/posed/office1-2 --mode rgbd_wheel --load-map maps/office/map.pkl \ --save-poses office1-2_poses.txt --save-map maps/office/map.pklOnce CROSS has placed the session in the map,
office1-2_poses.txtgets one line per frame in the map's coordinates (TUM format:time x y z qx qy qz qw). Saving again stores the map extended by this session.
Common options
| Option | What it does |
|---|---|
--frames N, --start N | process N frames, starting at frame N |
--no-viz | no viewer window |
--config my.yaml, --set key=value | change the configuration (Configuration) |
--fast | smaller images on a weaker GPU (only stereo_basalt and stereo_dpvo have a fast setting) |
Outdoors, add the outdoor settings: --config configs/overlays/outdoor.yaml.
Reusing a map
- Relocalize with the mode family the map was built with. A stereo map also accepts
mono_wheel,mono_basaltandmono_imu, with the map's left camera. - Another robot, lens or stereo rig can relocalize in a map and extend it in the stereo and single-camera
VGGT-Omega modes (
stereo_imuandmono_imuonly with images of the map's size; the single-camera scale is untested there). PnP and Depth Anything 3 need the map's own camera. - A map built with
--fastneeds--fast.
Guide: map once, relocalize later Reference: scoring against ground truth
Pick a sensor mode
A mode names your camera and your source of odometry in one word:
stereo_wheel is a stereo camera with wheel odometry.
Find your modethree questions
Recommendedfor your answers
stereo_wheel
Install with bash install.sh --stereo
Stereo pair + external odometry. VGGT-Omega measures the pose against the retrieved keyframes and the stereo baseline gives the metric scale.
In the benchmark: 93% relocalization success, 0.35% mapping error (of the path). Benchmark and caveats
$ python run.py my_sequence --mode stereo_wheel
- Also possible:
stereo_dpvo(87% relocalization success, 0.57% mapping error), which needs no odometry.
stereo_wheel, the reference mode of the benchmark. An
RGB-D camera with wheel odometry: rgbd_wheel needs no large model and no
non-commercial weights; the VGGT-Omega modes cope better with large changes in lighting and viewpoint. On an RTX
4000 Ada, relocalization runs at about 39 frames per second in rgbd_wheel, 10.6 in
stereo_basalt (20.7 with --fast).All 14 modes
Scroll sideways for every column.
| Camera | Wheel or other odometry | Basalt VIO | IMU, fused by CROSS | Camera only |
|---|---|---|---|---|
| Stereo | stereo_wheel--stereo |
stereo_basalt--stereo |
stereo_imu--stereo |
stereo_dpvo--mono |
| RGB-D | rgbd_wheelbase |
rgbd_basaltbase |
no mode | rgbd_dpvo--mono |
| RGB-D, learned poses scaled by depth | rgbd_ff_wheel--stereo |
rgbd_ff_basalt--stereo |
no mode | no mode |
| One camera | mono_wheel--stereo |
mono_basalt--stereo |
mono_imu--monomono_vio--mono |
mono_dpvo--mono |
Under each mode: the install option it needs (base: bash install.sh
alone).
Stereo, rgbd_ff_ and single-camera modes measure poses with one forward pass of
VGGT-Omega, fine-tuned for CROSS
(mono_dpvo: Depth Anything 3); the other rgbd_ modes match features and solve PnP on the
depth. _wheel modes read any odometry in odom_left.txt (wheel, INS, a phone's ARKit);
_basalt modes read odom_vio.txt, written by Basalt's stereo-inertial VIO.
KITTI 07a car, 303 m, outdoors
One drive, eleven sensor setups
One mode per panel, named under it: the camera input, the path so far and the map error, final at the end of the drive.
Panels by cameramap error, % of the path
- Stereo0.08–0.24%car odometry, Basalt VIO, IMU, visual odometry
- RGB-D0.09–0.24%car odometry, Basalt VIO, visual odometry
- One camera0.08–0.67%car odometry, Basalt VIO, IMU, visual odometry
Use your own data
Your own recordings go in a plain folder: images, a calibration file, one pose per image and, if you have them, odometry and depth.
run.py reads posed RGB-D folders, stereo folders, KITTI raw, TartanAir V2,
Virtual KITTI 2, OpenLORIS and Record3D (.r3d) files, and detects the format from the folder.
my_sequence/
├── calib.json intrinsics K, image size, fps
├── rgb/ 000000.png, 000001.png, … (or .jpg; read in name order)
├── depth/ optional: one depth image per frame
├── poses_left.txt one camera pose per image
└── odom_left.txt optional: the robot's odometry, same format
my_sequence/
├── calib.json as above, plus T_right_in_left
├── left/ right/ one image pair per frame, same names
├── depth/ optional
├── poses_left.txt poses of the left camera
└── odom_left.txt optional
File formats
calib.json{"K": [[fx, 0, cx], [0, fy, cy], [0, 0, 1]], "width": 640, "height": 480, "fps": 10}; stereo adds"T_right_in_left", the 4×4 pose of the right camera in the left camera's frame.- images
- Undistorted (pinhole). Both stereo cameras share
K;stereo_dpvoandstereo_imuneed rectified pairs. depth/- 16-bit PNG in millimeters or
.npyin meters. poses_left.txt- Camera-to-world poses, one 4×4 matrix per line as 16 numbers (row by row), OpenCV axes: x right, y down, z forward. The ground truth, used only to evaluate; without it, write your odometry poses here too.
odom_left.txt- Same format; CROSS uses the steps between consecutive poses. Without it, the odometry is simulated from
poses_left.txt: noise-free, unless--snradds noise. imu.txtimu.json- For
*_imuandmono_vio:t wx wy wz ax ay az(rad/s, m/s²), and the IMU's pose in the camera frame with its noise. gnss.txt- Optional GNSS fixes, to place the map on the Earth (Large scale).
Converters for public datasets are in scripts/datasets/ (OpenLORIS, TUM RGB-D) and
benchmark/datasets/ (KITTI, ROVER, NCLT). On a live robot, feed frames through the
Python API.
Configuration
A run's settings come from four layers, each overriding the one before.
- Mode file
configs/modes/<mode>.yaml, with shared pieces fromconfigs/base/ - Fast overlaywith
--fast:configs/overlays/fast/<mode>.yaml - Your files
--config a.yaml b.yaml, left to right - Single values
--set section.key=value
$ python run.py my_sequence --mode stereo_wheel \
--config configs/overlays/outdoor.yaml my_robot.yaml \
--set mapping.loop_closure.confidence=0.9995
Your own file (my_robot.yaml here) holds only the keys it changes, nested as in the mode files and
cross/core/config.py. An unknown key is ignored with a warning; --set stops instead.
Shipped overlays
| Overlay in configs/overlays/ | Use it for |
|---|---|
outdoor.yaml | outdoor scenes (larger distances) |
odometry_vio.yaml | a *_wheel mode fed by a low-drift VIO (ARKit / ARCore on a phone, a Basalt-class VIO) |
strict_merge.yaml | large or repetitive buildings, where a wrong relocalization costs more than a late one (acts in stereo_wheel, stereo_basalt, mono_wheel, mono_imu, mono_vio) |
stereo_lowband.yaml | remote sessions over a thin link: the left image only, at half the rate |
geo_anchor.yaml | with GNSS: place the map on the Earth without moving it (under evaluation) |
Camera mounting. The default assumes a forward-looking camera on a ground robot. For a tilted
camera set mapping.projection.estimate_vertical=true; for a down-looking one
mapping.projection.vertical=z.
Python API
Run CROSS inside your own program: build a session for a mode, optionally load a map, feed frames and read the pose in the map.
from cross.core.config import load_config
from cross.dataloader.auto import load_dataset
from cross.pipeline import mode_config_files, build_session
from cross.core.types import Camera
from cross.utils.lie_tensor import normalize_SE3
dataset = load_dataset("data/posed/office1-2")
camera = Camera(K=dataset.rgb_K, frame_width=dataset.rgb_width, frame_height=dataset.rgb_height)
config = load_config(*mode_config_files("rgbd", "external")) # = configs/modes/rgbd_wheel.yaml
session = build_session("rgbd", "external", camera, config)
session.load_map("maps/office/map.pkl") # optional: relocalize in a saved map
to_matrix = lambda pose: normalize_SE3(pose).matrix().detach().cpu().numpy()
for i, frame in enumerate(dataset.replay_data()):
if i == 0:
frame["delta_pose"] = None # the first frame has no motion
session.process(frame)
if session.localized():
pose, _, _ = session.belief(to_matrix) # 4x4 camera pose in the map
session.save_map("maps/office/map.pkl")
session.release()
On a robot, loop over your own frames instead; each frame is a dictionary with:
rgb- the color image, H × W × 3, uint8
depthorrgb_right- as the mode needs: depth in meters, or the stereo pair's right image
delta_pose- the odometry step: the camera's 4×4 motion since the previous frame
timestamp- seconds
The stereo modes also need the rig: build_session(..., T_right_in_left=...). The modes with a
monocular front end take more settings: the reference shows how run.py builds them.
More capabilities
CROSS runs light on the robot, heavy in the cloud, keeps large maps accurate with or without GPS, and a map made with one camera serves another.
Light on the robot, heavy in the cloud
The robot needs only a CPU: it runs its odometry and cross-edge, a small client without PyTorch
that sends images only when the GPU server will use them. Network delays and outages hold back only the map
corrections, never the robot's pose.
The live VIO is tested on x86-64 only; the edge replays recorded stereo folders (no live camera driver is included); the link is not encrypted: use a network you trust.
Large scale, with or without GPS
Maps stay on disk and load keyframe images only when used; long pose-graph optimizations run in the background.
Without GPS, verified loop closures keep a large map accurate; GPS (GNSS) fixes
(--gnss in scripts/map_and_reloc*.py) anchor it to latitude and longitude, and a later
session's fix and compass narrow the search.
In development: several robots building one map together.
A 3D world and navigation
Two optional packages build on a saved map. cross_world trains 3D Gaussians on its keyframes for a
web viewer and moves them with the map after a loop closure, without retraining (not yet for monocular maps).
cross_nav predicts a bird's-eye-view occupancy grid from the same VGGT-Omega pass, with a trajectory
policy, planners and a robot agent on top (bash install.sh --nav).
Run the benchmark
A fixed protocol with three tasks, run on the same sessions for every system: CROSS and 6 open-source baselines.
The tasks: T1 mapping accuracy (ATE), T2 localization in a stored map (recall) and T3 relocalization success, on KITTI, OpenLORIS-Scene, ROVER and SimChange v3; the scores come from 6,368 runs over 86 sessions (12,491 runs are published).
stereo_wheel)- CROSS93%
- LingBot-Map upper bound48%
- RTAB-Map same inputs41%
T3: the share of 10 s trials that start with no pose in a map from another session and end within 2 m indoors or 5 m outdoors, averaged over OpenLORIS-Scene, ROVER and SimChange v3. LingBot-Map runs on one camera without odometry and has no relocalization step, so its number is an upper bound (hatched). ROVER and SimChange odometry is simulated from ground truth plus noise. Benchmark and caveats
To reproduce it, follow benchmark/README.md: prepare the datasets, run the job list, build the tables. Tuning per dataset is not allowed.
Protocol Development split Benchmark Full results Trajectory replays
Troubleshooting
The problems that come up most often, each with its fix.
| Problem | What to do |
|---|---|
CUDA out of memory | close other GPU programs; in stereo_basalt and stereo_dpvo, try --fast |
| the new session never joins the map | the session must revisit places of the map, and the robot must move: placements are tested at new keyframes. Check the calibration (K, T_right_in_left); a map built with --fast needs --fast |
No module named 'dpvo' | the mode needs bash install.sh --mono |
CROSS asks [y/N] before it starts | the configuration has a documented risk; read the warning, or pass --accept-config-risk |
Still stuck? Open an issue on GitHub.
Licenses
The code is MIT-licensed; some of the learned models it downloads are for non-commercial research only.
Third-party components keep their own licenses:
- VGGT-Omega (the vendored code and both checkpoints, ours included):
FAIR Noncommercial Research License.
The stereo,
rgbd_ff_*and single-camera modes use it, exceptmono_dpvo. - Depth Anything 3: code under Apache-2.0, the DA3-LARGE-1.1 weights that
--monoand--da3download under CC BY-NC 4.0. - DPVO (built by
--mono): MIT.
rgbd_wheel, rgbd_basalt) use none of the non-commercial models. Check the licenses above
before any commercial use of the other modes.Citation
If you use CROSS in your research, please cite the NeurIPS 2026 paper (arXiv 2605.02227).
@inproceedings{wang2026cross,
title = {Change-Robust Online Topological Memory for Long-Term
Relocalization and Semantic Navigation},
author = {Wang, Jiaming and Chen, Jizhuo and Liu, Diwen and
Ghotavadekar, Atharva and Da, Jiaxuan and K{\"a}stner, Linh
and Soh, Harold},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2026},
eprint = {2605.02227},
archivePrefix = {arXiv}
}