Documentation: Install and run CROSS for multi-session mapping and relocalization

Open-source visual SLAM for multi-session mapping, tracking and relocalization on robots: map a place once and relocalize every later session in it, across changes in lighting, weather, season and furniture.

  • Runs onLinux with an NVIDIA GPU and CUDA
  • SensorsRGB-D, stereo or one camera; with wheel odometry, a VIO, an IMU or none
  • InterfaceCommand line and Python; no ROS node
  • LicenseMIT-licensed code. The learned models (VGGT-Omega, Depth Anything 3) are for non-commercial research; the RGB-D modes with PnP use neither. Licenses

Install: two commands

  1. # clone
  2. $ git clone https://github.com/jiaming-ai/CROSS.git && cd CROSS
  3. # install: .venv, PyTorch, CROSS, GTSAM and the models
  4. $ bash install.sh --stereo && source .venv/bin/activate

What CROSS does

A robot that comes back should know where it is: CROSS builds a map from a camera stream, saves it, and relocalizes later sessions in it.

  1. Session 1 Map The robot drives once. CROSS stores keyframes, their poses and how they connect.
  2. On disk Saved map map.pkl and map.pkl.store/: the graph and its keyframe images, which a VLM or an agent can read too.
  3. Every later session Relocalize and extend Starts with no pose, joins the map, reports poses in the map's coordinates, adds new places.

Between sessions the world changes: day turns to night, furniture moves, people walk by. CROSS keeps competing guesses of the pose until the evidence separates them and checks every loop closure, so one look-alike view cannot teleport the robot.

Guide: the ideas in five minutes

Install

One script installs everything into a local environment: PyTorch, CROSS, GTSAM and the model weights.

It needs Linux, an NVIDIA GPU with CUDA and about 10 GB of disk, and puts the environment in .venv (Python 3.11, via uv, which it fetches if missing).

Terminal
$ git clone https://github.com/jiaming-ai/CROSS.git
$ cd CROSS
$ bash install.sh --stereo
$ source .venv/bin/activate

Install options

OptionAdds
(none)the RGB-D modes with PnP (rgbd_wheel, rgbd_basalt)
--stereoVGGT-Omega and our fine-tuned checkpoint (Jamie-nus/VGGT-Omega-CROSS): stereo_wheel, stereo_basalt, stereo_imu, rgbd_ff_*, mono_wheel, mono_basalt
--mono--stereo + DPVO (built from source; needs nvcc matching PyTorch's CUDA), Depth Anything 3 and LightGlue: every mode

bash install.sh --help lists the rest. Check the installation with python -m pytest -q (a few minutes).

Tip. Run every command from the repository folder, with the environment active: model paths (models/...) are relative to it. A few models download on first use, so the first run needs internet; after that, export CROSS_HUB_OFFLINE=1 uses the cached copies.

Guide: install

Quick start

Map once, relocalize later: the whole workflow on two recordings of the same office.

The recordings come from OpenLORIS-Scene, a public dataset recorded by a wheeled robot with an RGB-D camera and wheel odometry. The quick start uses rgbd_wheel, which every install includes.

  1. Get two sessions

    Download the office package from Hugging Face (seven sessions, 10.6 GB; unpacking needs about 22 GB free) with the benchmark's script, then convert two sessions to CROSS's folder layout:

    Terminal
    # py7zr unpacks the archives; the PATH prefix finds uv if install.sh has just installed it
    $ PATH="$HOME/.local/bin:$PATH" uv pip install py7zr
    $ bash benchmark/datasets/download_openloris.sh data/openloris office1-1_7
    $ python scripts/datasets/convert_openloris.py data/openloris/office1-1 data/posed/office1-1
    $ python scripts/datasets/convert_openloris.py data/openloris/office1-2 data/posed/office1-2
  2. Session 1: build the map and save it

    Terminal
    $ python run.py data/posed/office1-1 --mode rgbd_wheel --save-map maps/office/map.pkl

    A Rerun window shows the map as it grows (on a server, add --no-viz). The map is map.pkl and the folder map.pkl.store/ beside it: copy or move them together.

  3. Session 2: relocalize in the saved map

    Terminal
    $ python run.py data/posed/office1-2 --mode rgbd_wheel --load-map maps/office/map.pkl \
        --save-poses office1-2_poses.txt --save-map maps/office/map.pkl

    Once CROSS has placed the session in the map, office1-2_poses.txt gets one line per frame in the map's coordinates (TUM format: time x y z qx qy qz qw). Saving again stores the map extended by this session.

Common options

OptionWhat it does
--frames N, --start Nprocess N frames, starting at frame N
--no-vizno viewer window
--config my.yaml, --set key=valuechange the configuration (Configuration)
--fastsmaller images on a weaker GPU (only stereo_basalt and stereo_dpvo have a fast setting)

Outdoors, add the outdoor settings: --config configs/overlays/outdoor.yaml.

Reusing a map

  • Relocalize with the mode family the map was built with. A stereo map also accepts mono_wheel, mono_basalt and mono_imu, with the map's left camera.
  • Another robot, lens or stereo rig can relocalize in a map and extend it in the stereo and single-camera VGGT-Omega modes (stereo_imu and mono_imu only with images of the map's size; the single-camera scale is untested there). PnP and Depth Anything 3 need the map's own camera.
  • A map built with --fast needs --fast.

Guide: map once, relocalize later Reference: scoring against ground truth

Pick a sensor mode

A mode names your camera and your source of odometry in one word: stereo_wheel is a stereo camera with wheel odometry.

Find your modethree questions

Camera
Motion input
Scene

Recommendedfor your answers

stereo_wheel

Install with bash install.sh --stereo

Stereo pair + external odometry. VGGT-Omega measures the pose against the retrieved keyframes and the stereo baseline gives the metric scale.

In the benchmark: 93% relocalization success, 0.35% mapping error (of the path). Benchmark and caveats

Run on your folder
$ python run.py my_sequence --mode stereo_wheel
  • Also possible: stereo_dpvo (87% relocalization success, 0.57% mapping error), which needs no odometry.
Which to try first. A stereo camera with any odometry: stereo_wheel, the reference mode of the benchmark. An RGB-D camera with wheel odometry: rgbd_wheel needs no large model and no non-commercial weights; the VGGT-Omega modes cope better with large changes in lighting and viewpoint. On an RTX 4000 Ada, relocalization runs at about 39 frames per second in rgbd_wheel, 10.6 in stereo_basalt (20.7 with --fast).

All 14 modes

Scroll sideways for every column.

Every mode, by camera and source of odometry, with the install option each needs
Camera Wheel or other odometry Basalt VIO IMU, fused by CROSS Camera only
Stereo stereo_wheel--stereo stereo_basalt--stereo stereo_imu--stereo stereo_dpvo--mono
RGB-D rgbd_wheelbase rgbd_basaltbase no mode rgbd_dpvo--mono
RGB-D, learned poses scaled by depth rgbd_ff_wheel--stereo rgbd_ff_basalt--stereo no mode no mode
One camera mono_wheel--stereo mono_basalt--stereo mono_imu--monomono_vio--mono mono_dpvo--mono

Under each mode: the install option it needs (base: bash install.sh alone).

Stereo, rgbd_ff_ and single-camera modes measure poses with one forward pass of VGGT-Omega, fine-tuned for CROSS (mono_dpvo: Depth Anything 3); the other rgbd_ modes match features and solve PnP on the depth. _wheel modes read any odometry in odom_left.txt (wheel, INS, a phone's ARKit); _basalt modes read odom_vio.txt, written by Basalt's stereo-inertial VIO.

KITTI 07a car, 303 m, outdoors

One drive, eleven sensor setups

One mode per panel, named under it: the camera input, the path so far and the map error, final at the end of the drive.

Panels by cameramap error, % of the path

  • Stereo0.08–0.24%car odometry, Basalt VIO, IMU, visual odometry
  • RGB-D0.09–0.24%car odometry, Basalt VIO, visual odometry
  • One camera0.08–0.67%car odometry, Basalt VIO, IMU, visual odometry

Guide: choose a mode Reference: modes

Use your own data

Your own recordings go in a plain folder: images, a calibration file, one pose per image and, if you have them, odometry and depth.

run.py reads posed RGB-D folders, stereo folders, KITTI raw, TartanAir V2, Virtual KITTI 2, OpenLORIS and Record3D (.r3d) files, and detects the format from the folder.

RGB-D or one camera
my_sequence/
├── calib.json       intrinsics K, image size, fps
├── rgb/             000000.png, 000001.png, … (or .jpg; read in name order)
├── depth/           optional: one depth image per frame
├── poses_left.txt   one camera pose per image
└── odom_left.txt    optional: the robot's odometry, same format
Stereo
my_sequence/
├── calib.json       as above, plus T_right_in_left
├── left/  right/    one image pair per frame, same names
├── depth/           optional
├── poses_left.txt   poses of the left camera
└── odom_left.txt    optional

File formats

calib.json
{"K": [[fx, 0, cx], [0, fy, cy], [0, 0, 1]], "width": 640, "height": 480, "fps": 10}; stereo adds "T_right_in_left", the 4×4 pose of the right camera in the left camera's frame.
images
Undistorted (pinhole). Both stereo cameras share K; stereo_dpvo and stereo_imu need rectified pairs.
depth/
16-bit PNG in millimeters or .npy in meters.
poses_left.txt
Camera-to-world poses, one 4×4 matrix per line as 16 numbers (row by row), OpenCV axes: x right, y down, z forward. The ground truth, used only to evaluate; without it, write your odometry poses here too.
odom_left.txt
Same format; CROSS uses the steps between consecutive poses. Without it, the odometry is simulated from poses_left.txt: noise-free, unless --snr adds noise.
imu.txt imu.json
For *_imu and mono_vio: t wx wy wz ax ay az (rad/s, m/s²), and the IMU's pose in the camera frame with its noise.
gnss.txt
Optional GNSS fixes, to place the map on the Earth (Large scale).

Converters for public datasets are in scripts/datasets/ (OpenLORIS, TUM RGB-D) and benchmark/datasets/ (KITTI, ROVER, NCLT). On a live robot, feed frames through the Python API.

Guide: using your own data

Configuration

A run's settings come from four layers, each overriding the one before.

  1. Mode fileconfigs/modes/<mode>.yaml, with shared pieces from configs/base/
  2. Fast overlaywith --fast: configs/overlays/fast/<mode>.yaml
  3. Your files--config a.yaml b.yaml, left to right
  4. Single values--set section.key=value
Terminal
$ python run.py my_sequence --mode stereo_wheel \
    --config configs/overlays/outdoor.yaml my_robot.yaml \
    --set mapping.loop_closure.confidence=0.9995

Your own file (my_robot.yaml here) holds only the keys it changes, nested as in the mode files and cross/core/config.py. An unknown key is ignored with a warning; --set stops instead.

Shipped overlays

Overlay in configs/overlays/Use it for
outdoor.yamloutdoor scenes (larger distances)
odometry_vio.yamla *_wheel mode fed by a low-drift VIO (ARKit / ARCore on a phone, a Basalt-class VIO)
strict_merge.yamllarge or repetitive buildings, where a wrong relocalization costs more than a late one (acts in stereo_wheel, stereo_basalt, mono_wheel, mono_imu, mono_vio)
stereo_lowband.yamlremote sessions over a thin link: the left image only, at half the rate
geo_anchor.yamlwith GNSS: place the map on the Earth without moving it (under evaluation)

Camera mounting. The default assumes a forward-looking camera on a ground robot. For a tilted camera set mapping.projection.estimate_vertical=true; for a down-looking one mapping.projection.vertical=z.

Guide: changing the configuration Reference: configuration

Python API

Run CROSS inside your own program: build a session for a mode, optionally load a map, feed frames and read the pose in the map.

Python
from cross.core.config import load_config
from cross.dataloader.auto import load_dataset
from cross.pipeline import mode_config_files, build_session
from cross.core.types import Camera
from cross.utils.lie_tensor import normalize_SE3

dataset = load_dataset("data/posed/office1-2")
camera = Camera(K=dataset.rgb_K, frame_width=dataset.rgb_width, frame_height=dataset.rgb_height)
config = load_config(*mode_config_files("rgbd", "external"))  # = configs/modes/rgbd_wheel.yaml
session = build_session("rgbd", "external", camera, config)
session.load_map("maps/office/map.pkl")  # optional: relocalize in a saved map

to_matrix = lambda pose: normalize_SE3(pose).matrix().detach().cpu().numpy()
for i, frame in enumerate(dataset.replay_data()):
    if i == 0:
        frame["delta_pose"] = None  # the first frame has no motion
    session.process(frame)
    if session.localized():
        pose, _, _ = session.belief(to_matrix)  # 4x4 camera pose in the map
session.save_map("maps/office/map.pkl")
session.release()

On a robot, loop over your own frames instead; each frame is a dictionary with:

rgb
the color image, H × W × 3, uint8
depth or rgb_right
as the mode needs: depth in meters, or the stereo pair's right image
delta_pose
the odometry step: the camera's 4×4 motion since the previous frame
timestamp
seconds

The stereo modes also need the rig: build_session(..., T_right_in_left=...). The modes with a monocular front end take more settings: the reference shows how run.py builds them.

Guide: using CROSS from Python Reference: programmatic use

More capabilities

CROSS runs light on the robot, heavy in the cloud, keeps large maps accurate with or without GPS, and a map made with one camera serves another.

Light on the robot, heavy in the cloud

The robot needs only a CPU: it runs its odometry and cross-edge, a small client without PyTorch that sends images only when the GPU server will use them. Network delays and outages hold back only the map corrections, never the robot's pose.

The live VIO is tested on x86-64 only; the edge replays recorded stereo folders (no live camera driver is included); the link is not encrypted: use a network you trust.

cross-edge README

Large scale, with or without GPS

Maps stay on disk and load keyframe images only when used; long pose-graph optimizations run in the background. Without GPS, verified loop closures keep a large map accurate; GPS (GNSS) fixes (--gnss in scripts/map_and_reloc*.py) anchor it to latitude and longitude, and a later session's fix and compass narrow the search.

In development: several robots building one map together.

Reference: map files, GNSS and compass

A 3D world and navigation

Two optional packages build on a saved map. cross_world trains 3D Gaussians on its keyframes for a web viewer and moves them with the map after a loop closure, without retraining (not yet for monocular maps). cross_nav predicts a bird's-eye-view occupancy grid from the same VGGT-Omega pass, with a trajectory policy, planners and a robot agent on top (bash install.sh --nav).

CROSS World guide CROSS Nav README

Run the benchmark

A fixed protocol with three tasks, run on the same sessions for every system: CROSS and 6 open-source baselines.

The tasks: T1 mapping accuracy (ATE), T2 localization in a stored map (recall) and T3 relocalization success, on KITTI, OpenLORIS-Scene, ROVER and SimChange v3; the scores come from 6,368 runs over 86 sessions (12,491 runs are published).

Relocalization success (T3), overallCROSS with stereo + odometry (stereo_wheel)
  • CROSS93%
  • LingBot-Map upper bound48%
  • RTAB-Map same inputs41%

T3: the share of 10 s trials that start with no pose in a map from another session and end within 2 m indoors or 5 m outdoors, averaged over OpenLORIS-Scene, ROVER and SimChange v3. LingBot-Map runs on one camera without odometry and has no relocalization step, so its number is an upper bound (hatched). ROVER and SimChange odometry is simulated from ground truth plus noise. Benchmark and caveats

To reproduce it, follow benchmark/README.md: prepare the datasets, run the job list, build the tables. Tuning per dataset is not allowed.

Protocol Development split Benchmark Full results Trajectory replays

Troubleshooting

The problems that come up most often, each with its fix.

ProblemWhat to do
CUDA out of memoryclose other GPU programs; in stereo_basalt and stereo_dpvo, try --fast
the new session never joins the mapthe session must revisit places of the map, and the robot must move: placements are tested at new keyframes. Check the calibration (K, T_right_in_left); a map built with --fast needs --fast
No module named 'dpvo'the mode needs bash install.sh --mono
CROSS asks [y/N] before it startsthe configuration has a documented risk; read the warning, or pass --accept-config-risk

Still stuck? Open an issue on GitHub.

Licenses

The code is MIT-licensed; some of the learned models it downloads are for non-commercial research only.

Third-party components keep their own licenses:

  • VGGT-Omega (the vendored code and both checkpoints, ours included): FAIR Noncommercial Research License. The stereo, rgbd_ff_* and single-camera modes use it, except mono_dpvo.
  • Depth Anything 3: code under Apache-2.0, the DA3-LARGE-1.1 weights that --mono and --da3 download under CC BY-NC 4.0.
  • DPVO (built by --mono): MIT.
Commercial use. The RGB-D modes with PnP (rgbd_wheel, rgbd_basalt) use none of the non-commercial models. Check the licenses above before any commercial use of the other modes.

Citation

If you use CROSS in your research, please cite the NeurIPS 2026 paper (arXiv 2605.02227).

@inproceedings{wang2026cross,
  title     = {Change-Robust Online Topological Memory for Long-Term
               Relocalization and Semantic Navigation},
  author    = {Wang, Jiaming and Chen, Jizhuo and Liu, Diwen and
               Ghotavadekar, Atharva and Da, Jiaxuan and K{\"a}stner, Linh
               and Soh, Harold},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026},
  eprint    = {2605.02227},
  archivePrefix = {arXiv}
}