Benchmark: Visual multi-session SLAM under lighting, season and scene change

CROSS gives the best visual multi-session relocalization in this benchmark: from a cold start in a map from another session, it finds its place far more often than open-source SLAM systems run on the same sessions. It also ranks first in localization and mapping.

The figures show CROSS with a stereo camera and odometry (stereo_wheel), the recommended mode for these sensors. RTAB-Map is the other system that takes the same odometry: wheel odometry on OpenLORIS, close to the ground truth elsewhere (INS velocities on KITTI, ground truth plus noise on ROVER and SimChange). The leaderboard below ranks each system's best configuration instead.

Leaderboard

Each system's best configuration, ranked; filter by your robot's sensors

Sensor

Inputs

View

Relocalization success T3

Relocalization (T3): relocalization success, higher is better
RankSystemOverall successwithin 2 m indoors, 5 m outdoorsOpenLORISwithin 2 mROVERwithin 5 mSimChange v3within 2 m
1CROSSVGGT-Omega · dataset odometryStereo+ odometryOpenLORIS 84%ROVER 97%SimChange v3 97%
93%strict 89%
84%1 m: 76%97%3 m: 96%97%1 m: 96%
2LingBot-MapMonoCamera onlyupper boundOpenLORIS 53%ROVER 39%SimChange v3 51%
48%strict 40%
53%1 m: 46%39%3 m: 29%51%1 m: 47%
3RTAB-Mapdataset odometryRGB-D+ odometryOpenLORIS 36%ROVER 14%SimChange v3 79%
43%strict 41%
36%1 m: 29%14%3 m: 14%79%1 m: 79%
4ORB-SLAM3visual-inertialStereo+ IMUOpenLORIS 31%ROVER 23%SimChange v3 70%
41%strict 38%
31%1 m: 23%23%3 m: 22%70%1 m: 70%
5AMB3R-SLAMRGB-DCamera onlyOpenLORIS 8%ROVER 18%SimChange v3 28%
18%strict 17%
8%1 m: 7%18%3 m: 18%28%1 m: 25%
6VGGT-SLAM 2.0MonoCamera onlyOpenLORIS 11%ROVER 18%SimChange v3 13%
14%strict 11%
11%1 m: 8%18%3 m: 14%13%1 m: 10%
7MASt3R-SLAMMonoCamera onlyOpenLORIS 12%ROVER 0% (7/7 failed)SimChange v3 19% (2/43 failed)
10%strict 9%9/67 failed
12%1 m: 8%0%3 m: 0%7/7 failed19%1 m: 18%2/43 failed
  • Small type: the strict radius, 1 m indoors and 3 m outdoors.
  • Upper bound (hatched bar): LingBot-Map has no relocalization step, so all its query poses count.
  • k/N failed: k of N query sessions failed (a crash, a timeout or a failed map); their trials count as failed.
  • Trials scored: 1,998–2,009 for CROSS, ORB-SLAM3 and RTAB-Map; 409 for VGGT-SLAM 2.0; 315 for MASt3R-SLAM, LingBot-Map and AMB3R-SLAM. Systems that cannot store a map re-run the whole map session for every trial, so they are scored on a subset.

Localization in a stored map T2

Localization in a stored map (T2): localization recall, higher is better
RankSystemOverall recallwithin 2 m indoors, 5 m outdoorsOpenLORISwithin 2 mROVERwithin 5 mSimChange v3within 2 m
1CROSSVGGT-Omega · IMU fused by CROSSStereo+ IMUOpenLORIS 90%ROVER 97%SimChange v3 98%
95%strict 91%
90%1 m: 81%97%3 m: 95%98%1 m: 98%
2ORB-SLAM3visual-inertialStereo+ IMUOpenLORIS 26%ROVER 52%SimChange v3 82%
53%strict 46%
26%1 m: 23%52%3 m: 33%82%1 m: 82%
3RTAB-Mapvisual-inertial odometryRGB-D+ IMUOpenLORIS 16%ROVER 64%SimChange v3 72%
51%strict 39%
16%1 m: 9%64%3 m: 56%72%1 m: 54%
4LingBot-MapMonoCamera onlyupper boundOpenLORIS 35%ROVER 49%SimChange v3 60%
48%strict 28%
35%1 m: 22%49%3 m: 21%60%1 m: 40%
5AMB3R-SLAMMonoCamera onlyOpenLORIS 15%ROVER 41%SimChange v3 65%
40%strict 31%
15%1 m: 8%41%3 m: 33%65%1 m: 51%
6VGGT-SLAM 2.0MonoCamera onlyOpenLORIS 13%ROVER 15%SimChange v3 18%
15%strict 8%
13%1 m: 7%15%3 m: 9%18%1 m: 6%
7MASt3R-SLAMMonoCamera onlyOpenLORIS 5%ROVER 0% (7/7 failed)SimChange v3 13% (2/43 failed)
6%strict 5%9/67 failed
5%1 m: 3%0%3 m: 0%7/7 failed13%1 m: 13%2/43 failed
  • Small type: the strict radius, 1 m indoors and 3 m outdoors.
  • Upper bound (hatched bar): LingBot-Map has no relocalization step, so all its query poses count.
  • k/N failed: k of N query sessions failed (a crash, a timeout or a failed map); their frames count as not localized.

Mapping accuracy T1

Mapping accuracy (T1): ATE, overall as a share of the path length and per dataset in meters, lower is better
RankSystemOverall ATE% of path length, lower is betterKITTIATE, mOpenLORISATE, mROVERATE, mScale errormono
1CROSSVGGT-Omega scaled by depth · dataset odometryRGB-D+ odometryKITTI 1.98 mOpenLORIS 0.67 mROVER 0.10 m
0.33%
1.98 m0.67 m0.10 m
2RTAB-Mapdataset odometryStereo+ odometryKITTI 2.99 mOpenLORIS 1.61 mROVER 2.11 m
0.76%
2.99 m1.61 m2.11 m
3AMB3R-SLAMMonoCamera onlyKITTI 24.3 mOpenLORIS 1.94 mROVER 2.26 mscale error 36%
1.32%
24.3 m1.94 m2.26 m36%
4LingBot-MapMonoCamera onlyKITTI 28.5 mOpenLORIS 3.36 mROVER 4.49 mscale error 1,015%
1.89%
28.5 m3.36 m4.49 m1,015%
Not ranked: the mean leaves failed sequences out, or a dataset was not run
not rankedORB-SLAM3StereoCamera onlyKITTI 2.93 mOpenLORIS 0.61 m (5/22 failed)ROVER 0.88 m (3/8 failed)
0.49%8/40 failed
2.93 m0.61 m5/22 failed0.88 m3/8 failed
not rankedVGGT-SLAM 2.0MonoCamera onlyKITTI 60.3 m (1/10 failed)OpenLORIS 4.35 m (4/22 failed)ROVER 5.07 m (1/8 failed)scale error 4,182%
3.66%6/40 failed
60.3 m1/10 failed4.35 m4/22 failed5.07 m1/8 failed4,182%
not rankedMASt3R-SLAMMonoCamera onlyKITTI – (10/10 failed)OpenLORIS 1.12 m (7/22 failed)ROVER – (8/8 failed)
–25/40 failed
–10/10 failed1.12 m7/22 failed–8/8 failed–
  • k/N failed: k of N sequences failed: tracking lost (under 80% of the frames covered), a crash or a timeout. The mean leaves them out, so these rows are not ranked.
  • Scale error: how far a monocular map's scale is from metric. Monocular systems without metric input are aligned with Sim(3), so their ATE leaves this error out.
  • RGB-D on KITTI: KITTI has no depth camera. CROSS uses depth from stereo matching there; the baselines do not run KITTI's RGB-D setup, so their RGB-D rows have no overall value.

Numbers from the published benchmark of 2026-10-25 (commit 2b560ce): 6,368 of its 12,491 runs score these tables. Per-scene tables, the other runs, trajectories and failure cases: full results.

What we measure

Three tasks, and how every system is run

Three tasks under one protocol: the same recordings and radii for every system, and the same odometry for every system that takes it.

  1. T3 Relocalization

    Can it find its place from a cold start?

    Each query session is cut into 10 s trials: 100 frames at 10 Hz, a new trial every 50 frames. A trial loads the stored map and starts without knowing its pose. It succeeds when its final pose lies within the radius of the ground truth.

    • Indoors1 m · 2 m
    • Outdoors3 m · 5 m
  2. T2 Localization in a stored map

    How much of a query session is localized?

    A query session (another recording of the same place) runs once against the stored map, from its first frame and with no initial pose. Recall is the share of its frames that have a pose in the map within the radius; a frame without one counts as a miss.

    • Indoors1 m · 2 m
    • Outdoors3 m · 5 m
  3. T1 Mapping accuracy

    How accurate is the map of one session?

    One session is mapped from scratch. The error is the RMSE of the final trajectory after loop closure (ATE), aligned to the ground truth and divided by the length of the path. For CROSS the trajectory is its map keyframes, which are sparser than the other systems' keyframes.

    • DatasetsKITTI, OpenLORIS, ROVER
    • Failure< 80% of frames covered

Each dataset weighs the same in an overall score: the mean of OpenLORIS, ROVER and SimChange v3 for localization and relocalization (v3 replaced v2 on 25 October 2026; v2's tables are in the full results), and of KITTI, OpenLORIS and ROVER for mapping, where the error per path length lets kilometer drives and room-scale sessions compare. Localization and relocalization score only the frames that the map covers and in which the robot moves. Bold: the radius of the NeurIPS 2026 paper, in large type in the tables.

How every system is run

  • Same inputs

    Every system gets the same images at 10 Hz in real-time order. Every system that takes odometry gets the same stream: the dataset's odometry or a Basalt stereo VIO (OpenLORIS keeps its wheel odometry in the VIO rows).

  • Shipped defaults

    Each system runs its published defaults for the sensor, with no per-dataset tuning; the protocol lists the declared deviations. One is not listed there: outdoors (KITTI, ROVER), the CROSS row with Depth Anything 3 and visual odometry uses a different odometry-noise setting.

  • Map reuse

    CROSS saves and loads its map, ORB-SLAM3 loads and merges its atlas, and RTAB-Map loads its database in localization mode. MASt3R-SLAM, VGGT-SLAM 2.0, LingBot-Map and AMB3R-SLAM cannot store a map, so the map session and the query run as one stream, which keeps the map session's live state.

Datasets

4 datasets, 86 sessions, 2,205 relocalization trials

Real robots and rendered scenes, revisited under change: the same places at night, in another season, with people about and with the furniture moved.

ROVER

An outdoor robot on one campus route, over fourteen months

ROVER campus, map session on a September day: a building with outdoor stairs above a strip of lawn
Map, day
The same place in January, with a low winter sun shining into the camera
Winter
The same place at dusk, with the stair lights on
Dusk
The same place at night, almost black except for a few lamps
Night
  • Size8 sessions · 1,137 relocalization trials
  • Changesseason, weather, dusk and night
  • Odometrysimulated from the ground truth plus noise (the robot has no wheel odometry)
  • Tasksmapping, localization, relocalization

OpenLORIS-Scene

An indoor robot in an office, a corridor, a home, a café and a supermarket

OpenLORIS corridor, map session: a bright hall with tall windows and red armchairs
Map, corridor 1-1
The same place in session 1-2
1-2
The same place in session 1-4, darker and lit by lamps
1-4, lamps
The same place in session 1-5, with a person walking in front of the robot
1-5, a person
  • Size22 sessions in 5 scenes · 294 relocalization trials
  • Changesdaylight and lamps, people, moved furniture and goods
  • Odometrythe robot's wheel odometry
  • Tasksmapping, localization, relocalization

SimChange v3

Rendered scenes with camera noise and blur: a house, a restaurant and a lab corridor (photos: the v2 renders)

SimChange restaurant, map traversal: a sunlit terrace with tables, umbrellas and the sea
Map, day
The same view in evening light, with long shadows
Evening
The same view at night, lit only by lamps
Night
The same view with every movable object rearranged
Rearranged
  • Size46 traversals in 3 scenes · 774 relocalization trials
  • Changessun to night, moved objects, paths up to 0.3 m off the route, the route in reverse; alone or combined
  • Odometrysimulated from the ground truth plus noise
  • Taskslocalization, relocalization

KITTI

A car on town streets, country roads and a highway around Karlsruhe

KITTI sequence 00: a residential street with parked cars
Sequence 00
KITTI sequence 01: a highway
01, highway
KITTI sequence 02: a road between houses and trees
02
KITTI sequence 07: a town street lined with houses
07
  • Size10 drives of 0.4 to 5 km
  • Changesnone: one session per route
  • Odometrydead reckoning of the INS velocities (GPS positions not used; close to the ground truth)
  • Tasksmapping

Systems

How each one works, and how it reuses a map

CROSS against two classical feature-based systems and four learned feed-forward ones, all open source.

This project

  • Long-term relocalization with a probabilistic topological map: a keyframe graph with verified loop closures, and the camera pose tracked as a mixture of hypotheses until the evidence separates them. Relative poses come from VGGT-Omega (with any camera), PnP on depth (RGB-D) or the Depth Anything 3 back end (mono). MIT-licensed code; the VGGT-Omega and Depth Anything 3 weights are for non-commercial use only, and the RGB-D modes with PnP use neither.

    Sensors
    RGB-D, stereo or mono · camera only, + IMU or + odometry
    Map reuse
    saves and loads its map

Classical, feature-based

  • Visual and visual-inertial SLAM on ORB features, with a multi-map atlas.

    Sensors
    RGB-D, stereo or mono · camera only or + IMU
    Map reuse
    loads and merges its atlas
  • Graph SLAM with appearance-based loop-closure detection.

    Sensors
    RGB-D or stereo · + odometry, or its own visual or visual-inertial odometry
    Map reuse
    loads its database in localization mode

Learned, feed-forward

  • Dense SLAM on the MASt3R two-view reconstruction model.

    Sensors
    mono · camera only
    Map reuse
    none: map and query run as one stream
  • SLAM built from submaps of the VGGT feed-forward model.

    Sensors
    mono · camera only
    Map reuse
    none: map and query run as one stream
  • Streaming feed-forward reconstruction over a cache of the map session's keyframes, with no relocalization step.

    Sensors
    mono · camera only
    Map reuse
    none: map and query run as one stream
  • Depth Anything 3 submaps joined in a hierarchical Sim(3) pose graph; metric with stereo or depth input.

    Sensors
    RGB-D, stereo or mono · camera only
    Map reuse
    none: map and query run as one stream

Run CROSS on your robot

Install it with one script, map once, and relocalize in every later session. Or reproduce these numbers and add your own system.