System Evaluation

---

We open sourced our evaluation toolkit at eval-vislam.

Acciracy(APE, RPE, ARE, RRE, Completeness)

Usage:
./accuracy <groundtruth> <input> <fix scale>

Arguments:
<groundtruth>  Path to sequence folder, e.g. ~/VISLAM-Dataset/A0.
<input>        SLAM camera trajectory file in TUM format.
<fix scale>    Set to 1 for VISLAM, set to 0 for VSLAM.

Initialization Scale Error and Time

Usage:
./initialization <groundtruth> <input> <has inertial>

Arguments:
<groundtruth>  Path to sequence folder, e.g. ~/VISLAM-Dataset/A0.
<input>        SLAM camera trajectory file in TUM format.
<has inertial> Set to 1 for VISLAM, set to 0 for VSLAM.

Robustness

Usage:
./robustness <groundtruth> <input> <fix scale>

Arguments:
<groundtruth>  Path to sequence folder, e.g. ~/VISLAM-Dataset/A0.
<input>        SLAM camera trajectory file in TUM format.
<fix scale>    Set to 1 for VISLAM, set to 0 for VSLAM.

Relocalization Time

Usage:
./relocalization <groundtruth> <input> <has inertial>

Arguments:
<groundtruth>  Path to sequence folder, e.g. ~/VISLAM-Dataset/A0.
<input>        SLAM camera trajectory file in TUM format.
<has inertial> Set to 1 for VISLAM, set to 0 for VSLAM.

Representative Monocular SLAM Results

---

Due to the randomness in these SLAM systems, they may not always produce the same results. So, we run each benchmark 10 times to take the average of the results. Some algorithms, when running against some sequences, may produce inconsistent results. Therefore, we remove these failure cases (i.e., APEs were unusually large) by inspection, and then compute the average of the remaining ones.

The number of failures for each sequence

VSLAM Tracking accuracy

VISLAM Tracking accuracy

Initialization quality

Tracking robustness

Relocalization time

System Reference

---

1. [1] Klein G, Murray D. Parallel tracking and mapping for small AR workspaces. In: 6th IEEE and ACM International Symposium on Mixed and Augmented Reality. Nara, Japan, 2007: 225–234 DOI:10.1109 / ISMAR.2007.4538852

2. [2] Mur-Artal R, Tardos J D. ORB-SLAM2: an open-source SLAM system for monocular, stereo, and RGB-D cameras. IEEE Transactions on Robotics, 2017, 33(5): 1255–1262 DOI:10.1109 / tro.2017.2705103.

3. [3] Engel J, Schöps T, Cremers D. LSD-SLAM: Large-Scale Direct Monocular SLAM. Computer Vision–ECCV 2014. Cham: Springer International Publishing, 2014: 834−849 DOI:10.1007 / 978-3-319-10605-2_54.

4. [4] Engel J, Koltun V, Cremers D. Direct sparse odometry. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(3): 611–625 DOI:10.1109 / tpami.2017.2658577.

5. [5] Mourikis A I, Roumeliotis S I. A multi-state constraint kalman filter for vision-aided inertial navigation. In: IEEE International Conference on Robotics and Automation. Roma, Italy, 2007: 3565–3572 DOI:10.1109 / ROBOT.2007.364024.

6. [6] Leutenegger S, Lynen S, Bosse M, Siegwart R, Furgale P. Keyframe-based visual–inertial odometry using nonlinear optimization. The International Journal of Robotics Research, 2015, 34(3): 314–334 DOI:10.1177 / 0278364914554813.

7. [7] Qin T, Li P L, Shen S J. VINS-mono: A robust and versatile monocular visual-inertial state estimator. IEEE Transactions on Robotics, 2018, 34(4): 1004–1020 DOI:10.1109 / tro.2018.2853729.