Large scale analysis of dataset and simulation biases in SLAM research.

Abstract

The performance of visual SLAM and localization methods is generally reported on famous datasets. These datasets are generally captured under human supervision and hence, are prone to human biases. In this work we expose two such biases (capture bias and negative world bias) in well-known SLAM datasets. Photo-realistic simulators provide a platform for gathering data without human supervision and hence human bias. However, not every simulator is suited for benchmarking visual localization methods. This is due to the difficulty of calibrating the first-view camera of these simulators. The calibration parameters (both intrinsic and extrinsic), while routinely provided with the datasets, are not generally available for simulators and virtual worlds. We propose a novel and user-friendly method to calibrate these simulators which is an essential requirement for using them for visual navigation. We demonstrate our method on a well-known simulator (MINOS), as well as a highly popular open world game (GTA-V). Finally, we also analyze the simulation-to-reality gap of these virtual platforms and propose a method to reduce this gap. We show that the performance of visual navigation algorithms (e.g., simultaneous localization and mapping: SLAM) significantly degrades when tested on novel situations available in virtual worlds.