This research is based on a new and accurate technique for Visual 3D Odometry using a stereo-pair of cameras. It investigates a new image-based approach to tracking the 6dof trajectory of a stereo camera pair using a corresponding reference image pairs instead of explicit 3D feature reconstruction of the scene. A dense minimisation approach is employed which directly uses all grey-scale information available within the stereo pair (or stereo region) leading to very robust and precise results. Metric 3D structure constraints are imposed by consistently warping corresponding stereo images to generate novel viewpoints at each stereo acquisition. An iterative non-linear trajectory estimation approach is formulated based on a quadrifocal relationship between the image intensities within adjacent views of the stereo pair. A robust M-estimation technique is used to reject outliers corresponding to moving objects within the scene or other outliers such as occlusions and illumination changes. The technique is applied to recovering the trajectory of a moving vehicle in long and difficult sequences of images.
The 3D visual odometry algorithm was tested on real full-scale sequences as can be seen in the videos below. Several test sequences from different streets in Versailles, France, were used to validate the results. The estimated 3D trajectory from a stereo-pair of cameras was superimposed on Google Earth satelite images of the area.
This first sequence is that of a relatively straight road. The distance travelled by the car has been measured using road markings in the images and satellite views with a precision of 2.9 cm/pixel for the Versailles region. The path length measured by both Google earth and the tracker was about 440m. It is difficult to register the satellite image with the projection of the trajectory since no three non-collinear points were available and the best that can be said is that they have approximately the same absolute length (ignoring tilt of the cameras and the incline of the road). Throughout the sequence several moving vehicles pass in front of the cameras and at one stage a car is overtaken.
This second sequence is particularly illustrative since a full loop of the round-about was performed. In particular this enables the drift to be measured at the crossing point in the trajectory. In the case of this round-about the drift at the crossing point was approximately 20cm in the vertical direction to the road-plane. Considering that the trajectory around the round-about is approximately 200m long (measured using Google earth), this makes a drift of 0.01% on the measurable axis. In the case of large scale scenes such as this one it was necessary to detect and update the reference image periodically when it was no longer visible or too approximate. Due to the highly redundant amount of data, the robust estimator was able to successfully reject pedestrians and moving cars from the estimation process. It can be noted, however, that all static information available was used to estimate the pose (including the parked cars) therefore leading to a very precise result with minimal drift over large displacements.
(Click on image or here to play)
Recently a publication was presented at the International Conference on Robotics and automation (ICRA'07 ) in Rome where we were selected as finalist for the best vision paper prize. The publication is now available here:
- A.I. Comport, E. Malis, P. Rives. Accurate Quadri-focal Tracking for Robust 3D Visual Odometry. In IEEE Int. Conf. on Robotics and Automation, ICRA'07, Rome, Italy, April 2007.
Some of the new results for the LAAS blimp sequence were presented at the conference. To see the sequence and the estimated trajectory click here (approximately 12M).
The full round about sequence showing the reference images as well as robust outlier rejection can be downloaded here (approximately 6.8M).
This research was carried out during the postdoctorat of Andrew Comport at INRIA Sophia-Antipolis, under the surpervision of Ezio Malis and Patrick Rives.