ToF and stereo data fusion

--
 --

Current 3D video applications require the availability of depth information, that can be acquired real-time by stereo vision systems and Time-of-Flight (ToF) cameras. We consider an heterogeneous acquisition system, made of two high resolution standard cameras L, R (stereo pair) and one ToF camera T. The stereo system and the ToF camera must be properly calibrated together in order to jointly operate. We introduced a generalized multi-camera calibration technique. We derived a probabilistic fusion algorithm that allow us to obtain high quality depth information from the information of both the ToF camera and the stereo-pair.

- Paper [1] and [2] provide a general formalization of the data fusion problem. The fusion algorithm is derived in a probabilistic setup, that allows the decoupling of the information from the stereo pair and the information from the ToF camera. Accurate models for the measurement errors of the stereo and ToF systems are derived and then used into the probabilistic fusion framework. Paper [2] presents a simpler version of the approach based on a local ML optimization. In paper [1] we introduced a more advanced measurement error model accounting for the mixed pixels effect together with a global MAP-MRF optimization scheme using an extended version of Loopy Belief Propagation with site-dependent labels.

- Paper [3] provides an Amplitude Modulation transmission model for the ToF camera T.

- Paper [4] provides a different approach for the fusion of the two data sources that extends the locally consistent framework used in stereo vision to the case where two different depth data sources are available.

- Paper [5] extends the framework of paper [4] by introducing novel confidence measures for the stereo and ToF data and using them to drive the locally consistent fusion process

- Paper [6] further extends the approach by exploiting a Convolutional Neural Network for the estimation of the confidence information. A novel synthetic dataset has also been constructed and used for the training of the deep network.

In order to test the effectiveness of our algorithm, we created datasets with ground-truth, available here (dataset of papers 2-3), here (dataset of paper 4) and here (dataset of paper [6]).