Dataset

 Here you can find the synthetic and real datasets used for the experimental evaluation of the approach described in the paper "Deep Learning for Multi-Path Error Removal in ToF Sensors" accepted for publication at the ECCV 2018 Workshop "Geometry Meets Deep Learning".

Abstract

The removal of Multi-Path Interference (MPI) is one of the major open challenges in depth estimation with Time-of-Flight (ToF) cameras. In this paper we propose a novel method for MPI removal and depth refinement exploiting an ad-hoc deep learning architecture working on data from a multi-frequency ToF camera. In order to estimate the MPI we use a Convolutional Neural Network (CNN) made of two subnetworks: a coarse network analyzing the global structure of the data at a lower resolution and a fine one exploiting the output of the coarse network in order to remove the MPI while preserving the small details. The critical issue of the lack of ToF data with ground truth is solved by training the CNN with synthetic information. Finally, the residual zeromean error is removed with an adaptive bilateral filter guided from a noise model for the camera. Experimental results prove the effectiveness of the proposed approach on both synthetic and real data.

 

The full paper can be downloaded from here

The additional material can be downloaded from here

flowchart

Flowchart of the proposed method

 

Dataset

We provide the synthetic and the real datasets which have been used for the experimental results of the paper. These contain data acquired from a multi-frequency ToF camera. The synthetic dataset contains the ToF acquisitions simulated with the Sony ToF Explorer simulator and it is divided in a training set and a test set. The training set contains the simulated ToF acquisitions on 40 scenes obtained by rendering 20 unique scenes from different viewpoints. The test set contains the simulated ToF acquisition on 14 scenes. The various scenes contain furnitures and objects of various shapes in different environments e.g., living rooms, kitchen rooms or offices. Furthermore, some outdoor locations with non-regular structure are also included in the dataset.They appear realistic and suitable for the simulation of ToF acquisition systems. The Sony ToF Explorer simulator is realized by Sony EuTEC starting from the work of Meister et al.[3].

We used this dataset for the training and the performance evaluation of the proposed method.

For each scene sample in the datasets, the following data are provided:

  • The 320x240 ToF depth maps captured at 20, 50 and 60 MHz. The depth maps are already phase unwrapped to have an unambiguos range of 15 meters.
  • The 320x240 ToF amplitude images captured at 20, 50 and 60 MHz.
  • The 320x240 ToF intensity images captured at 20, 50 and 60 MHz.
  • The ground truth depth map w.r.t. the ToF point of view.

    •  

    The real dataset contains the data acquired with a SoftKinetic ToF camera at 20, 50 and 60 MHz. The ToF camera has been calibrated in order to correct the wiggling error acting on the depth and amplitude [4]. We recorded the data on a wooden box (depicted below) that is about 1.5 meters wide and 1 meter high and composed by a 90o and a 135o angle. We used this scene for the acquisitions since corners are the typical scenario where MPI is acting. The real world dataset is composed by 8 scenes, each captured by looking at the box from different viewpoints and by placing in it objects made of wood, polystyrene and ceramic. We used an active stereo system in combination with the ToF camera in order to estimate a reliable depth of the scenes. The 2 acquisitions systems are geometrically calibrated in order to project the ground truth depth map from the stereo to the ToF sensor.

    We used this dataset for evaluation only.

    For each scene sample in the dataset, the following data are provided:

  • The calibrated 320x239 ToF depth maps captured at 20, 50 and 60 MHz. The depth maps are phase unwrapped to have an unambiguos range of 15 meters.
  • The calibrated 320x239 ToF amplitude images captured at 20, 50 and 60 MHz.
  • The 320x239 ToF intensity images captured at 20, 50 and 60 MHz.
  • The ground truth depth map w.r.t. the ToF point of view.

    •  

    flowchart

    Wooden box used for the ToF acquisitions

     

     

    Download

    • Synthetic Dataset and Real Dataset (further information about the data is available here)

      If you use these datasets please cite [1]. The datasets can be used for research purpose only.

      A subset of the scenes contained in the synthetic dataset has also been used for the experimantal results of the method for MPI correction presented in [2]. The paper can be found here and the additional material here.

     

    Contacts

    For any information you can write to lttm@dei.unipd.it . Have a look at our website http://lttm.dei.unipd.it for other works and datasets on this topic.

     

    References

    [1]   G. Agresti and P. Zanuttigh, Deep learning for multi-path error removal in ToF sensors, Geometry Meets Deep Learning, ECCVW18, Munich, Germany, 2018.

    [2]   G. Agresti and P. Zanuttigh, Combination of spatially-modulated ToF and structured light for MPI-free depth estimation, 3D Reconstruction in the Wild, ECCVW18, Munich, Germany, 2018.

    [3]   M. Lindner and A. Kolb, Lateral and depth calibration of PMD-distance sensors, International Symposium on Visual Computing. Springer, Berlin, Heidelberg, 2006.

    [4]   S. Meister et al., Simulation of time-of-flight sensors using global illumination, In Bronstein, M., Favre, J., Hormann, K., eds.: Vision, Modeling and Visualization, The Eurographics Association (2013)

     

    xhtml/css website layout by Ben Goldman  - http://realalibi.com