Visual-Tactile Fusion for 3D Objects Reconstruction from a Single Depth View and a Single Gripper Touch for Robotics Tasks

The planning of robotic manipulation and grasping tasks depends on the reconstruction of the 3D object's shape. Most of the existing 3D object reconstruction methods are based on visual sensing that are limited due to the lack of the object's occluded side information. The goal of this paper is to overcome these limitations and improve the 3D objects' reconstruction by adding the tactile sensing to the visual data. In this paper, a novel multi-modal (visual and tactile) semi-supervised generative model is presented to reconstruct the complete 3D object's shape using a single arbitrary depth-view and a single dexterous-hand's touch. The presented approach takes the strength of the autoencoder and generative networks to provide an end-to-end trainable model with high generalization ability. The 3D voxel grids of the depth and tactile data are the only requirements of the proposed model to predict a high resolution voxel grids of 64(3) for the incomplete shape. This research generates its tactile dataset based on the kinematic model of the shadow dexterous hand. The developed dataset has aligned depth, tactile and ground truth voxel grids of different resolutions (40(3), 64(3) and 128(3)) from different camera views. Experimental results show that the proposed multi-modal model outperforms other state-of-the-art methods.