GSIR: Generalizable 3D Shape Interpretation and Reconstruction

Jianren Wang
Zhaoyuan Fang
CMU
CMU
ECCV 2020
[Download Paper]
[GitHub Code]
[Slides]
[Poster]



3D shape interpretation and reconstruction are closely related to each other but have long been studied separately and often end up with priors that are highly biased towards the training classes. In this paper, we present an algorithm, Generalizable 3D Shape Interpretation and Reconstruction (GSIR), designed to jointly learn these two tasks to capture generic, class-agnostic shape priors for a better understanding of 3D geometry. We propose to recover 3D shape structures as cuboids from partial reconstruction and use the predicted structures to further guide full 3D reconstruction. The unified framework is trained simultaneously offline to learn a generic notion and can be fine-tuned online for specific objects without any annotations. Extensive experiments on both synthetic and real data demonstrate that introducing 3D shape interpretation improves the performance of single image 3D reconstruction and vice versa, achieving the state-of-the-art performance on both tasks for objects in both seen and unseen categories.


Overview of the Algorithm

Our whole model consists of four learnable functions connected by five deterministic projection functions.




Paper and Bibtex

Citation
 
Jianren Wang, Zhaoyuan Fang.
GSIR: Generalizable 3D Shape Interpretation and Reconstruction
In ECCV 2020.

[Bibtex] [Paper] [ArXiv]
@inproceedings{wang2020gsir,
    title={GSIR: Generalizable 3D Shape Interpretation and Reconstruction},
    author={Wang, Jianren and Fang, Zhaoyuan},
    booktitle={European Conference on Computer Vision},
    pages={498--514},
    year={2020},
    organization={Springer}
}


Acknowledgements

We would like to thank Shubham Tulsiani and members of the CMU Center for Perceptual Computing and Learning for their insightful advice and technical support. This work was supported by PanGU Young Investigator Award to JW.