Unveiling limitations in single-view 3D object reconstruction models

3D object reconstruction models typically learn from a single dataset, ShapeNetCore, and are evaluated against similar datasets that measure aspects closely related to ShapeNetCore. We tackled the problem of limited performance assessment by proposing novel benchmarks to reveal their robustness to new challenges. To demonstrate our benchmark's effectiveness, we selected three state-of-the-art models for comparison: 3D-C2FT, Pix2Vox, and Occupancy Networks. This selection covers two well-known 3D shape representations: voxel and occupancy function as an implicit representation. We first investigated the effect of changing background color on performance. We found that this seemingly simple variable causes a drastic decrease in performance. We observed that models perform sufficiently close to the original scenario with changing input object sizes in 2D. Further, we adapted a novel dataset 3DCoMPaT++ for 3D reconstruction evaluation. 3DCoMPaT++ offers rich material and part annotations. We assessed reconstruction performance by slightly changing viewpoints and varying styles in 2D input images. The results show that models struggle to adapt to novel settings. Performance degrades drastically in the best-case scenario, and surprisingly, the model performing the best in the standard ShapeNetCore experiment, Pix2Vox, scores the worst across all novel dataset experiments. We also evaluated models at the part level to identify the most challenging parts. We transferred part-level point clouds to part-annotated surface points from 3DCoMPaT++ using point cloud registration with L2 distance. We then utilized our version of F-Score@0.01, Part F-Score@0.01, for evaluation. This experiment quantitatively confirmed the known issue of poor performance in finer details and thin parts, unlike previous works that only made qualitative observations.