Ghost on the Shell:
An Expressive Representation of General 3D Shapes

ICLR 2024 (Oral)

Zhen Liu1,2, Yao Feng1,3,*, Yuliang Xiu1,*, Weiyang Liu 1,4†
Liam Paull 2, Michael J. Black 1,‡, Bernhard Schölkopf 1,‡,
1Max Planck Institute for Intelligent Systems,
2Mila, Université de Montréal, 3ETH Zürich, 4University of Cambridge
*Equal contribution   †Directional lead   ‡Shared last author
arXiv Paper Code

The creation of photorealistic virtual worlds requires the accurate modeling of 3D surface geometry for a wide range of objects. For this, meshes are appealing since they 1) enable fast physics-based rendering with realistic material and lighting, 2) support physical simulation, and 3) are memory-efficient for modern graphics pipelines. Recent work on reconstructing and statistically modeling 3D shape, however, has critiqued meshes as being topologically inflexible. To capture a wide range of object shapes, any 3D representation must be able to model solid, watertight, shapes as well as thin, open, surfaces. Recent work has focused on the former, and methods for reconstructing open surfaces do not support fast reconstruction with material and lighting or unconditional generative modelling. Inspired by the observation that open surfaces can be seen as islands floating on watertight surfaces, we parameterize open surfaces by defining a manifold signed distance field on watertight templates. With this parameterization, we further develop a grid-based and differentiable representation that parameterizes both watertight and non-watertight meshes of arbitrary topology. Our new representation, called Ghost-on-the-Shell (G-Shell), enables two important applications: differentiable rasterization-based reconstruction from multiview images and generative modelling of non-watertight meshes. We empirically demonstrate that G-Shell achieves state-of-the-art performance on non-watertight mesh reconstruction and generation tasks, while also performing effectively for watertight meshes.

Take-away Message

G-Shell models both watertight and non-watertight meshes of different shape topology in a differentiable way. Mesh extraction with G-Shell is stable -- no need to compute MLP gradients but simply do sign checks on grid vertices.
G-Shell enables

  • Differentiable rasterization-based inverse rendering of non-watertight meshes. No need to compute integrals on ray samples but simply rasterize your meshes in a parallel manner. As a result, physics-based inverse rendering for non-watertight meshes is made easy with G-Shell.
  • Template-free non-watertight generation. A grid of SDF and manifold SDF (mSDF) scale values are sufficient to parameterize a mesh -- therefore, generation is simply to generate a grid of scalars. Methods like diffusion models can be extended to non-watertight meshes without much effort.

Core Idea

A non-watertight mesh can be seen as some island floating on a watertight shell. We define a manifold signed distance field (mSDF) on the shell. The desired open surface is therefore bounded by mSDF zero isolines. To extract non-watertight meshes is therefore to identify zero isolines.

Reconstruction Show Cases

Rasterization-based inverse rendering with G-Shell can easily handle images rendered with realistic lighting and materials.
We show results on DeepFashion3D datasets. All topological details are almost perfectly reconstructed with G-Shell.





Some seemly difficult shapes can also be easily reconstructed with G-Shell.


Hybrid Watertight and Non-Watertight Reconstruction

Inverse rendering with G-Shell can produce watertight meshes and non-watertight meshes at the same time. If only the upper surface is visible, a single-layered mesh is reconstructed without adding any lower surface.


G-Shell with FlexiCubes

G-Shell can be implemented with FlexiCubes for better mesh topology on non-watertight meshes.




Physical Simulation of Reconstructed Mesh

We show a physical simulation video of a clothing mesh reconstructed with G-Shell, along with the corresponding ground truth mesh (with the same motion sequence on the same SMPL-X human body).


Generation


Unconditional Generation

The grid structure of G-Shell enables diffusion models. We propose G-MeshDiffusion, a diffusion model for non-watertight mesh generation. Training G-MeshDiffusion is easy: simply collect a dataset with G-Shell-based reconstruction and train a 3D U-Net-based diffusion model.



Interpolation

G-MeshDiffusion captures the underlying semantics and learns to perform reasonable interpolation between samples.


Halloween Ghost on the Shell


Acknowledgement

We sincerely thank Peter Kulits for paper proofreading and Zhouyingcheng Liao for creating physical simulation demos. Authors listed as equal contribution (resp., shared last author) are allowed to switch their orders in the author list in their resumes and websites. The paper title was proposed during an after-dinner coffee chat among Zhen Liu, Yao Feng, Yuliang Xiu, Weiyang Liu and Tim Xiao, especially due to Yuliang Xiu (for proposing "Shell") and Weiyang Liu (for the connection to the manga series Ghost in the Shell).



Disclosure. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039B, and by the Machine Learning Cluster of Excellence, EXC number 2064/1 - Project number 390727645. MJB has received research gift funds from Adobe, Intel, Nvidia, Meta/Facebook, and Amazon. MJB has financial interests in Amazon, Datagen Technologies, and Meshcapade GmbH. While MJB is a part-time employee of Meshcapade, his research was performed solely at, and funded solely by, the Max Planck Society. LP is supported by the Canada CIFAR AI Chairs Program and NSERC Discovery Grant. WL was supported by the German Research Foundation (DFG): SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms, TP XX, project number: 276693517. YF is partially supported by the Max Planck ETH Center for Learning Systems. YX is funded by the European Union's Horizon $2020$ research and innovation programme under the Marie Skłodowska-Curie grant agreement No.$860768$ (CLIPE).

BibTeX

@inproceedings{Liu2024gshell,
    title={Ghost on the Shell: An Expressive Representation of General 3D Shapes},
    author={Liu, Zhen and Feng, Yao and Xiu, Yuliang and Liu, Weiyang 
      and Paull, Liam and Black, Michael J and Sch{\"o}lkopf, Bernhard},
    booktitle={ICLR},
    year={2024}
}