Jun Gao on Towards Generative Modeling of 3D Objects Learned from Images | Toronto AIR Seminar

Abstract: With the increasing demand for creating large 3D virtual worlds in many industries, there is an immense need for diverse and interesting 3D content. A.I. is existentially enabling this quest. In this talk, I will discuss our recent work on 3D object synthesis in the form of textured meshes, allowing them to be easily used in existing graphics engines. We devise a new differentiable 3D representation based on a tetrahedral grid to enable high-quality recovery of 3D mesh with arbitrary topology. With inspiration from Nvdiffrec (Munkberg, et. al.), we further develop a generative model to combine differentiable iso-surfacing models with differentiable rendering and produce complex textures with materials for mesh generation. Our model further enables a novel avenue in interactive tooling for 3D mesh creation, making it accessible to novice users. Paper: Shen, Tianchang, et al. "Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis." NeurIPS. 2021. https://nv-tlabs.github.io/DMTet/ Munkberg, Jacob, et al. "Extracting Triangular 3D Models, Materials, and Lighting From Images." CVPR. 2022. https://nvlabs.github.io/nvdiffrec/ Gao, Jun, et al. "GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images." NeurIPS. 2022. https://nv-tlabs.github.io/GET3D/ Bio: Jun Gao is a PhD student at the University of Toronto with Prof. Sanja Fidler, he is also a Research Scientist at Nvidia’s Toronto AI lab, led by Sanja Fidler. He obtained his Bachelor’s degree from Peking University in 2018. His research mainly focuses on 3D deep learning and its interaction with 2D images. Toronto AIR Seminar: The Toronto AI Robotics Seminar Series is a set of events featuring young robotics and AI experts. The talks are given by local as well as global speakers and organized by the Faculty and Students at University of Toronto’s Department of Computer Science. We welcome students, researchers and robotics enthusiasts from around the world to join us and interact with the Toronto Robotics Community. Find out more at: https://robotics.cs.toronto.edu/toron...