Paulo Gotardo
gotardop <at> gmail <dot> com
I'm a research engineer/manager with Google XR in Zurich, developing core digital human technologies (computer vision & graphics, machine learning & GenAI) to ingest innovation and impact Google's apps, services, and platforms that are used by billions of people.
After having previously spent 10 years working with Disney Research in Pittsburgh & Zurich, where I helped give superpowers to actors in movies, I am now thrilled to help give superpowers to virtually everyone through Google XR.
More about me:
Prior to joining Disney Research, I received my BSc (2000) and MSc (2002) degrees in Informatics from Federal University of Parana (UFPR), Brazil, and my PhD degree (2010) in Electrical and Computer Engineering from The Ohio State University (OSU), USA. While at OSU, I was also a postdoc at the Computational Biology and Cognitive Science Lab (CBCSL) and a graduate research associate with the Advanced Computing Center for the Arts and Design (ACCAD).
Research Work
Here's a list of conference and journal papers describing some of my research work. If you are looking for source code related to my work on nonrigid structure-from-motion, you can jump to my old OSU page here.
M. Bühler, G. Li, E. Wood, L. Helminger, X. Chen, T. Shah, D. Wang, S. Garbin, S. Orts-Escolano, O. Hilliges, D. Lagun, J. Riviere, P. Gotardo, T. Beeler, A. Meka, K. Sarkar
Cafca: High-quality Novel View Synthesis of Expressive Faces from Casual Few-shot Captures
Siggraph Asia 2024
We present a novel volumetric prior on human faces that allows for high-fidelity expressive face modeling from as few as three input views captured in the wild. Our key insight is that an implicit prior trained on synthetic data alone can generalize to extremely challenging real-world identities and expressions and render novel views with fine idiosyncratic details like wrinkles and eyelashes. [Project Page]
A. Comas, D. Qiu, M. Chai, M. Buehler, A. Raj, R. Gao, Q. Xu, M. Matthews, P. Gotardo, S. Orts-Escolano, T. Beeler
MagicMirror: Fast and High-Quality Avatar Generation with Constrained Search Space
ECCV 2024
We introduce a novel framework for 3D human avatar generation and customization via text prompts. We utilize a conditional Neural Radiance Fields (NeRF) model as a subspace prior for avatar generation with Text-to-Image Diffusion Models. [Project Page]
Prashanth Chandran, Gaspard Zoss, Paulo Gotardo, Derek Bradley
Infinite 3D Landmarks: Improving Continuous 2D Facial Landmark Detection
Computer Graphics Forum 2024
In this work, we examine 3 important issues in the practical use of state-of-the-art facial landmark detectors and show how a combination of specific architectural modifications can directly improve their accuracy and temporal stability. [Project Page]
G. Li, K. Sarkar, A. Meka, M. Buehler, F. Mueller, P. Gotardo, O. Hilliges, T. Beeler
ShellNeRF: Learning a Controllable High-Resolution Model of the Eye and Periocular Region
Eurographics 2024
We present ShellNeRF - a novel method for high-resolution novel view synthesis and animation of the periocular face region. Our method allows for controlling expressions and eye gaze and renders novel views at an unprecedented level of detail. [Project Page]
Lingchen Yang, Gaspard Zoss, Prashanth Chandran, Paulo Gotardo, Markus Gross, Barbara Solenthaler, Eftychios Sifakis, Derek Bradley
An Implicit Physical Face Model Driven by Expression and Style
SIGGRAPH Asia 2023
We propose a face model based on a data-driven, implicit neural physics that can be driven by both expression and style separately. At the core, we present a framework for learning implicit physics-based actuations for multiple subjects simultaneously, trained on a few arbitrary performance capture sequences from a small set of identities. [Project Page]
Kripasindhu Sarkar, Marcel C. Bühler, Gengyan Li, Daoye Wang, Delio Vicini, Jérémy Riviere, Yinda Zhang, Sergio Orts-Escolano, Paulo Gotardo, Thabo Beeler, Abhimitra Meka
LitNeRF: Intrinsic Radiance Decomposition for High-Quality View Synthesis and Relighting of Faces
SIGGRAPH Asia 2023
We present a novel technique for high-quality capture of a human face for 3D view synthesis and relighting using a sparse, compact capture rig consisting of 15 cameras and 15 lights. Our method combines a volumetric representation of the face reflectance with traditional multi-view stereo based geometry reconstruction. The proxy geometry allows us to anchor the 3D density field to prevent artifacts and guide the disentanglement of intrinsic radiance components of the face appearance such as diffuse and specular reflectance, direct and indirect light transport fields. Our hybrid representation significantly improves the state-of-the-art quality for arbitrarily dense renders of a face from desired camera viewpoint as well as environmental, directional, and near-field lighting. [Project Page]
Yingyan Xu, Gaspard Zoss, Prashanth Chandran, Markus Gross, Derek Bradley, Paulo Gotardo
ReNeRF: Relightable Neural Radiance Fields with Nearfield Lighting
ICCV 2023
We propose ReNeRF, a relightable radiance field model based on the intuitive and powerful approach of image-based relighting, which implicitly captures global light transport for arbitrary objects without complex, error-prone simulations. ReNeRF is simple and provides full control over viewpoint and lighting, without simplistic assumptions about how light interacts with the scene. ReNeRF generalizes to novel, continuous lighting directions, including nearfield lighting effects. [Project Page]
Christopher Otto, Prashanth Chandran, Gaspard Zoss, Markus Gross, Paulo Gotardo, Derek Bradley
A Perceptual Shape Loss for Monocular 3D Face Reconstruction
Pacific Graphics 2023
In this work, we propose a new loss function for monocular face capture, inspired by how humans would perceive the quality of a 3D face reconstruction given a particular image. It is widely known that shading provides a strong indicator for 3D shape in the human visual system. As such, our new perceptual shape loss aims to judge the quality of a 3D face estimate using only shading cues. [Project Page]
Sebastian Weiss, Jonathan Moulin, Prashanth Chandran, Gaspard Zoss, Paulo Gotardo, Derek Bradley
Graph-Based Synthesis for Skin Micro Wrinkles
Eurographics Symposium on Geometry Processing 2023
We present a novel graph-based simulation approach for generating micro wrinkle geometry on human skin, which can easily scale up to the micro-meter range and millions of wrinkles. [Project Page]
Prashanth Chandran, Gaspard Zoss, Paulo Gotardo, Derek Bradley
Continuous Landmark Detection with 3D Queries
CVPR 2023
We propose the first facial landmark detection network that can predict continuous, unlimited landmarks. Our method allows the user to specify the number and location of the desired landmarks at inference time, as continuous 3D query points relative to a 3D template model. [Project Page]
T. Schnabel, B. Gözcü, P. Gotardo, L. Lingens, D. Dorda, F. Vetterli, A. Emhemmed, P. Nalabothu, Y. Lill, B. Benitez, A. Mueller, M. Gross, B. Solenthaler
Automated and Data-Driven Plate Computation for Presurgical Cleft Lip and Palate Treatment
IPCAI 2023 (Bench to Bedside Award)
Cleft lip and palate is the most frequent craniofacial malformation in newborns, without effective preventive measures. The use of intra-oral orthopedic plates reduces the cleft size, facilitating surgical treatment. This project, Burden-Reduced Cleft Lip and Palate Care and Healing (BRCCH), aims at an automatic, image-based design (e.g., using smartphone videos) of personalized oral plates that are fabricated using 3D printers. Ultimately, the goal is to facilitate the use of plate therapy in low-income countries. This project is funded by the Botnar Research Center for Child Health (BRCCH) and implemented in collaboration with the Computer Graphics Lab (CGL) at ETH Zürich and the team of Dr. Andreas Müller from the University Hospital in Basel. [Project page]
Gaspard Zoss, Prashanth Chandran, Eftychios Sifakis, Markus Gross, Paulo Gotardo, Derek Bradley
Production-Ready Face Re-Aging for Visual Effects
SIGGRAPH Asia 2022
This paper presents the first practical, fully-automatic and production-ready method for re-aging faces in video images. We show how a longitudinal re-aging dataset can be constructed using a state-of-the-art facial re-aging method that, although failing on real images, does provide photoreal re-aging on synthetic faces. We leverage such synthetic data and formulate facial re-aging as a practical image-to-image translation task with a simple U-Net. [Project page]
C. Otto, J. Naruniec, L. Helminger, T. Etterlin, G. Mignone, P. Chandran, G. Zoss, C. Schroers, M. Gross, P. Gotardo, D. Bradley, R. Weber
Learning Dynamic 3D Geometry and Texture for Video Face Swapping
Pacific Graphics 2022
We approach face swapping as learning simultaneous facial autoencoders for the source and target identities, using a shared encoder network with identity-specific decoders. Our decoders first lift the latent code into a 3D representation, before using a differentiable renderer, thus allowing for artistic control over the result. Training does not require 3D supervision, leading to better results than when using off-the-shelf monocular 3D face reconstruction. [Project page]
Prashanth Chandran, Gaspard Zoss, Markus Gross, Paulo Gotardo, Derek Bradley
Facial Animation with Disentangled Identity and Motion using Transformers
SIGGRAPH/EG Symposium on Computer Animation, SCA 2022
We present a 3D+time morphable model that learns a motion manifold using a transformer autoencoder. This new model can synthesize temporal sequences of 3D meshes with arbitrary length and identity. [Project page]
Daoye Wang, Prashanth Chandran, Gaspard Zoss, Derek Bradley, Paulo Gotardo
MoRF: Morphable Radiance Fields for Multiview Neural Head Modeling
SIGGRAPH 2022
We present MoRF, morphable radiance fields that extend NeRFs into generative models for synthesizing photorealistic human heads with controllable and fully disentangled identity and 3D pose. MoRF allows for applications such as synthesizing new photorealistic subjects or quickly fitting a NeRF to one or more full-head portrait images. [Project page]
Sebastian Winberg, Gaspard Zoss, Prashanth Chandran, Paulo Gotardo, Derek Bradley
Facial Hair Tracking for High Fidelity Performance Capture
SIGGRAPH 2022 (Best Paper Honorable Mention)
We reconstruct and track individual facial hairs over complex performance sequences in a traditional multiview setup. We additionally create a realistic approximation of the dynamic clean-shaven facial surface, as if the actor had been captured without facial hair, thus removing the need to actually shave. [Project page]
Yingyan Xu, Jérémy Riviere, Gaspard Zoss, Prashanth Chandran, Derek Bradley, Paulo Gotardo,
Improved Lighting Models for Facial Appearance Capture
Eurographics 2022, short paper
We compare the results obtained with a state-of-the-art appearance capture method [RGB∗20], with and without our proposed improvements to the lighting model. [Project page]
Prashanth Chandran, Gaspard Zoss, Markus Gross, Paulo Gotardo, Derek Bradley
Shape Transformers: Topology-Independent 3D Shape Models Using Transformers
Eurographics 2022
We present a new nonlinear parametric 3D shape model based on transformer architectures. [Project page]
Prashanth Chandran, Sebastian Winberg, Gaspard Zoss, Jérémy Riviere, Markus Gross, Paulo Gotardo, Derek Bradley
Rendering with Style: Combining Traditional and Neural Approaches for High-Quality Face Rendering
SIGGRAPH Asia 2021
We propose to combine incomplete, high-quality renderings showing only facial skin with recent methods for neural rendering of faces, in order to automatically and seamlessly create photo-realistic full-head portrait renders from captured data without the need for artist intervention. [Project page]
Prashanth Chandran, Gaspard Zoss, Markus Gross, Paulo Gotardo, Derek Bradley
Adaptive Convolutions for Structure-Aware Style Transfer
CVPR 2021
We propose Adaptive convolutions; a generic extension of AdaIN, which allows for the simultaneous transfer of both statistical and structural styles in real time. [Project page]
Jérémy Riviere, Paulo Gotardo, Derek Bradley, Abhijeet Ghosh, Thabo Beeler
Single-Shot High-Quality Facial Geometry and Skin Appearance Capture
SIGGRAPH 2020
We propose a new light-weight face capture system capable of reconstructing both high-quality geometry and detailed appearance maps from a single exposure. [Project page]
Paulo Gotardo, Jérémy Riviere, Derek Bradley, Abhijeet Ghosh, Thabo Beeler
Practical Dynamic Facial Appearance Modeling and Acquisition
SIGGRAPH Asia 2018
We present a method to acquire dynamic properties of facial skin appearance, including dynamic diffuse albedo encoding blood flow, dynamic specular intensity, and per-frame high resolution normal maps for a facial performance sequence. [Project page]
Zdravko Velinov, Marios Papas, Derek Bradley, Paulo Gotardo, Parsa Mirdehghan, Steve Marschner, Jan Novak, Thabo Beeler
Appearance Capture and Modeling of Human Teeth
SIGGRAPH Asia 2018
We present a system specifically designed for capturing the optical properties of live human teeth such that they can be realistically re-rendered in computer graphics. [Project page]
Dan Calian, Tomas Simon, Paulo Gotardo, Jean-François Lalonde, Iain Matthews, Kenny Mitchell
From Faces to Outdoor Light Probes
Eurographics 2018
This paper presents an approach to directly estimate an HDR light probe from a single LDR photograph, shot outdoors with a consumer camera, without specialized calibration targets or equipment. [Project page]
Paulo Gotardo, Tomas Simon, Yaser Sheikh, Iain Matthews
Photogeometric Scene Flow for High-Detail Dynamic 3D Reconstruction
ICCV 2015
This paper proposes photogeometric scene flow (PGSF) for high-quality dynamic 3D reconstruction. Results are obtained as the coupled solution of multiview stereo, photometric stereo, and optical flow (with relighting). [Project page]
Yannick Hold-Geoffroy, Jinsong Zhang, Paulo Gotardo, Jean-François Lalonde
Single Day Outdoor Photometric Stereo (IEEE Trans. on PAMI, accepted 2019) [Project page]
X-hour Outdoor Photometric Stereo (3DV 2015, Best Paper Award, runner up) [Project page]
What is a Good Day for Outdoor Photometric Stereo? (ICCP 2015) [Project page]
This work investigates the numerical conditioning and solutions for outdoor photometric stereo under uncontrolled, natural illumination in which the main light source, the Sun, shines from nearly co-planar directions throughout the day. We show the events that contribute to making the problem solvable over variable weather and short time intervals.
Paulo Gotardo, Onur Hamsici, Aleix Martinez
Learning Spatially-Smooth Mappings in Non-Rigid Structure from Motion (ECCV 2011)
Kernel Non-Rigid Structure from Motion (ICCV 2011)
Non-Rigid Structure from Motion with Complementary Rank-3 Spaces (CVPR 2011)
Computing Smooth Time-Trajectories for Camera and Deformable Shape in Structure from Motion with Occlusion (IEEE Trans. PAMI, 33(10), 2011)
While I was a PhD student and then postdoc at The Ohio State University, I developed state-of-the-art models and algorithms for matrix factorization and non-rigid structure from motion (NR-SfM), which were published in main computer vision venues and subsequently achieved 2nd, 3rd, and 4th places in the first NR-SfM challenge at CVPR 2017. More info and source codes are found on my old OSU home page, here: [Project page]
Paulo Gotardo, Alan Price
Integrated Space: Authoring in an Immersive Environment with 3D Body Tracking
ACM SIGGRAPH 2010 (Posters)
In a time before Microsoft's Kinect, this project explored the use of real-time stereo vision and skeletonization to provide 3D human body awareness in an inexpensive, immersive environment system. The goal was to enhance the user experience of immersion in a virtual scene projected in 3D, allowing for both the user and the virtual scene to become aware of each other's presence as part of a single, integrated 3D space. We focused on enabling authoring applications with direct manipulation of virtual objects, with users interacting from a first-person perspective (demo video). This emphasis contrasts with the avatar-based, reactive focus of game interfaces. For more info, please see my old OSU page: [Project page]
Paulo Gotardo, Kim L. Boyer, Joel Saltz, Subha V. Raman
A New Deformable Model for Boundary Tracking in Cardiac MRI and Its Application to the Detection of Intra-ventricular Dyssynchrony
CVPR 2006
Intra-ventricular dyssynchrony (IVD) in the left ventricle (LV) is caused by the asynchronous activation of the LV walls. Guidelines for resynchronization therapy rely on measures that do not reliably predict successful patient response to treatment, in part due to poor characterization of IVD. We present a two-class statistical pattern recognition approach for the detection of IVD in the LV from routinely acquired MRI sequences depicting complete cardiac cycles.
Paulo Gotardo, Olga R.P. Bellon, Luciano Silva, Kim L. Boyer
Range Image Segmentation into Planar and Quadric Surfaces Using an Improved Robust Estimator and Generic Algorithm (IEEE Trans. Systems, Man, and Cybernetics - Part B, 34(6), 2004)
Range Image Segmentation by Surface Extraction Using an Improved Robust Estimator (CVPR 2003)
We present a novel robust estimator to iteratively detect and extract distinct planar and quadric surface patches in depth images. Our robust estimator extends M-estimator Sample Consensus/Random Sample Consensus (MSAC/RANSAC) to use local surface orientation, enhancing inlier/outlier classification when processing noisy range data describing multiple structures. An efficient approximation to the true geometric distance between a point and a quadric surface is also proposed. A genetic algorithm was specifically designed to accelerate the optimization process.