[Collaborative Research]

Spatio-Temporally Consistent Face Mesh Reconstruction on Videos

Kim Youwang, Lee Hyun, Kim Sung-Bin, Suekyeong Nam, Janghoon Ju, Tae-Hyun Oh

Abstract

We propose STREAM, a 3D parametric face reconstruction method on videos with spatio-temporal consistency. We design a novel optimization method that fits the neural reparameterized 3D face meshes on videos. By leveraging spatio-temporal cues inherent in face videos, we develop novel multi-view and temporal consistency losses. We harmonize our spatio-temporal losses with a 2D landmark loss, and fit the per-view/-frame accurate and consistent face meshes on videos. The results show that our method nicely infuses spatio-temporal consistency into reconstructed 3D faces with favorable reconstruction accuracy

CVPR 2023 Workshop