A robust and accurate facial 3D reconstruction method from images acquired by mobile devices at home for facial growth monitoring

Diseases affecting facial growth require highly accurate facial 3D scan to diagnose, monitor and plan treatment. Currently, the gold standard for capturing such data is to use digital stereo-photogrammetry systems in a clinical setting with healthcare professionals allowing for sub-millimetre accuracy. Recently, 3D reconstruction from facial images captured by smartphone has shown potential as an alternative in the same settings [9-12]. The project in which this PhD takes place aims to instead use patients’ smartphones at home to acquire by themselves images suitable for highly accurate facial 3D reconstruction through a purpose-made mobile application. 

In Computer Vision, highly accurate facial 3D models can be obtained from multiple images with optimization-based methods relying on motion analysis and geometrical constraints [3,4]. Recently, deep learning approaches combining geometry and light properties estimation have emerged [5-7]. While they achieve impressive results from images captured by laypersons, these methods do not leverage all the progress made to integrate multi-view geometry constraints in deep learning-based Structure-from-Motion [1,2]. They also often rely on 3DMM models which are not representative of the morphology of patients with facial dysmorphology. 

In the proposed research, we will build upon these recent works and aim to combine the strengths of traditional geometry-based 3D reconstruction with the versatility of deep-learning-based facial reconstruction to handle facial dysmorphology. The project will aim to reach the sub-millimetre accuracy required for measuring anatomical structures for facial growth monitoring while being as robust as possible to real world conditions of home acquisition. 

References: 

[1] Xingkui W, et al. DeepSFM: Structure From Motion Via Deep Bundle Adjustment. Proc. of the IEEE/CVF Eur. Conf. on Computer Vision, 2020. 

[2] Jianyuan W, et al. Deep Two-View Structure-from-Motion Revisited. Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2021. 

[3] Agrawal S, et al. High Accuracy Face Geometry Capture using a Smartphone Video. Proc. of the IEEE/CVF Winter Conf. on Applications of Computer Vision, 2020. 

[4] Booth J, et al. 3D Reconstruction of “In-the-Wild” Faces in Images and Videos. IEEE Trans. on Pattern Analysis and Machine Intelligence (Vol. 40, Issue: 11, Nov. 2018) 

[5] Yandong W, et al. Self-Supervised 3D Face Reconstruction via Conditional Estimation. Proc. of the IEEE/CVF Int. Conf. on Computer Vision, 2021. 

[6] Abdallah D, et al. Towards High Fidelity Monocular Face Reconstruction with Rich Reflectance using Self-supervised Learning and Ray Tracing. Proc. of the IEEE/CVF Int. Conf. on Computer Vision, 2021. 

[7] Tianye Li, et al. Topologically Consistent Multi-View Face Inference Using Volumetric Sampling. Proc. of the IEEE/CVF Int. Conf. on Computer Vision, 2021. 

[9] Nightingale RC, et al. A Method for Economical Smartphone-Based Clinical 3D Facial Scanning. Journal of Prosthodontics, 2020;29(9):818-825. 

[10] Mai H and Lee D. Accuracy of Mobile Device–Compatible 3D Scanners for Facial Digitization: Systematic Review and Meta-Analysis. Journal Medical Internet Research, 2020. 

[11] Gallardo Y, et al. Evaluation of the 3D error of 2 face-scanning systems: An in vitro analysis. Journal of Prothestic Dentistry, 2021. 

[12] Salazar-Gamarra R, et al. Monoscopic photogrammetry to obtain 3D models by a mobile device: a method for making facial prostheses. J. Otolaryngol Head Neck Surg. 2016;45(1):33.