Multimodal Face Data Sets—A Survey of Technologies, Applications, and Contents

Due to their ease-of-use, biometric verification methods to control access to digital devices

have become ubiquitous. Many rely on supervised machine learning, a process that is notoriously data-

hungry. At the same time, biometric data is sensitive from a privacy perspective, and a comprehensive review

from a data set perspective is lacking. In this survey, we present a comprehensive review of multimodal

face data sets (e.g., data sets containing RGB color plus other channels such as infrared or depth). This

follows a trend in both industry and academia to use such additional modalities to improve the robustness

and reliability of the resulting biometric verification systems. Furthermore, such data sets open the path to

a plethora of additional applications, such as 3D face reconstruction (e.g., to create avatars for VR and

AR environments), face detection, registration, alignment, and recognition systems, emotion detection,

anti-spoofing, etc. We also provide information regarding the data acquisition setup and data attributes

(ethnicities, poses, facial expressions, age, population size, etc.) as well as a thorough discussion of related

applications and state-of-the-art benchmarking. Readers may thus use this survey as a tool to navigate the

existing data sets both from the application and data set perspective. To existing surveys we contribute, to

the best of our knowledge, the first exhaustive review of multimodalities in these data sets.