Sign Language Recognition for Patient-Doctor Communication: A Multimedia/Multimodal Dataset

Sign Language translation increases the life quality of deaf people and their social integration. It is an extraordinarily challenging task that requires detecting hand gestures, facial expressions, upper body movements, and finding a temporal relationship among them and the translated words. This paper explores this complex task by a sensor fusion approach, with the objective of defining a medical vocabulary for patient-doctor communication translation for the Italian Sign Language (LIS). Indeed, the availability of a multimodal dataset can be a key factor in supporting machine learning models for Sign Language recognition. In this context, a multimedia/multimodal database has been developed by collecting synchronized data for face, manual, and body from different sensors, i.e. a mm-wave RADAR, a lidar, RGB and RGB-D cameras.