Neural Network Based on Convolutional, Recurrent Layers and an Attention Mechanism for Visual Speech Recognition

Abstract

Visual speech recognition is the task of classifying spoken words or letters from a video stream capturing lip movements. This paper presents the synthesis and study of a neural network architecture for visual speech recognition based on a combination of convolutional and recurrent neural networks with an attention mechanism. The model was trained and evaluated on the AVLetters2 dataset in the most challenging speakerindependent mode. The model architecture includes an encoder based on convolutional layers for extracting spatial features, recurrent layers based on GRU units for modeling temporal dependencies, and an attention mechanism for highlighting informative fragments of the speech sequence. To assess the accuracy of the model, five-fold cross-validation was performed. Model hyperparameters were selected using Bayesian optimization, which allowed us to determine the optimal configuration of the model parameters and the training process. As a result of the experiments, an average recognition accuracy of 14.3 % was achieved. Analysis of the results revealed signi­ ficant variability in recognition quality depending on the characteristics of the speakers (accuracy ranged from 3.9 to 31.9 %), which indicates the need to further improve the invariance of the model to inter-speaker differences.