Lip Reading Using Computer Vision and Deep Learning

Abstract: More than 13% of U.S. adults suffer from hearing loss. Some causes include exposure to loud noises, physical head injuries, and presbycusis. We propose using an autonomous speechreading algorithm to help the deaf or hard-of-hearing by translating visual lip movements in live-time into coherent sentences. We accomplish this by using a supervised ensemble deep learning model to classify lip movements into phonemes, then stitch phonemes back into words. Our dataset consists of images of segmented mouths that are each labeled with a phoneme.