Domain Specific Wav2vec 2.0 Fine-tuning For The SE&R 2022 Challenge

The performance of Automatic Speech Recognition systems (ASRs) has increased significantly with the development of modern neural network topologies and the use of massive amount of data to train the models [1]. Although the accuracy of recent models improved for high-resource languages, such as English, the development of ASR models in other languages is still a difficult task using the same technologies [2, 3]. In this scenario, Self-Supervised Learning (SSL), a method in which representations with semantic information are learned by using unlabelled data, emerged as an important advance, allowing the training of deeper models using less labelled data [4, 5]. In this line of work, this paper explores the use of the Wav2vec 2.0 [6], a framework for self-supervised learning of discrete representations from raw audio data. Wav2vec 2.0 (Figure 1) is inspired by previous works in unsupervised pre-training for speech recognition, that is, Wav2vec [7] and Vq-Wav2vec [4].