Tex-ViT: A Generalizable, Robust, Texture-based dual-branch cross-attention deepfake detector

Deepfakes, which employ Generative Adversarial Networks (GANs) to produce highly realistic facial modification, are widely regarded as the prevailing method. Traditional Convolutional Neural Networks (CNNs) have been able to identify bogus media, but they struggle to perform well on different datasets and are vulnerable to adversarial attacks due to their lack of robustness. Vision transformers have demonstrated potential in the realm of image classification problems, but they require enough training data. Motivated by these limitations, this publication introduces Tex-ViT (Texture-Vision Transformer), which enhances CNN features by combining ResNet (Residual Networks) with a vision transformer. The model combines traditional ResNet features with a texture module that operates in parallel on sections of ResNet before each down-sampling operation. The texture module then serves as an input to the dual branch of the cross-attention vision transformer. It specifically focuses on improving the global texture module, which extracts feature map correlation. Empirical analysis reveals that fake images exhibit smooth textures that do not remain consistent over long distances in manipulations. Experiments were performed on different categories of FaceForensics++ (FF++), such as Deepfakes (DF), Face2Face (f2f), Faceswap (FS), and Neural Texture (NT), together with other types of GAN datasets in cross-domain scenarios. Furthermore, experiments also conducted on FF++, DFDCPreview, and Celeb-DF dataset underwent several post-processing situations, such as blurring, compression, and noise. The model surpassed the most advanced models in terms of generalization, achieving a 98% accuracy in cross-domain scenarios. This demonstrates its ability to learn the shared distinguishing textural characteristics in the manipulated samples. These experiments provide evidence that the proposed model is capable of being applied to various situations and is resistant to many postprocessing procedures. Keywords: Deepfake detector, Texture, Gram matrices, Generalization, Robustness. 1 Introduction With the advancements in technology, especially GANs, it is possible to generate highly realistic content that can easily deceive the naked eye. Deepfake is a current state of the art of visual and audio manipulation. Deepfake is a technology where highly astonishing, realistic, and believable content is created using deep learning technology (Figure 1). Visual deepfakes can be classified into five categories: lip sync, attribute manipulation, full-image synthesis, body re-enactment, and face ap [1]. The application of the deepfake has benefitted the education and entertainment industry in various ways.