Detection of deepfake audio and image manuplation with self-supervised learning approach

Deepfake technology has become a rapidly spreading threat through audio and video manipulations and poses serious security risks. Detection of such fake content is of critical importance in terms of preserving media credibility and preventing the negative effects of fake content on society. In this thesis, it is aimed to detect deepfake video content using two different deep learning-based approaches. In the first approach, a self-supervised method is applied. In this method, pre-training is performed to estimate the rotation angle of the images with a ResNet18-based RotationNet model using the "deepfake-and-real-images" dataset, and then the feature extractor layers of this model are fixed and fine-tuned for real/fake classification. In the second approach, a multi-modality deep learning approach is proposed. In the study, videos taken from the widely used DeepFake Detection Challenge dataset are used. In the proposed method, an MTCNN-based face detection and alignment method is used to obtain face images from video streams. These face images are then given as input to the CNN and LSTM-based model. Mel spectrograms extracted from audio streams were analyzed with a CNN-based model. An audiovisual fusion model was developed that processes features derived from both modalities and their combination. Predictions from the image, audio, and fusion models were combined with learnable weights to create an ensemble model, and the final decision was made based on this model. As a result of the experimental studies, it was observed that the developed self-supervised approach detected deepfake videos with 88.89% accuracy, 83.15% recall and 88.10% F1 score. It was observed that the multimodal model could successfully detect 92.17% accuracy, 90.36% recall and 92.02% F1 score. These studies aim to contribute to the studies in this field by revealing the effectiveness of the methods used in deepfake detection.