The Generalisability Gap - Evaluating Deepfake Detectors Across Domains

The rise of deepfakes has raised concerns about the potential misuse of this technology for disinformation, fraud, and other malicious purposes. To counter this threat, researchers have developed deepfake detectors that can identify deepfakes with widely published claims of 99%+ accuracy. Yet, this space evolves extremely quickly, with new and more realistic algorithms being released on a weekly basis.

To keep up with the evolving threat landscape, detectors need to be generalisable across different datasets and deepfake generation methods. With the release of a new deepfake dataset - DeepAction (Bohacek & Farid, 2024) - we took the opportunity to assess the generalisability of open-source deepfake detectors across the dataset and deepfake generation methods.

While much of our work involves evaluating closed-source models, there's much to be learned from open-source models and we hope that the lessons learned from our assessment will be useful to the wider community evaluating deepfake detectors and those building generalisable models.

1. When fine-tuned on new data, deepfake detectors can perform well even outside their original training domain (faces) - achieving 95%+ AUC scores on action videos. This suggests the underlying architectures are capable of adapting to new types of deepfakes when properly trained.
2. However, pre-trained models perform poorly when tested on new types of deepfakes without fine-tuning - even for models designed to be more generalisable and pre-trained on large datasets, AUC scores drop to 67-71%. This highlights the challenge of building truly generalisable detectors.
3. Generalisability exists on a spectrum - while newer architectures like UCF and CLIP show promise, their effectiveness must be validated on unseen datasets that mirror real-world deployment conditions.
4. Feature separation in the latent space does not automatically translate to strong classification performance. However, when a model learns meaningful representations, adapting it to new domains can be as simple as retraining the final classification layer - as demonstrated by CLIP's rapid improvement from 71% to 94% AUC by freezing the underlying parameters and training a new classifier.

The DeepAction Dataset, is a comprehensive collection of human action videos consisting of 3,100 AI-generated clips from seven text-to-video models (BD AnimateDif (Lin & Yang, 2024), CogVideoX-5B (Yang et al., 2024), RunwayML Gen3 (RunwayML, n.d.), Stable Diffusion (Blattmann et al., 2023), Veo (Veo, n.d.), and VideoPoet (Kondratyuk et al., 2024) and 100 matching real videos from Pexels.

We evaluated three open-source deepfake detectors, each representing different approaches to detection:

1. Xception (Rössler et al., 2019) serves as our baseline candidate. Built on the XceptionNet architecture (Chollet, 2017), it's widely adopted in deepfake detection research. We use the pre-trained model from Yan et al. (2024), fine-tuned on the FaceForensics++ (FF++) dataset.
   
2. UCF (Yan, Zhang, Fan, et al., 2023) represents recent advances in generalisable detection. Released in 2023, it employs contrastive regularization to learn both common and specific forgery features. Our evaluation uses the pre-trained model from Yan, Zhang, Yuan, et al. (2023), fine-tuned on FF++.
   
3. CLIP (Radford et al., 2021) demonstrates the potential of large-scale training. Though designed for general visual-linguistic tasks, its rich representations make it effective for deepfake detection. Recent benchmarks in Yan et al. (2024) show it outperforming other state-of-the-art detectors. On the DeepAction dataset, Bohacek & Farid (2024) achieved 85% frame accuracy and 97% video accuracy using just an SVM trained on CLIP embeddings. We use the ViT-B/16 variant from Yan et al. (2024), fine-tuned on FF++.

This selection spans traditional specialist models (Xception), modern generalization-focused approaches (UCF), and large-scale representation learning (CLIP). All three models were pre-trained on FF++ to enable fair comparison on the DeepAction dataset.

To evaluate how well these algorithms would be able to adapt from the facial recognition domain to action videos, we fine-tuned them on the DeepAction data before evaluating them on the test set. The frame-level results (N = 10564) are summarised in Table 1 below.

Table 1: Performance of deepfake detectors trained and tested on the DeepAction dataset at the frame level

While surveillance of emerging deepfake threats and continuous improvement of detector models by fine-tuning on the latest techniques is effective, it is not always possible to know the exact nature of such threats in advance. To evaluate the generalisability of pre-trained models, we take our three models pre-trained on the FF++ dataset and evaluate them on DeepAction without further fine-tuning. The results are summarised in Table 3 below.

Table 3: Performance of deepfake detectors trained on FF++ at the frame level

While in a typical evaluation of a black box model, one would be left to speculate on why the models may not be as generalisable as one would hope, the benefit of using open-source models is that we can dive one level deeper into the feature embedding space to understand what might be going on.

Figure 2: Visualising feature embeddings

The Xception model reveals characteristics of a specialist architecture that excels with training data but struggles to generalize. Examining the t-SNE embeddings shows that before fine-tuning, the model fails to meaningfully separate real from fake videos - all data points appear randomly distributed. However, after fine-tuning on DeepAction, a clear separation emerges between real and fake content, suggesting the model can learn the task but requires explicit training examples.

Figure 3: Leave-one-out analysis by testing data subset

The UCF model shows some structure in its TSNE embeddings when trained on FF++, though without a clear separation between real and fake data points. This structure likely emerges from UCF's contrastive regularization technique, which explicitly separates content-specific from forgery-specific features (Figure 4).

Figure 4: UCF architecture (Yan, Zhang, Fan, et al., 2023) - separating an input content into content and forgery-specific features

The TSNE visualization of CLIP's embeddings presents an intriguing puzzle: despite showing clear separation between real and fake data points, the model achieves only marginally better classification performance than Xception or UCF (AUC 0.71 vs 0.67).

This disconnect highlights a fundamental insight about deep learning models: the ability to learn discriminative features (visible in the TSNE plot) does not automatically translate to strong classification performance. The classification head must still learn to correctly weigh and combine these features for the specific task at hand.

Fortunately, for our pre-trained model, this is a simple remedy. By freezing CLIP's pre-trained parameters and training only a new classification layer on the DeepAction dataset, we achieved:

• AUC score: 0.94 (up from 0.71)
• Accuracy: 0.85 (up from 0.62)

This dramatic improvement mirrors Bohacek & Farid (2024) findings with their SVM approach (0.84 accuracy) and suggests that while CLIP learns robust, transferable features, adapting these features to new domains may require targeted fine-tuning of the classification layer.

Our evaluation of three open-source deepfake detectors reveals both the potential and limitations of generalisable synthetic media detection.

When fine-tuned on new domains, deepfake detectors show remarkable adaptability - even models originally trained on facial deepfakes achieve 0.95+ AUC scores on synthetic action videos. However, without fine-tuning, performance degrades significantly as test data diverges from the training distribution, even for newer architectures like UCF and CLIP that were designed for better generalization.

Our analysis also reveals a critical insight: strong feature learning doesn't guarantee strong classification. While models can learn rich, discriminative features (evident in their latent space), translating these features into accurate predictions often requires targeted fine-tuning of the classification layer.

We have evaluated the generalisability of three open-source deepfake detectors across different datasets and deepfake generation methods, and found that fine-tuning the models on the DeepAction dataset significantly improved their performance and deepfake detectors. Even those trained on facial images, can easily adapt to other types of images.

However, generalising pre-trained models still remains a challenge. While newer architectures like UCF and CLIP show promise in being more generalisable, it is important to validate their performance on unseen datasets that hopefully mirrors the distribution of the data they are likely to be deployed on. Understanding this distinction between feature learning and classification performance provides a path forward: future detectors may benefit more from focusing on robust feature extraction and making it available rather than trying to build one-size-fits-all classifiers.

Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., Jampani, V., & Rombach, R. (2023). Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets. https://arxiv.org/abs/2311.15127

Bohacek, M., & Farid, H. (2024). Human Action CLIPS: Detecting AI-generated Human Motion. https://arxiv.org/abs/2412.00526

Chollet, F. (2017). Xception: Deep learning with depthwise separable convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1251–1258. http://openaccess.thecvf.com/content_cvpr_2017/html/Chollet_Xception_Deep_Learning_CVPR_2017_paper.html

Kondratyuk, D., Yu, L., Gu, X., Lezama, J., Huang, J., Schindler, G., Hornung, R., Birodkar, V., Yan, J., Chiu, M.-C., Somandepalli, K., Akbari, H., Alon, Y., Cheng, Y., Dillon, J., Gupta, A., Hahn, M., Hauth, A., Hendon, D., … Jiang, L. (2024). VideoPoet: A Large Language Model for Zero-Shot Video Generation. https://arxiv.org/abs/2312.14125

Li, Y., Yang, X., Sun, P., Qi, H., & Lyu, S. (2020). Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics. https://arxiv.org/abs/1909.12962

Lin, S., & Yang, X. (2024). AnimateDiff-Lightning: Cross-Model Diffusion Distillation. https://arxiv.org/abs/2403.12706

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning Transferable Visual Models From Natural Language Supervision. https://arxiv.org/abs/2103.00020

Rössler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Nießner, M. (2019). FaceForensics++: Learning to Detect Manipulated Facial Images. https://arxiv.org/abs/1901.08971

RunwayML. (n.d.). Introducing Gen-3 Alpha. https://runwayml.com/research/introducing-gen-3-alpha

Veo. (n.d.). https://deepmind.google/technologies/veo

Yan, Z., Yao, T., Chen, S., Zhao, Y., Fu, X., Zhu, J., Luo, D., Yuan, L., Wang, C., Ding, S., & others. (2024). DF40: Toward Next-Generation Deepfake Detection. arXiv Preprint arXiv:2406.13495.

Yan, Z., Zhang, Y., Fan, Y., & Wu, B. (2023). UCF: Uncovering Common Features for Generalizable Deepfake Detection. https://arxiv.org/abs/2304.13949

Yan, Z., Zhang, Y., Yuan, X., Lyu, S., & Wu, B. (2023). DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Advances in Neural Information Processing Systems (Vol. 36, pp. 4534–4565). Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2023/file/0e735e4b4f07de483cbe250130992726-Paper-Datasets_and_Benchmarks.pdf

Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y., Hong, W., Zhang, X., Feng, G., Yin, D., Gu, X., Zhang, Y., Wang, W., Cheng, Y., Liu, T., Xu, B., Dong, Y., & Tang, J. (2024). CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer. https://arxiv.org/abs/2408.06072

1. While this is similar to Bohacek & Farid (2024) train/test split, it also means that the number of real videos (19) evaluated in the test set is relatively limited and standard errors may be high. This is a limitation of the dataset and should be taken into account when interpreting the results.
2. Since none of our models consider temporal features, each frame is scored and evaluated independently. Results at the video level are then decided by the class with the highest proportion of frames (i.e. majority voting).