Stay Rational, Resist Illusions: De-biasing Source-Free Domain Adaptation with CLIP

Abstract Source-free domain adaptation (SFDA) aims to adapt a source model pre-trained on a labeled source domain to an unlabeled target domain, under the constraint that the original source data is no longer accessible. While some existing approaches achieve promising adaptation performance by approximating certain target samples as source-like samples to construct a pseudo-source domain and reduce domain discrepancies, they typically rely solely on the source model for this construction. This overlooks a critical issue: the source model may suffer from illusions when identifying pseudo-source samples, due to its inherited spurious correlations between domain-specific features (e.g., environment) and domain-invariant features (e.g., class identity) learned from the source domain. As a result, the reliability of the generated pseudo-source domain is compromised. In this work, we propose a novel Staying Rational and Resisting Illusions (SRRI) method for SFDA to resolve the two fundamental challenges: confirmation bias and domain shift. In particular, we innovatively leverage the CLIP (Contrastive Language-Image Pre-training) model as a source of external knowledge to mitigate feature confounding. SRRI employs knowledge distillation to guide the source model in disentangling class-discriminative causal features from domain-specific spurious features, building upon this, we design a CLIP-guided dual-model validation and class balancing strategy to ensure the reliability and richness of the pseudo-source domain samples even when poor distillation occurs in some tasks. Furthermore, we propose a dynamic pseudo-source domain optimization mechanism that continuously fine-tunes the task-specific prompts of the CLIP during adaptation to further correct the bias of the target model on hard samples, while periodically reconstructing and refining the pseudo-source domain. Ultimately, we apply robust supervised learning and distribution alignment to these respective domains. Extensive experiments show that SRRI outperforms the state-of-the-art methods. Data Availability All the datasets are publicly available. References Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. Proceedings of the 38th international conference on machine learning (Vol. 139, pp. 8748–8763). PMLR. Litrico, M., Del Bue, A., & Morerio, P. (2023). Guiding Pseudo-Labels with Uncertainty Estimation for Source-Free Unsupervised Domain Adaptation. Proceedings of the ieee/cvf conference on computer vision and pattern recognition (cvpr) (pp. 19904–19914). Venkateswara, H., Eusebio, J., Chakraborty, S., & Panchanathan, S. (2017). Deep hashing network for unsupervised domain adaptation. Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) (pp. 5018–5027). IEEE. Cui, C., Meng, F., Zhang, C., Liu, Z., Zhu, L., Gong, S., & Lin, X. (2023). Adversarial source generation for source-free domain adaptation. IEEE Transactions on Circuits and Systems for Video Technology, 34(6), 4887–4898. Chen, D., Wang, D., Darrell, T., & Ebrahimi, S. (2022). Contrastive Test-Time Adaptation. Proceedings of the ieee/cvf conference on computer vision and pattern recognition (cvpr) (pp. 11110–11120). Du, Y., Yang, H., Chen, M., Luo, H., Jiang, J., Xin, Y., & Wang, C. (2024). Generation, augmentation, and alignment: A pseudo-source domain based method for source-free domain adaptation. Machine Learning, 113(6), 3611–3631. https://doi.org/10.1007/s10994-024-06540-3 Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) (pg (pp. 10984–10995) Saenko, K., Kulis, B., Fritz, M., & Darrell, T. (2010). Adapting visual category models to new domains. Proceedings of the 11th european conference on computer vision (ECCV) (pp. 213–226). Springer Ge, C., Huang, R., Xie, M., Lai, Z., Song, S., Li, S., & Huang, G. (2023). Domain Adaptation via Prompt Learning. IEEE Transactions on Neural Networks and Learning Systems. https://doi.org/10.1109/TNNLS.2023.3323812 Wang, F., Han, Z., Gong, Y., & Yin, Y. (2022). Exploring domain-invariant parameters for source free domain adaptation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) (pp. 16263–16272). Gou, J., Yu, B., Maybank, S. J., & Tao, D. (2021). Knowledge distillation: A survey. International Journal of Computer Vision, 129(6), 1789–1819. https://doi.org/10.1007/s11263-021-01453-z Gulrajani, I., & Lopez-Paz, D. (2020). In search of lost domain generalization. arXiv. Qu, S., Chen, G., Zhang, J., Li, Z., He, W., & Tao, D. (2022). BMD: A general class-balanced multicentric dynamic prototype strategy for source-free domain adaptation. arXiv. He, J., Wu, L., Tao, C., & Lv, F. (2024). Source-free domain adaptation with unrestricted source hypothesis. Pattern Recognition. https://doi.org/10.1016/j.patcog.2024.110246 Yang, S., Wang, Y., van de Weijer, J., Herranz, L., & Jui, S. (2021). Exploiting the intrinsic neighborhood structure for source-free domain adaptation. Advances in neural information processing systems 34 (NeurIPS 2021) (pp. 14194–14206). Liang, J., Hu, D., & Feng, J. (2020). Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. International conference on machine learning (pp. 6028–6039). Cui, S., Wang, S., Zhuo, J., Li, L., Huang, Q., & Tian, Q. (2020). Towards discriminability and diversity: Batch nuclear-norm maximization under label insufficient situations. Proceedings of the ieee/cvf conference on computer vision and pattern recognition (pp. 3941–3950). Ishii, M., & Sugiyama, M. (2021). Source-free domain adaptation via distributional alignment by matching batch normalization statistics. arXiv. Lao, Q., Jiang, X., & Havaei, M. (2021). Hypothesis disparity regularized mutual information maximization. Proceedings of the thirty-fifth AAAI conference on artificial intelligence (Vol. 35, pp. 8433–8441). AAAI Press. Yang, S., Wang, Y., Wang, K., Jui, S., & van de Weijer, J. (2022a). Attracting and dispersing: A simple approach for source-free domain adaptation. Advances in neural information processing systems 35 (NeurIPS 2022) (pp. 28247–28259) Kim, Y., Cho, D., Han, K., Panda, P., & Hong, S. (2021). Domain adaptation without source data. IEEE Transactions on Artificial Intelligence, 2(6), 508–518. Tang, S., Shi, Y., Ma, Z., Li, J., Lyu, J., Li, Q., & Zhang, J. (2021). Model adaptation through hypothesis transfer with gradual knowledge distillation. Proceedings of the ieee/rsj international conference on intelligent robots and systems (iros) (pp. 5679–5685). Li, J., Du, Z., Zhu, L., Ding, Z., Lu, K., & Shen, H. T. (2022a). Divergence-Agnostic Unsupervised Domain Adaptation by Adversarial Attacks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11), 8196–8211. https://doi.org/10.1109/TPAMI.2021.3101569 Li, R., Pu, Y., Li, Z., Wu, C., Xie, H., & Lian, D. (2025). Invariant representation learning via decoupling style and spurious features. Machine Learning, 114(2), 37. Li, S., Lv, F., Xie, B., Liu, C. H., Liang, J., & Qin, C. (2021). Bi-classifier determinacy maximization for unsupervised domain adaptation. Proceedings of the aaai conference on artificial intelligence (Vol. 35, pp. 8455–8464). Liang, J., Hu, D., Wang, Y., He, R., & Feng, J. (2022). Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11), 8602–8617. https://doi.org/10.1109/TPAMI.2021.3101834 Yang, T., Zhou, S., Wang, Y., Lu, Y., & Zheng, N. (2022b). Test-time batch normalization. arXiv. Ma, Q., Zeng, J., Zhang, J., Zu, C., Wu, X., Zhou, J., & Wang, Y. (2024). Source-free domain adaptation via dynamic pseudo labeling and self-supervision. Pattern Recognition. https://doi.org/10.1016/j.patcog.2024.110793 Arazo, E., Ortego, D., Albert, P., O’Connor, N. E., & McGuinness, K. (2020). Pseudo-labeling and confirmation bias in deep semi-supervised learning. International joint conference on neural networks (IJCNN) (pp. 1–8). IEEE. Zhang, B., Wang, Y., Hou, W., Wu, H., Wang, J., Okumura, M., & Shinozaki, T. (2021a). Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in neural information processing systems 34 (NeurIPS 2021) (Vol. 34, pp. 18408–18419). Pan, S. J., & Yang, Q. (2010). A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10), 1345–1359. Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-CAM: Visual explanations from deep networks via gradient-based localization. Proceedings of the ieee international conference on computer vision (iccv) (pp. 618–626). Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., & Duerig, T. (2021). Scaling up visual and vision-language representation learning with noisy text supervision. In M. Meila, & T. Zhang (Eds.), Proceedings of the 38th international conference on machine learning (Vol. 139, pp. 4904–4916). PMLR. Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., & Sutskever, I. (2021). Zero-shot text-to-image generation. Proceedings of the 38th international conference on machine learning (Vol. 139, pp. 8821–8831). PMLR. Roy, S., Trapp, M., Pilzer, A., Kannala, J., Sebe, N., Ricci, E., & Solin, A. (2022). Uncertainty-Guided Source-Free Domain Adaptation. Proceedings of the european conference on computer vision (eccv) (pg (pp. 537–555). Springer. Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., & Kiela, D. (2022). FLAVA: A foundational language and vision alignment model. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) (pp. 15638–15650). Peng, X., Bai, Q., Xia, X., Huang, Z., Saenko, K., & Wang, B. (2019). Moment matching for multi-source domain adaptation. Proceedings of the IEEE/CVF international conference on computer vision (ICCV) (pp. 1406–1415). IEEE. Tarashima, S., Shu, X., & Tagawa, N. (2025). Vilaad: Enhancing" attracting and dispersing” source-free domain adaptation with vision-and-language model. arXiv preprint arXiv:2503.23529 Liang, J., He, R., Sun, Z., & Tan, T. (2019). Distant supervised centroid shift: A simple and efficient approach to visual domain adaptation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) (pp. 2988–2997). Yue, Z., Sun, Q., & Zhang, H. (2023). Make the U in UDA matter: Invariant consistency learning for unsupervised domain adaptation. Advances in neural information processing systems 36 (NeurIPS 2023) (Vol. 36, pp. 69372–69389). Tang, S., Zou, Y., Song, Z., Lyu, J., Chen, L., Ye, M., & Zhang, J. (2022). Semantic consistency learning on manifold for source data-free unsupervised domain adaptation. Neural Networks, 152, 114–126. https://doi.org/10.1016/j.neunet.2022.04.020 Tang, S., Chang, A., Zhang, F., Zhu, X., Ye, M., & Zhang, C. (2024a). Source Free Domain Adaptation via Target Prediction Distribution Searching. International Journal of Computer Vision. https://doi.org/10.1007/s11263-024-02035-7 Tian, J., Zhang, J., Li, W., & Xu, D. (2022). VDM-DA: Virtual domain modeling for source data-free domain adaptation. IEEE Transactions on Circuits and Systems for Video Technology, 32(6), 3749–3760. https://doi.org/10.1109/TCSVT.2021.3138865 Tian, Q., Sun, H., Peng, S., Zheng, Y., Wan, J., & Lei, Z. (2023). Dcl: Dipolar confidence learning for source-free unsupervised domain adaptation. IEEE Transactions on Circuits and Systems for Video Technology, 34(6), 4342–4353. Saito, K., Watanabe, K., Ushiku, Y., & Harada, T. (2018). Maximum Classifier Discrepancy for Unsupervised Domain Adaptation. Proceedings of the ieee/cvf conference on computer vision and pattern recognition (pp. 3723–3732). Chhabra, S., Venkateswara, H., & Li, B. (2023). Generative alignment of posterior probabilities for source-free domain adaptation. Proceedings of the ieee/cvf winter conference on applications of computer vision (wacv) (pp. 4125–4134). Kumar, A., Sattigeri, P., Wadhawan, K., Karlinsky, L., Feris, R., Freeman, W. T., & Wornell, G. W. (2018). Co-regularized alignment for unsupervised domain adaptation. Advances in neural information processing systems 31 (NeurIPS 2018) (Vol. 31, pp. 7546–7556). Long, M., Cao, Z., Wang, J., & Jordan, M. I. (2018). Conditional adversarial domain adaptation. Advances in neural information processing systems 31 (NeurIPS 2018) (Vol. 31, pp. 1640–1650). Peng, X., Usman, B., Kaushik, N., Hoffman, J., Wang, D., & Saenko, K. (2017). VisDA: The visual domain adaptation challenge. arXiv. Li, Y., Cao, Y., Li, J., Wang, Q., & Wang, S. (2024). Data-efficient CLIP-powered dual-branch networks for source-free unsupervised domain adaptation. arXiv. Huang, J., Guan, D., Xiao, A., & Lu, S. (2021). Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data. Advances in neural information processing systems 34 (NeurIPS 2021) (pp. 27693–27705). Xu, B., Zhang, L., Mao, Z., Wang, Q., Xie, H., & Zhang, Y. (2020). Curriculum learning for natural language understanding. Proceedings of the 58th annual meeting of the association for computational linguistics (pp. 6095–6104). Association for Computational Linguistics. Zhang, Y., Chen, H., Chen, X., Deng, Y., Xu, C., & Wang, Y. (2021b). Data-free knowledge distillation for image super-resolution. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) (pp. 7852–7861). Xu, X., Qi, L., Zhou, J., & Geng, X. (2025). Batstyler: Advancing multi-category style generation for source-free domain generalization. IEEE Transactions on Circuits and Systems for Video Technology. https://doi.org/10.1109/tcsvt.2025.3525715 Xu, R., Li, G., Yang, J., & Lin, L. (2019). Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. Proceedings of the IEEE/CVF international conference on computer vision (ICCV) (pp. 1426–1435). IEEE. Tang, S., Su, W., Ye, M., & Zhu, X. (2024b). Source-free domain adaptation with frozen multimodal foundation model. Proceedings of the ieee/cvf conference on computer vision and pattern recognition (pp. 23711–23720). Lee, J., Jung, D., Yim, J., & Yoon, S. (2022). Confidence Score for Source-Free Unsupervised Domain Adaptation. Proceedings of the international conference on machine learning (icml) (Vol. 162, pp. 12140–12157). PMLR. Yuan, J., Zheng, H., Yu, H., & Luo, X. (2025). Entangle-then-disentangle: A novel approach for enhancing large vision-language model. Machine Learning, 114(8), 171. Zhang, W., Shen, L., & Foo, C.-S. (2025). Source-free domain adaptation guided by vision and vision-language pre-training. International Journal of Computer Vision, 133(2), 844–866. Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022). Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9), 2337–2348. Zhou, L., Li, N., Ye, M., Zhu, X., & Tang, S. (2024). Source-free domain adaptation with class prototype discovery. Pattern Recognition. https://doi.org/10.1016/j.patcog.2023.109974 Li, X., Du, Z., Li, J., Zhu, L., & Lu, K. (2022b). Source-free active domain adaptation via energy-based locality preserving transfer. Proceedings of the 30th ACM international conference on multimedia (pp. 5802–5810). Association for Computing Machinery (ACM). Author information Authors and Affiliations Contributions Qing Tian: Supervision; Writing - review & editing. Yongjiang Liu: Conceptualization; Methodology; Writing - original draft. Keyang Cheng, Weihua Ou, Jianping Gou: Writing - review & editing. Corresponding author Ethics declarations Conflict of interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Tianqing reports financial support was provided by the National Natural Science Foundation of China under Grant 62676199 and 62176128. Tianqing reports financial support was provided by the Basic Research Program of Jiangsu under Grant BK20231143. Tianqing reports financial support was provided by the Open Project of Key Laboratory of Tibetan Information Processing of the Ministry of Education under Grant QHSF-CS-2609. Tianqing reports financial support was provided by the Fundamental Research Funds for the Central Universities No. NJ2023032. Tianqing reports financial support was provided by the 333 High-Level Talent Project of Jiangsu Province. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Additional information Editor: Bo Han. Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Supplementary Information Below is the link to the electronic supplementary material. Rights and permissions Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law. About this article Cite this article Tian, Q., Liu, Y., Cheng, K. et al. Stay Rational, Resist Illusions: De-biasing Source-Free Domain Adaptation with CLIP-Verified Pseudo-Source Domain. Mach Learn 115, 222 (2026). https://doi.org/10.1007/s10994-026-07163-2 Received: Revised: Accepted: Published: Version of record: DOI: https://doi.org/10.1007/s10994-026-07163-2