Traditional unsupervised multi-source domain adaptation (UMDA) assumes that all source domain data is directly accessible. However, privacy preservation requires that all data and computation be performed locally, which poses three challenges for domain adaptation approaches: (1) Minimizing the domain distance requires the pairwise calculation of the data from source and target domains, which is not accessible; (2) The communication cost and privacy security limit the application of UMDA methods (e.g., the domain adversarial training); (3) Since users have no authority to check the data quality, the irrelevant or malicious source domains are more likely to appear, which causes negative transfer.
To address the above issues, a research group led by Dr Fei Wu and Dr Chao Wu proposed a decentralized, unsupervised domain adaptation paradigm that satisfies privacy-preserving requirements. This paradigm is called Knowledge Distillation based Decentralized Domain Adaptation KD3A (KD3A) and performs knowledge distillation of models from multiple source domains to create migratable consensus knowledge.
First, they propose a multi-source knowledge extraction method, Knowledge Vote, to obtain high-quality cross-domain consensus knowledge. Second, they develop a dynamic weighting strategy Consensus Focus to identify malicious and irrelevant source domains according to the quality of consensus knowledge of different domains. Finally, they present a decentralized optimization strategy for H-divergence, BatchNorm MMD and then they analyze the decentralized generalization bound of KD3A from a theoretical perspective. The algorithm is reflected below:
Numerous experiments on DomainNet show that KD3A significantly outperforms other domain adaptation methods like UMDA. Moreover, KD3A is robust to negative migration and can reduce the communication cost by a factor of 100X compared to other decentralized UMDA methods.
This work is accepted by ICML2021 and could be accessed at [2011.09757v4] KD3A: Unsupervised Multi-Source Decentralized Domain Adaptation via Knowledge Distillation (arxiv.org).
About Dr Chao Wu
Chao Wu is a Associated Professor at School of Public Affairs, Zhejiang University. He is now the director of Computational Social Science Research Center (ZJUCCS). He is also an Honorary Research Fellow of Data Science Institute and Department of Computing, Imperial College London, and the Director of Big Data Research Center - ZJPTCC. His main research interests are about distributed machine learning, privacy protection, and computable social analytics. He is also the creator of Mo, which is web-based platform for both AI modelling and education.
About Professor Fei Wu
Fei Wu received his B.Sc., M.Sc. and Ph.D. degrees in computer science from Lanzhou University, University of Macau and Zhejiang University in 1996, 1999 and 2002 respectively. From October, 2009 to August 2010, Fei Wu was a visiting scholar at Prof. Bin Yu's group, University of California, Berkeley. Currently, He is a Qiushi distinguished professor of Zhejiang University at the college of computer science. He is the vice-dean of college of computer science, and the director of Institute of Artificial Intelligence of Zhejiang University.
He is currently the Associate Editor of Multimedia System, the editorial members of Frontiers of Information Technology & Electronic Engineering. He has won various honors such as the Award of National Science Fund for Distinguished Young Scholars of China (2016).
His research interests mainly include Artificial Intelligence, Multimedia Analysis and Retrieval and Machine Learning.
About SIAS
Shanghai Institute for Advanced Study of Zhejiang University (SIAS) is a jointly launched new institution of research and development by Shanghai Municipal Government and Zhejiang University in June, 2020. The platform represents an intersection of technology and economic development, serving as a market leading trail blazer to cultivate a novel community for innovation amongst enterprises.
SIAS is seeking top talents working on the frontiers of computational sciences who can envision and actualize a research program that will bring out new solutions to areas include, but not limited to, Artificial Intelligence, Computational Biology, Computational Engineering and Fintech.