Researchers from the Laboratory for Open Systems and Networks (E5) at the Jožef Stefan Institute have published the paper "Adaptive dual-mode distillation for robust and communication-efficient federated learning under statistical and model heterogeneity" in IEEE Access.
The paper addresses practical challenges in federated learning, particularly data and model heterogeneity as well as high communication costs. The authors propose a unified confidence-aware distillation framework that leverages unlabeled public data to support robust knowledge aggregation. The system operates in three complementary modes, allowing for adaptive weighting of client contributions without requiring homogeneous model architectures.
Evaluation across multiple architectures and datasets shows that the proposed methods consistently outperform existing solutions. Specifically, in extremely heterogeneous conditions, the methods achieve significant improvements in accuracy and reductions in communication costs, with results indicating up to a 153% improvement in global accuracy and up to a 225% client-level improvement after a single round of distillation.