Communication-Efficient Federated Self-Distillation Method for Medical Image Segmentation
College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
| Abstract: | Federated learning is a distributed learning framework designed to facilitate collaborative training of deep learning models at the network edge, ensuring the privacy and security of users while effectively responding to the challenges posed by data silos within the healthcare domain. However, deploying intricate medical segmentation network models in realistic settings often leads to substantial communication overhead for federated learning clients, thereby severely undermining the operational efficiency of federated systems. In light of these issues, we propose an innovative federated learning approach known as FedSKD, which is intended to alleviate the significant communication overhead incurred during medical image segmentation tasks in federated learning. This method strikes a delicate balance between communication costs and model training accuracy. A key feature of the FedSKD framework is incorporating a self-knowledge distillation mechanism, which encompasses both prediction map distillation loss and feature map distillation loss. This integration plays a key role in enhancing model accuracy without the need for training an additional resource-intensive mentor model, thereby optimizing computational efficiency. Additionally, we introduce parameter difference dynamic compression technology to compress communications, resulting in a reduction of communication costs by more than 30% within the federated system. Furthermore, extensive experiments utilizing FedSKD were conducted on widely recognized public medical segmentation datasets such as KiTS19 and COVID-19. The experimental findings unequivocally demonstrate that our FedSKD not only significantly diminishes the expenses associated with federated communication but also enhances overall model accuracy. |
| Keywords: | Federated Learning; Medical Image Segmentation; Self-Knowledge Distillation; Dynamic Parameter Difference Compression |
| DOI: | 10.57237/j.cst.2024.03.001 |
View:
Downloads:
Received: 29 May 2024
Accepted: 12 July 2024
Published Online: 18 July 2024
Funding
| 1. | National Natural Science Foundation of China (No. 61902046) |
| 2. | Science and Technology Research Program of Chongqing Municipal Education Commission (No. KJZD-K202200606) |
| 3. | Natural Science Foundation of Chongqing (No. CSTB2022NSCQ-MSX0277) |
References
| [1] | Singh A, Sengupta S, Lakshminarayanan V. Explainable deep learning models in medical image analysis [J]. Journal of imaging, 2020, 6(6): 52. |
| [2] | Liu S, Liu C, Ji Y, et al. Regional consistency for semi-supervised segmentation of 3D medical images [J]. IEEE Signal Processing Letters, 2023. |
| [3] | Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation [C] // Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. Springer International Publishing, 2015: 234-241. |
| [4] | Zhang Y, Xi R, Zeng L, et al. Structural Priors Guided Network for the Corneal Endothelial Cell Segmentation [J]. IEEE Transactions on Medical Imaging, 2023. |
| [5] | Zhang X, Liu A, Jiang P, et al. MSAIF-Net: A Multi-Stage Spatial Attention based Invertible Fusion Network for MR Images [J]. IEEE Transactions on Instrumentation and Measurement, 2023. |
| [6] | Langer S G. Cyber-security issues in healthcare information technology [J]. Journal of digital imaging, 2017, 30: 117-125. |
| [7] | Xu J, Glicksberg B S, Su C, et al. Federated learning for healthcare informatics [J]. Journal of healthcare informatics research, 2021, 5: 1-19. |
| [8] | McMahan B, Moore E, Ramage D, et al. Communication-efficient learning of deep networks from decentralized data [C] // Artificial intelligence and statistics. PMLR, 2017: 1273-1282. |
| [9] | Li T, Sahu A K, Zaheer M, et al. Federated optimization in heterogeneous networks [J]. Proceedings of Machine learning and systems, 2020, 2: 429-450. |
| [10] | Karimireddy S P, Kale S, Mohri M, et al. Scaffold: Stochastic controlled averaging for federated learning [C] // International conference on machine learning. PMLR, 2020: 5132-5143. |
| [11] | Ribero M, Vikalo H. Communication-efficient federated learning via optimal client sampling [J]. arxiv preprint arxiv:2007.15197, 2020. |
| [12] | Rothchild D, Panda A, Ullah E, et al. Fetchsgd: Communication-efficient federated learning with sketching [C] // International Conference on Machine Learning. PMLR, 2020: 8253-8265. |
| [13] | Bernstein J, Wang Y X, Azizzadenesheli K, et al. signSGD: Compressed optimisation for non-convex problems [C] // International Conference on Machine Learning. PMLR, 2018: 560-569. |
| [14] | Konecný J, McMahan H B, Yu F X, et al. Federated learning: Strategies for improving communication efficiency [J]. arxiv preprint arxiv: 1610.05492, 2016, 8. |
| [15] | Liu P, Sun M, Zhou S K. Multi-site organ segmentation with federated partial supervision and site adaptation [J]. arxiv preprint arxiv: 2302.03911, 2023. |
| [16] | Abouei J, Seyedmohammadi S J, Sheikholeslami S M, et al. MoFLeuR: Motion-based Federated Learning Gesture Recognition [J]. Authorea Preprints, 2023. |
| [17] | Fraboni Y, Vidal R, Kameni L, et al. Clustered sampling: Low-variance and improved representativity for clients selection in federated learning [C] // International Conference on Machine Learning. PMLR, 2021: 3407-3416. |
| [18] | Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network [J]. arxiv preprint arxiv:1503.02531, 2015. |
| [19] | Wu C, Wu F, Lyu L, et al. Communication-efficient federated learning via knowledge distillation [J]. Nature communications, 2022, 13(1): 2032. |
| [20] | Li K, Yu L, Wang S, et al. Towards cross-modality medical image segmentation with online mutual knowledge distillation [C] // Proceedings of the AAAI conference on artificial intelligence. 2020, 34(01): 775-783. |
| [21] | Zhu Z, Hong J, Zhou J. Data-free knowledge distillation for heterogeneous federated learning [C] // International conference on machine learning. PMLR, 2021: 12878-12889. |
| [22] | Kim K, Ji B M, Yoon D, et al. Self-knowledge distillation with progressive refinement of targets [C] // Proceedings of the IEEE/CVF international conference on computer vision. 2021: 6567-6576. |
| [23] | Ji M, Shin S, Hwang S, et al. Refine myself by teaching myself: Feature refinement via self-knowledge distillation [C] // Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 10664-10673. |
| [24] | Yang S, Zheng X, Xu Z, et al. A Lightweight Approach for Network Intrusion Detection based on Self-Knowledge Distillation [C] // ICC 2023-IEEE International Conference on Communications. IEEE, 2023: 3000-3005. |
Join the Editorial Board
We invite active, qualified and high profile scientists and researchers to join as Editorial Board Members.
Join UsBecome a Reviewer
Scholars with a strong interest in reviewing are invited to join the reviewer panel to ensure the quality of the research to be published.
Join Us