Yongjin Shin, Gihun Lee, Seungjae Shin, Se-Young Yun, and Il-Chul Moon, FEWER:Federated Weight Recovery, Workshop on DistributedML, CoNEXT Conference (CoNEXT Workshop 2020), Virtual Conference, Dec 1, 2020

Abstract

In federated learning, the local devices train the model with their local data, independently; and the server gathers the locally trained model to aggregate them into a shared global model. Therefore, federated learning is an approach to decouple the model training from directly assessing the local data. However, the requirement of periodic communications on model parameters results in a primary bottleneck for the efficiency of federated learning. This work proposes a novel federated learning algorithm, Federated Weight Recovery(FEWER), which enables a sparsely pruned model in the training phase. FEWER starts with the initial model training with an extremely sparse state, and FEWER gradually grows the model capacity until the model reaches a dense model at the end of the training. The level of sparsity becomes the leverage to either increasing the accuracy or decreasing the communication cost, and this sparsification can be beneficial to practitioners. Our experimental results show that FEWER achieves higher test accuracies with less communication costs for most of the test cases. 

@inproceedings{10.1145/3426745.3431335, 

author = {Shin, Yongjin and Lee, Gihun and Shin, Seungjae and Yun, Se-young and Moon, Il-Chul}, 

title = {FEWER: Federated Weight Recovery}, 

year = {2020}, 

isbn = {9781450381826}, 

publisher = {Association for Computing Machinery}, 

address = {New York, NY, USA}, 

url = {https://doi.org/10.1145/3426745.3431335}, 

doi = {10.1145/3426745.3431335}, 

booktitle = {Proceedings of the 1st Workshop on Distributed Machine Learning}, 

pages = {1–6}, 

numpages = {6}, 

location = {Barcelona, Spain}, 

series = {DistributedML'20} 

} 

Source Website:

https://dl.acm.org/doi/pdf/10.1145/3426745.3431335