The purpose of this work is to follow the state-of-the-art works on Federated Learning
[TOC]
-
Federated Learning for Generalization,Robustness, Fairness: A Survey and Benchmark - ArXiv’23
They argued that Generalization, Robustness, and Fairness interact with each other to jointly enhance the practical federation deployment and this is the first work to simultaneously investigate the related research development and uniformly benchmark multi-view experimental analysis on the Generalization, Robustness, and Fairness realms.
-
Heterogeneous federated learning: State-of-the-art and research challenges - ACM Computing Surveys‘23
They summarized the various research challenges in HFL from five aspects: statistical heterogeneity, model heterogeneity, communication heterogeneity, device heterogeneity, and additional challenges. They classify existing methods from three different levels according to the HFL procedure: data-level, model-level, and server-level.
-
FedAvg: Communication-Efficient Learning of Deep Networks from Decentralized Data - PMLR’17
Google built a basic paradigm for federated learning.
Local SGD + Server average aggregation
- FedPox: Federated optimization in heterogeneous networks - MLSys’20
- SCAFFOLD: SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning - ICML'20
- MOON: Model-Contrastive Federated Learning - CVPR'21
- FedOPT: Adaptive federated optimization - ArXiv’21
- MFL: Accelerating Federated Learning via Momentum Gradient Descent - IEEE TPDS'20
- Mime: Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning - ArXiv’21
- Mimelite: Breaking the centralized barrier for cross-device federated learning - NeurIPS'21
- FedGBO: Accelerating Federated Learning With a Global Biased Optimiser - IEEE TC'22
- FedLocal: Local Adaptivity in Federated Learning: Convergence and Consistency - ArXiv’21
- FedDA: Accelerated Federated Learning with Decoupled Adaptive Optimization - PMLR'22
- FedUR: FedUR: Federated Learning Optimization Through Adaptive Centralized Learning Optimizers - IEEE TSP'23
They proposed federated distillation (FD) and Federated Augmentation, where each device collectively trains a generative model (GAN), and thereby augments its local data towards yielding an IID dataset.
2.FedHome: Cloud-Edge based Personalized Federated Learning for In-Home Health Monitoring - IEEE TMC’20
They proposed a novel cloud-edge based federated learning framework for in-home health monitoring, which learns a shared global model in the cloud from multiple homes at the network edges and achieves data privacy protection by keeping user data locally. They designed a generative convolutional autoencoder (GCAE) to achieve accurate and personalized health monitoring by refining the model with a generated class-balanced dataset from user's personal data.
...
1.Optimizing Federated Learning on Non-IID Data with Reinforcement Learning - IEEE INFORCOM’20 code
They proposed FAVOR an experience-driven control framework that intelligently chooses the client devices to participate in each round of federated learning to counterbalance the bias introduced by non-IID data and to speed up convergence. They propose a mechanism based on deep Q-learning that learns to select a subset of devices in each communication round to maximize a reward that encourages the increase of validation accuracy and penalizes the use of more communication rounds.
2.Multi-Armed-Bandit-Based-Client-Scheduling-for-Federated-Learning - IEEE TWC'20 Cited by 188 code
3.Federated Learning with Class Imbalance Reduction - EUSIPCO'21
They designed an estimation scheme to reveal the class distribution without the awareness of raw data. They proposed a multi-arm bandit based algorithm that can select the client set with minimal class imbalance.
4.TiFL: A Tier-based Federated Learning System - HPDC'20
They proposed a Tier-based Federated Learning system, which divides clients into tiers based on their training performance and selects clients from the same tier in each training round to mitigate the straggler problem caused by heterogeneity in resource and data quantity. TiFL employs an adaptive tier selection approach to update the tiering on-the-fly based on the observed training performance and accuracy.
5.FedSAE: A Novel Self-Adaptive Federated Learning Framework in Heterogeneous Systems - IJCNN'21 code
They proposed FedSAE which leverages the complete information of devices' historical training tasks to predict the affordable training workloads for each device. In this way, FedSAE can estimate the reliability of each device and self-adaptively adjust the amount of training load per client in each round.
6.HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients - ICLR‘2021 code
They proposed an easy-to-implement framework HeteroFL that can train heterogeneous local models and aggregate them stably and effectively into a single global inference model.
...
1.Federated Meta-Learning with Fast Convergence and Efficient Communication - ArXiv’18 code
They proposed a federated meta-learning framework FedMeta, where a parameterized algorithm (or meta-learner) is shared, instead of a global model in previous approaches.
2.Personalized Federated Learning: A Meta-Learning Approach - ArXiv’20 code
Per-FedAvg - Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach - NeurIPS'20 code
They studied a personalized variant of the federated learning in which our goal is to find an initial shared model that current or new users can easily adapt to their local dataset by performing one or a few steps of gradient descent with respect to their own data.
3.pFedMe - Personalized Federated Learning with Moreau Envelopes - NeurIPS'20 code
They proposed an algorithm for personalized FL (pFedMe) using Moreau envelopes as clients' regularized loss functions, which help decouple personalized model optimization from the global model learning in a bi-level problem stylized for personalized FL.
...
1.MOCHA - Federated Multi-Task Learning - NeurIPS'17 code
They proposed a novel systems-aware optimization method MOCHA, which is robust to practical systems issues. Their method and theory for the first time consider issues of high communication cost, stragglers, and fault tolerance for distributed multi-task learning.
2.Three Approaches for Personalization with Applications to Federated Learning - ArXiv'20 code
They proposed and analyzed three approaches: user clustering, data interpolation, and model interpolation.
3.VIRTUAL - Variational Federated Multi-Task Learning - ArXiv’21
They introduced VIRTUAL, an algorithm for federated multi-task learning for general non-convex models.In VIRTUAL the federated network of the server and the clients is treated as a star-shaped Bayesian network, and learning is performed on the network using approximated variational inference.
4.FedAMP - Personalized Cross-Silo Federated Learning on Non-IID Data - AAAI'21
They proposed FedAMP, a new method employing federated attentive message passing to facilitate similar clients to collaborate more. They established the convergence of FedAMP for both convex and non-convex models, and propose a heuristic method to further improve the performance of FedAMP when clients adopt deep neural networks as personalized models.
5.Federated Multi-Task Learning under a Mixture of Distributions - NeurIPS'21 code
They proposed to study federated MTL under the flexible assumption that each local data distribution is a mixture of unknown underlying distributions. This assumption encompasses most of the existing personalized FL approaches and leads to federated EM-like algorithms for both client-server and fully decentralized settings. Moreover, it provides a principled way to serve personalized models to clients not seen at training time.
...
1.CFL - Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints - IEEE TNNLS'20 code
They presented clustered FL (CFL), a novel federated multitask learning (FMTL) framework, which exploits geometric properties of the FL loss surface to group the client population into clusters with jointly trainable data distributions.
2.FL+HC - Federated learning with hierarchical clustering of local updates to improve training on non-IID data - IJCNN'20
The presented a modification to FL by introducing a hierarchical clustering step (FL+HC) to separate clusters of clients by the similarity of their local updates to the global joint model.
static clustering
3.IFCA - An Efficient Framework for Clustered Federated Learning NeurIPS'20 code
They proposed a new framework dubbed the Iterative Federated Clustering Algorithm (IFCA), which alternately estimates the cluster identities of the users and optimizes model parameters for the user clusters via gradient descent.
4.FeSEM - Multi-center federated learning: clients clustering for better personalization - World Wide Web'23 code
They proposed a novel multi-center aggregation mechanism to cluster clients using their models' parameters. It learns multiple global models from data as the cluster centers, and simultaneously derives the optimal matching between users and centers. They then formulate it as an optimization problem that can be efficiently solved by a stochastic expectation maximization (EM) algorithm.
5.FlexCFL - Flexible Clustered Federated Learning for Client-Level Data Distribution Shift - TPDS'21 code
FlexCFL leverages a novel decomposed data-driven measure called euclidean distance of Decomposed Cosine similarity (EDC) for client clustering. Another design that makes FlexCFL more practical is we maintain an auxiliary server to address the cold start issue of new devices. Furthermore, FlexCFL can detect the client-level data distribution shift based on Wasserstein distance and migrate clients with affordable communication.
semi-dynamic clustering
6.FedSim - FedSim: Similarity guided model aggregation for Federated Learning - Neurocomputing'22 code
FedSim decomposes FL aggregation into local and global steps. Clients with similar gradients are clustered to provide local aggregations, which thereafter can be globally aggregated to ensure better coverage whilst reducing variance.
...
1.FedMA - Federated Learning with Matched Averaging - ArXiv'20 code
FedMA constructs the shared global model in a layer-wise manner by matching and averaging hidden elements (i.e. channels for convolution layers; hidden states for LSTM; neurons for fully connected layers) with similar feature extraction signatures.
2.FedDist - A Federated Learning Aggregation Algorithm for Pervasive Computing: Evaluation and Comparison - PerCom'21 code
They propose a novel aggregation algorithm, termed FedDist, which is able to modify its model architecture (here, deep neural network) by identifying dissimilarities between specific neurons amongst the clients. This permits to account for clients' specificity without impairing generalization.
1.pFedLA - Layer-wised Model Aggregation for Personalized Federated Learning - CVPR'21 code
They proposed a dedicated hypernetwork per client on the server side, which is trained to identify the mutual contribution factors at layer granularity. Meanwhile, a parameterized mechanism is introduced to update the layer-wised aggregation weights to progressively exploit the inter-user similarity and realize accurate model personalization.
2.pFedHN - Personalized Federated Learning using Hypernetworks - PMLR'21 code
In this approach, a central hypernetwork model is trained to generate a set of models, one model for each client. This architecture provides effective parameter sharing across clients while maintaining the capacity to generate unique and diverse personal models.
...
1.Ditto - Ditto: Fair and Robust Federated Learning Through Personalization - PMLR’21 code
They propose employing a simple, general framework for personalized federated learning, Ditto, that can inherently provide fairness and robustness benefits, and develop a scalable solver for it.
2.FAIR - FAIR: Quality-Aware Federated Learning with Precise User Incentive and Model Aggregation - IEEE INFOCOM'21
They proposed a novel system named FAIR. FAIR integrates three major components: 1) learning quality estimation: we leverage historical learning records to estimate the user learning quality; 2) quality-aware incentive mechanism: within the recruiting budget, they model a reverse auction problem to encourage the participation of high-quality learning users; and 3) model aggregation: we devise an aggregation algorithm that integrates the model quality into aggregation and filters out non-ideal model updates, to further optimize the global learning model.
3.FedPAC - Personalized federated learning with feature alignment and classifier collaboration - ICLR'23 code
They conducted explicit local-global feature alignment by leveraging global semantic knowledge for learning a better representation. Moreover, we quantify the benefit of classifier combination for each client as a function of the combining weights and derive an optimization problem for estimating optimal weights.
...
...
LEAF: A Benchmark for Federated Settings - ArXiv'19
A 10-class handwritten digits image classification task, which is divided into 1,000 clients, each with only two classes of digits.
A handwritten digits and characters image classification task, which is built by resampling the EMNIST according to the writer and downsampling to 10 classes ('a'-'j').
It's a synthetic federated dataset.
A 28*28 grayscale images classification task, which comprises 70,000 fashion products from 10 categories.
A tweets sentiment analysis task, which contains 772 clients, each client is a different Twitter account.