/fl-ml

This is official code for ACIIDS2022 paper "Meta-learning and Personalization layer in Federated learning"

Primary LanguageJupyter NotebookMIT LicenseMIT

Meta-learning and Personalization layer in Federated learning

Official code for ACIIDS2022 paper "Meta-learning and Personalization layer in Federated learning".

General information

  • Supervisor: Prof. Lê Hoài Bắc
  • Reviewer: Dr. Nguyễn Tiến Huy
  • Authors:
    • Nguyễn Bảo Long - MSSV: 18120201
    • Cao Tất Cường - MSSV: 18120296
  • Reporting date: 15/03/2022 at Computer Science No.1, University of Science, VietNam National University - Ho Chi Minh City.

Contact

How to run

  • Dataset configuration: Dataset is configured as in Personalized Federated Learning with Moreau Envelopes (NeurIPS 2020).

  • 2 ways to run simulation (read Flower's doc for more detail):

    • Normal mode: Run file run.sh. This file contains all the command codes that output the results of this thesis. After running this file, function start_simulation() in file ./main.py will be called.

    • Debug mode: Run 2 files ./run_client.sh and ./run_server.sh. File ./run_server.sh calls main() in file ./server/server_main.py in order to create and run a server. File ./run_client.sh creates a certain number of clients by calling function main() in file ./client/client_main.py multiple times.

Some information about source code

  • Folder ./client: Defines types of clients of FL systems (based on Flower framework).

  • Folder ./client_worker: Defines training methods (meta learning, using learn2learn and conventional FL training) and testing methods. These functions in here will be called by clients in ./client.

  • Folder ./data: Contains data generator (./data/mnist, ./data/cifar), and data loader (./data/dataloaders) for each client.

  • Folder ./document: Contains a presentation, a thesis and relevant documents.

  • Folder ./experiments: Results of FedAvg, FedAvgMeta, FedPer, FedPerMeta, FedMeta(MAML), FedMeta(Meta-SGD), FedMeta-Per(MAML), FedMeta-Per(Meta-SGD) running on MNIST, CIFAR-10, and on 2 types of client (new client, local client).

  • Folder ./model: Defines models and model wrapper for MNIST, CIFAR-10.

  • Folder ./personalized_weight: A temporary folder, generated during the execution of algorithms using personalization layer. This folder contains personalization layer of each client.

Results

  • We proposed FedMeta-Per (FedMeta-Per(MAML), FedMeta-Per(Meta-SGD)), a combination of Meta-learning and Personalization layer into a FL system.

MNIST

  • Classification results (%) of local client using MNIST dataset
$acc_{micro}$ $acc_{macro}$ $P_{macro}$ $R_{macro}$ $F1_{macro}$
FedAvg 85.03 82.14±14.76 82.03±13.88 81.54±14.33 79.43±16.83
FedPer 77.29 75.48±14.84 76.07±14.99 74.01±15.13 72.32±15.99
FedAvgMeta 84.84 81.56±16.68 80.71±17.02 81.18±16.16 78.31±19.8
FedPerMeta 75.91 74.11±16.2 75.68±15.94 72.93±15.58 71.22±16.77
FedMeta(MAML) 92.99 91.14±5.99 90.56±6.24 90.98±5.9 90.16±6.28
FedMeta(Meta-SGD) 98.02 96.35±4.62 96.49±4.1 95.64±5.94 95.80±5.51
FedMeta-Per(MAML) 99.37 99.12±1.29 99.11±1.3 98.82±1.99 98.94±1.6
FedMeta-Per(Meta-SGD) 98.92 98.15±3.32 98.42±1.95 98.42±1.96 98.20±2.94
  • Classification results (%) on new client using MNIST dataset
$acc_{micro}$ $acc_{macro}$ $P_{macro}$ $R_{macro}$ $F1_{macro}$
FedAvg 83.92 81.69±19.71 79.57±20.18 80.46±17.84 77.66±22.54
FedPer 78.3 76.19±18.79 75.91±17.52 74.73±17.32 72.72±19.3
FedAvgMeta 84.34 82.37±17.42 81.38±16.25 80.91±15.62 78.78±19.31
FedPerMeta 77.47 75.56±20.33 75.09±19.52 74.92±18.85 72.60±21.37
FedMeta(MAML) 92.96 91.88±5.88 90.14±7.97 90.74±5.95 90.02±7.34
FedMeta(Meta-SGD) 96.39 93.53±8.39 93.73±10.26 88.65±14.06 89.31±14.56
FedMeta-Per(MAML) 93.6 93.57±5.58 93.64±5.56 90.98±6.98 91.83±6.43
FedMeta-Per(Meta-SGD) 96.62 95.88±3.58 95.73±4.11 94.34±5.05 94.85±4.61

CIFAR-10

  • Classification results (%) of local client using CIFAR-10 dataset
$acc_{micro}$ $acc_{macro}$ $P_{macro}$ $R_{macro}$ $F1_{macro}$
FedAvg 19.02 19.29±25.11 15.57±23.7 20.65±25.55 16.85±23.92
FedPer 13.22 12.99±19.39 18.34±28.59 14.14±20.83 10.52±14.91
FedAvgMeta 40.3 38.47±31.52 32.84±32.06 39.33±30.35 33.81±30.61
FedPerMeta 18.57 17.48±22.55 20.02±27.4 18.43±23.47 14.54±18.67
FedMeta(MAML) 69.02 68.76±14.86 67.42±21.16 66.56±13.48 61.14±20
FedMeta(Meta-SGD) 78.63 78.73±11.59 74.65±21.12 75.25±14.09 72.87±18.31
FedMeta-Per(MAML) 86.6 86.52±6.31 86.43±5.88 85.47±6.87 85.33±6.77
FedMeta-Per(Meta-SGD) 85.61 85.68±7.22 86.26±6.35 85.36±6.83 85.08±7.32
  • Classification results (%) of new client using CIFAR-10 dataset
$acc_{micro}$ $acc_{macro}$ $P_{macro}$ $R_{macro}$ $F1_{macro}$
FedAvg 24.63 24.83±22.57 18.36±20.15 24.44±21.95 20.52±20.45
FedPer 14.4 14.52±20.15 12.59±20.65 14.23±19.58 10.66±13.79
FedAvgMeta 43.39 43.54±18 33.45±21.44 42.87±16.98 35.14±17.22
FedPerMeta 13.33 13.57±19.62 11.99±19.52 13.53±19.08 10.05±13.17
FedMeta(MAML) 61.69 61.64±12.49 52.66±26.06 59.94±12.35 50.76±19.2
FedMeta(Meta-SGD) 68.36 67.89±15.11 70.3±22.37 66.86±15.02 60.24±21.52
FedMeta-Per(MAML) 64.22 63.70±12.29 57.06±24.99 61.63±12.66 53.68±19.06
FedMeta-Per(Meta-SGD) 69.97 69.13±14.63 66.53±24.91 67.82±15.34 62.42±20.94

Visualization

  • FedMeta-Per vs. (FedAvg, FedAvgMeta, FedPer, FedPerMeta): The proposed methods achieves higher degree in term of convergence and accuracy compared with FedAvg and FedPer.

  • FedMeta-Per vs. FedMeta: Improved personalization is the reason why results on local clients of FedMeta-Per achieve faster convergence and higher accuracy than FedMeta. Regarding the new clients, 2 algorithms achieve the same degree of convergence. However, the personalization layer at each FedMeta-Per client will improve over time as the client participates in one or more local training step (new client becomes local client).