/FL-bench

Benchmark of federated learning. Dedicated to the community. πŸ€—

Primary LanguagePythonGNU General Public License v2.0GPL-2.0

Federated Learning Benchmark

GitHub License GitHub closed issues GitHub Discussions GitHub Repo stars GitHub Repo forks

This is a benchmark for evaluating well-known federated learning (FL) and personalized federated learning (pFL) methods. This benchmark is not complicated and easy to extend.

Environment Preparation πŸš€

I offer three approaches to install all required packages.

  1. pip install -r requirements.txt

  2. conda env create -f environment.yml

  3. poetry (Recommended πŸ‘. To keep all dependencies you install are identical to mine.)

    • If you are new to poetry πŸ‘€:
      1. pip install poetry
      2. cd to the root directory of FL-bench and run poetry install
      3. Run poetry shell to enter the virtual environement or poetry run ${your_command}.
      4. Have fun. πŸ€—
    • If you are not at China mainland:
      1. Comment mirror related configs in pyproject.toml.
      2. Run poetry lock --no-update to update the poetry.lock.

Method 🧬

Traditional FL Methods

Personalized FL Methods

More reproductions/features would come soon or later (depends on my mood 🀣).

Easy Run πŸƒβ€β™‚οΈ

ALL classes of methods are inherited from FedAvgServer and FedAvgClient. If you wanna figure out the entire workflow and detail of variable settings, go check ./src/server/fedavg.py and ./src/client/fedavg.py.

# partition the CIFAR-10 according to Dir(0.1) for 100 clients
cd data
python generate_data.py -d cifar10 -a 0.1 -cn 100
cd ../

# run FedAvg under default setting.
cd src/server
python fedavg.py

About methods of generating federated dastaset, go check data/README.md for full details.

Monitor πŸ“ˆ (optional and recommended πŸ‘)

  1. Run python -m visdom.server on terminal.
  2. Run src/server/${algo}.py --visible 1
  3. Go check localhost:8097 on your browser.

Generic Arguments πŸ”§

πŸ“’ All generic arguments have their default value. Go check get_fedavg_argparser() in FL-bench/src/server/fedavg.py for full details about generic arguments.

About the default values and hyperparameters of advanced FL methods, go check corresponding FL-bench/src/server/${algo}.py for full details.

Argument Description
--dataset The name of dataset that experiment run on.
--model The model backbone experiment used.
--seed Random seed for running experiment.
--join_ratio Ratio for (client each round) / (client num in total).
--global_epoch Global epoch, also called communication round.
--local_epoch Local epoch for client local training.
--finetune_epoch Epoch for clients fine-tunning their models before test.
--test_gap Interval round of performing test on clients.
--eval_test Non-zero value for performing evaluation on joined clients' testset before and after local training.
--eval_train Non-zero value for performing evaluation on joined clients' trainset before and after local training.
--local_lr Learning rate for client local training.
--momentum Momentum for client local opitimizer.
--weight_decay Weight decay for client local optimizer.
--verbose_gap Interval round of displaying clients training performance on terminal.
--batch_size Data batch size for client local training.
--use_cuda Non-zero value indicates that tensors are in gpu.
--visible Non-zero value for using Visdom to monitor algorithm performance on localhost:8097.
--global_testset Non-zero value for evaluating client models over the global testset before and after local training, instead of evaluating over clients own testset. The global testset is the union set of all client's testset. NOTE: Activating this setting will considerably slow down the entire training process, especially the dataset is big.
--save_log Non-zero value for saving algorithm running log in FL-bench/out/${algo}.
--straggler_ratio The ratio of stragglers (set in [0, 1]). Stragglers would not perform full-epoch local training as normal clients. Their local epoch would be randomly selected from range [--straggler_min_local_epoch, --local_epoch).
--straggler_min_local_epoch The minimum value of local epoch for stragglers.
--save_model Non-zero value for saving output model(s) parameters in FL-bench/out/${algo}.
--save_fig Non-zero value for saving the accuracy curves showed on Visdom into a .jpeg file at FL-bench/out/${algo}.
--save_metrics Non-zero value for saving metrics stats into a .csv file at FL-bench/out/${algo}.

Supported Datasets 🎨

This benchmark only support algorithms to solve image classification task for now.

Regular Image Datasets

  • MNIST (1 x 28 x 28, 10 classes)

  • CIFAR-10/100 (3 x 32 x 32, 10/100 classes)

  • EMNIST (1 x 28 x 28, 62 classes)

  • FashionMNIST (1 x 28 x 28, 10 classes)

  • Syhthetic Dataset

  • FEMNIST (1 x 28 x 28, 62 classes)

  • CelebA (3 x 218 x 178, 2 classes)

  • SVHN (3 x 32 x 32, 10 classes)

  • USPS (1 x 16 x 16, 10 classes)

  • Tiny-ImageNet-200 (3 x 64 x 64, 200 classes)

  • CINIC-10 (3 x 32 x 32, 10 classes)

  • DomainNet (3 x ? x ?, 345 classes)

Medical Image Datasets

Acknowledgement πŸ€—

This benchmark does not contain the feature/method you interested? Describe them in here. I can't guarantee your request will be accomplised in time or even considered. So feel free to do it! πŸ’‘

Some reproductions in this benchmark are referred to https://github.com/TsingZ0/PFL-Non-IID, which is a great FL benchmark. πŸ‘

This benchmark is still young, which means I will update it frequently and unpredictably. Therefore, periodically fetching the latest code is recommended. πŸ€–

If this benchmark is helpful to your research, it's my pleasure. 😏