/FLBenchmark-toolkit

Federated Learning Framework Benchmark (UniFed)

Primary LanguagePythonApache License 2.0Apache-2.0

UniFed: All-In-One Federated Learning Platform to Unify Open-Source Frameworks

🌟For the benchmark result📊, please check our website.👈👈👈

image

Evaluation Scenarios

Cross-device horizontal

Scenario name Modality Task type Performance metrics Client number Sample number
celeba Image Binary Classification
(Smiling vs. Not smiling)		 Accuracy 894 20,028
femnist Image Multiclass Classification
(62 classes)		 Accuracy 178 40,203
reddit Text Next-word Prediction Accuracy 813 27,738

Cross-silo horizontal

Scenario name Modality Task type Performance metrics Client number Sample number
breast_horizontal Medical Binary Classification AUC 2 569
default_credit_horizontal Tabular Binary Classification AUC 2 22,000
give_credit_horizontal Tabular Binary Classification AUC 2 150,000
student_horizontal Tabular Regression
(Grade Estimation)		 MSE 2 395
vehicle_scale_horizontal Image Multiclass Classification
(4 classes)		 Accuracy 2 846

Cross-silo vertical

Scenario name Modality Task type Performance metrics Vertical split details
breast_vertical Medical Binary Classification AUC A: 10 features 1 label
B: 20 features
default_credit_vertical Tabular Binary Classification AUC A: 13 features 1 label
B: 10 features
dvisits_vertical Tabular Regression
(Number of consultations Estimation)		 MSE A: 3 features 1 label
B: 9 features
give_credit_vertical Tabular Binary Classification AUC A: 5 features 1 label
B: 5 features
motor_vertical Sensor data Regression
(Temperature Estimation)		 MSE A: 4 features 1 label
B: 7 features
student_vertical Tabular Regression
(Grade Estimation)		 MSE A: 6 features 1 label
B: 7 features
vehicle_scale_vertical Image Multiclass Classification
(4 classes)		 Accuracy A: 9 features 1 label
B: 9 features

Installation

Requirements

  • Python
    Recommend to use Python 3.9. It should also work on Python>=3.6, feel free to contact us if you encounter any problems.
    You can set up a sandboxed python environment by conda easily: conda create -n flbenchmark python=3.9
  • Command line tools: git, wget, unzip
  • Docker Engine
  • NVIDIA Container Toolkit if you want to use GPU.

Install From PyPI

pip install flbenchmark colink

Launch a benchmark (auto-deployment on AWS with a controller)

We highly recommend to use cloud servers(e.g. AWS) to run the benchmark. Here we provide an auto-deploy script on AWS.

Set up the controller server

  • Set up a web server(e.g. Apache2, Nginx, PHP built-in web server) with PHP support with files on controller folder.
  • Change CONTROLLER_URL="http://172.31.2.2" in install_auto.sh to the controller's web url.

Launch servers

  • Launch enough servers you need for the benchmark(e.g. 179 servers for femnist) on AWS and set the user data to the following script. (remember to replace the http://172.31.2.2 with the controller's web url)
#!/bin/bash
sudo -i -u ubuntu << EOF
echo "Hello World" > testfile.txt
wget http://172.31.2.2/install_auto.sh
bash install_auto.sh > user_data.out
EOF
  • Wait for server setups, when they are finished you could see all servers' ip under controller/servers.
  • Get server list via python get_server_list.py, and copy the server_list.json to exp folder.

Set up the framework operator

  • Change the working directory to exp for later steps.
  • Register users via python register_users.py.
  • Launch corresponding framework operators ./start_po.a <unifed_protocol_name> <repo_url>. For example, ./start_po.a unifed.flower https://github.com/CoLearn-Dev/colink-unifed-flower.git.

Start an evaluation

Launch a benchmark (manual deployment)

Alternatively, you could also manually deploy on your own cluster. You need to prepare enough Ubuntu 20.04 LTS servers based on the needs of the benchmark, and you should set up the environment on these servers. Here we provide one script to set up the environment.

Set up servers

  • Download install.sh on home directory.
  • Change SERVER_IP="172.16.1.1" in install.sh to corresponding server ip.
  • Execute install.sh.
  • Record all servers' ip and ~/server/host_token.txt in the following json format.
{
    "test-0": [
        "http://172.31.4.48:80",
        "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJwcml2aWxlZ2UiOiJob3N0IiwidXNlcl9pZCI6IjAyMGJhNzkyNzk0ZTlmMWUwZWZmNTEyOGM4NDdjZmE0MmRlNTllY2I1ODM4MzU4MDBmN2QwMzM1Yzg2YWFjZTViOSIsImV4cCI6MTY4OTM0ODAyM30.UP5JUYdbL-MkZDTSVuBHnIHoun1VkfcRgsBLV119v6A"
    ],
    "test-1": [
        "http://172.31.15.143:80",
        "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJwcml2aWxlZ2UiOiJob3N0IiwidXNlcl9pZCI6IjAyNzJhNjgxNDg0NDMwNTFmZTI2NDFlYmZiNjM2MDgxODM5YmQ5NDdkZGFhNTcwYjY3MjU0MTI2NjU5YzBmZjVjYSIsImV4cCI6MTY4OTM0ODAzNH0.2HHQYzcMjif0ZkhSyltlOSC1ydsgWS8H_no5wWvohw0"
    ]
}

Set up the framework operator

  • Change the working directory to exp for later steps.
  • Put the json file about servers from the last step to exp/server_list.json.
  • Register users via python register_users.py and record the user id for later use.
  • Launch corresponding framework operators ./start_po.a <unifed_protocol_name> <repo_url>. For example, ./start_po.a unifed.crypten https://github.com/CoLearn-Dev/colink-unifed-crypten.git.

Start an evaluation

  • Generate the exp/config.json from wizard. Remember to replace the user_id with the user id you got from the previous steps.
  • Start an evaluation via python run_task.py

License

This project is licensed under Apache License Version 2.0. By contributing to the project, you agree to the license and copyright terms therein and release your contribution under these terms.