We run the federated experiments for the following data distributions, number of clients, and participation rates:
| Data Distribution | (Number of Clients, Participation Rate) |
|---|---|
| {IID, Non-IID} | (10, 1), (100, 1) |
If participation rate is 1, e.g., (10, 1) then all clients are considered at every federation round, whereas if participation rate is 0.1, e.g., (100, 0.1) then only 10% of the total clients are considered at every round.
IID refers to the case where the training dataset is randomly partitioned across all participating clients and non-IID in the case where the training dataset is first sorted by the prediction class (classification task) and then assigned in a round-robin fashion to the participating clients.
| Domain | Model (#params) | Hyperparameters1 | Centralized Dataset | Federated Dataset |
|---|---|---|---|---|
| FashionMNIST | 2-FC (120k) | {opt: SGD, lr: 0.02, b: 32, le:4} | Train: 60000, Test: 10000 | IID: random shuffle, non-IID(2): examples from 2 classes per client |
| CIFAR-10 | 6-CNN (1.6M) | {opt: SGD w/ Momentum, lr: 0.005, m: 0.75, b: 32, le:4} | Train: 50000, Test: 10000 | IID: random shuffle, non-IID(5): examples from 5 classes per client |
| CIFAR-100 | VGG-16 (15M) | {opt: SGD w/ Momentum, lr: 0.01, m: 0.9, b: 128, le:4} | Train: 50000, Test: 10000 | IID: random shuffle, non-IID(50): examples from 50 classes per client |
[1]: opt stands for optimizer, lr for learning rate, b for batch size, le for local epochs within each round.
The fedpurgemerge_main directory contains all the source code required to run FedSparsify and other pruning techniques.
Here, we will illustrate a prototypical example based on FashionMNIST using different pruning techniques. The same logic holds for all other domains. For other domains, such as in CIFAR, where more than one models are tested we can uncomment the model generation commands and instead of CNN simply use VGG or ResNet.
For instance, to run a federated experiments over 100 learners with 10 random learners participating at each round with each learner training for 4 epochs locally and learning a model with a final sparsification degree of 80%, we need to run the fedpurgemerge_main/fashion_mnist_main.py script as follows:
cd fedpurgemerge_main; python3 fashion_mnist_main.py --iid_distribution=True --federation_rounds=200 --learners_num=100 --participation_rate=0.1 --local_epochs=4 --batch_size=32 --train_with_global_mask=True --start_sparsification_at_round=1 --sparsity_level=0.8 --sparsification_frequency=1 --merging_op="FedAvg" --purging_op="fedsparsify-global"
Similar flags as the ones used for the above case of FedSparsify-Global, but, we need to change the value of the purging_op flag to "fedsparsify-local".
For OneShot Pruning we can run the following command where we specify when the pruning should take place, e.g., at round 50. This command will simply prune the model at the specified pruning level and keep the sparsity at this degree for the remainder of the federated execution. For instance, if we want to prune a model at 90% soarsity at round 90 and tune the pruned model for another 10 rounds the pruned model, then we simply specify a total number of 100 federation rounds and start model pruning at round 90:
cd fedpurgemerge_main; python3 fashion_mnist_main.py --iid_distribution=True --federation_rounds=100 --learners_num=100 --participation_rate=0.1 --local_epochs=4 --batch_size=32 --train_with_global_mask=True --start_sparsification_at_round=90 --sparsity_level=0.8 --merging_op="FedAvg" --purging_op="oneshot"
In the fedpurgemerge_main/fashion_mnist_main.py we need to set the following flags:
cd fedpurgemerge_main; python3 fashion_mnist_main.py --iid_distribution=True --federation_rounds=200 --learners_num=100 --participation_rate=0.1 --local_epochs=4 --batch_size=32 --train_with_global_mask=True --sparsity_level=0.8 --merging_op="FedAvg" --purging_op="snip"
For GraSP we execute the same command but with --purging_op="grasp".
For PruneFL all the running scripts for all domains and models are located under fedpurgemerge_main/prunefl.
First, we need to convert the tensorflow models into .tflite models and then into the .onnx format. The critical script to accomplish this is located under utils/benchmark_model_inference.py. The script first generates the model structure / architecture and then sets the model weights learned during federated training. Once the new file is generated then the bash script under scripts/benchmark_model_inference.sh can be used to run the inference benchmark and report the corresponding measurements.