Question about the experiment setting
Closed this issue · 4 comments
Hi, @samiul272 , I would like to know whether all your experiments in the paper set cfg['model_split_mode'] as dynamic, can you tell me?
Yes. All experiments are done with 'dynamic' split mode.
Yes. All experiments are done with 'dynamic' split mode.
Ok, I see. I would like to ask whether you have compared FedRolex with FedDropout and HeteroFL in 'dynamic' and 'fix' cases respectively. Is your method the best in both cases?
In my opinion, the memory of a certain device is fixed. For example, for a client, after selecting a small model with a pruning rate of 0.8 in the t communication round, it is unlikely to choose a model with a pruning rate of 0.1 (big model) in the t+1 communication round, and subsequent communication rounds, because of its own memory limitation. So I think maybe 'fix' split mode is more reflective of real world scenarios. What's your opinion?
Ok, I see. I would like to ask whether you have compared FedRolex with FedDropout and HeteroFL in 'dynamic' and 'fix' cases respectively. Is your method the best in both cases?
We compared only with dynamic. There was not enough time to do the tests for fixed mode but I think all algorithms will have a lower accuracy score compared to dynamic as this has been shown in the HeteroFL paper.
In my opinion, the memory of a certain device is fixed. For example, for a client, after selecting a small model with a pruning rate of 0.8 in the t communication round, it is unlikely to choose a model with a pruning rate of 0.1 (big model) in the t+1 communication round, and subsequent communication rounds, because of its own memory limitation. So I think maybe 'fix' split mode is more reflective of real-world scenarios. What's your opinion?
You are correct in assuming this for devices with a fixed load. However, our paper (and HeteroFL) was designed with mobile devices in mind and it is quite possible for a mobile device to choose a lower pruning rate if it has fewer apps running at a given time. You can check the details on why dynamic mode is needed in that paper too.
We compared only with dynamic. There was not enough time to do the tests for fixed mode but I think all algorithms will have a lower accuracy score compared to dynamic as this has been shown in the HeteroFL paper.
I agree with you that this is predictable because the fixed mode does not take into account data heterogeneity, and some modules of a large model may not be able to see all the data from clients.
You are correct in assuming this for devices with a fixed load. However, our paper (and HeteroFL) was designed with mobile devices in mind and it is quite possible for a mobile device to choose a lower pruning rate if it has fewer apps running at a given time. You can check the details on why dynamic mode is needed in that paper too.
Thank you for your reminding. I think what you said is quite reasonable. Thank you.