Fails to converge if n_clients=1
Closed this issue · 1 comments
In the federated learning with encryption example, MSE blows up if n_clients=1. Why is this?
The MSE blows up because the gradient descent optimization diverges. (A nice introduction can be found here: https://blog.paperspace.com/intro-to-optimization-in-deep-learning-gradient-descent/ first two paragraphs are relevant).
The optimization process is sensitive to the learning rate parameter eta. If it is chosen too big, then the error will keep increasing until everything blows up. If you set eta to something smaller, then eventually, the optimization will produce sensible results (e.g.: set eta to 0.001).
However,
in the current form of the optimization algorithm, the effect of the eta parameter changes with the size of the respective dataset. This is somewhat annoying for experimentation, as you would have to tune eta every time you change the number of parties.
Therefore, I submitted a change (PR #69) which removes this un-nice-ness. The learning rate eta will be independent of the dataset size. Which makes playing around way more fun! With 50 iterations
and an eta of 1.5 you will always get a MSE of 3775.50, irrespective of the number of parties.