The results in the paper do not correspond to the code

[wangzihuixmu]

the results from table 1 is participant_model's performance on validation?
In the paper, we can get MNIST_CLA_10 accuracy is 73(94) from table 1, but after run the code, we can get valid accuracy which from RFFL_run.py 391 line is 53(94).

[XinyiYS]

Hi @wangzihuixmu,

Thanks for raising this. I did track down the empirical discrepancy. There is an implementation detail missed out in the code. See below for RFFL_run.py.
Current lines 361-363:
rs = torch.div(rs, rs.sum())
r_threshold.append( threshold * (1.0 / len(R_set)) )
q_ratios = torch.div(rs, torch.max(rs))

After adding the implementation detail:
rs = torch.div(rs, rs.sum())
r_threshold.append( threshold * (1.0 / len(R_set)) )

Added lines begin
rs = torch.div(rs, torch.max(rs))
rs = torch.tanh(rs * 2) # 2 is constant called the altruism degree
rs = torch.div(rs, torch.max(rs))
Added lines end

q_ratios = torch.div(rs, torch.max(rs))

You will be able to reach the higher average validation accuracy with this. This implementation detail does not impact the theory of this work. It is based on the analysis of our subsequent full conference paper in NeurIPS-2021 with code. These two works have slightly different algorithmic details, so adding these lines does not give you the implementation of the other work but can help verify the empirical results.

[XinyiYS]

Closing due to inactivity.