ju-chen/Efficient-Prompt

Questions on calculating the accuracy in validation phase

Alienge opened this issue · 0 comments

In val.py file

`

    sim_ensemble = torch.zeros(0).to(device)
    test_num = int(len(similarity)/valEnsemble)
    for enb in range(valEnsemble):
          sim_ensemble = torch.cat([sim_ensemble, similarity[enb*test_num:enb*test_num+test_num].unsqueeze(0)], dim=0) 
    target_final = targets[:test_num]

    sim_final = torch.mean(sim_ensemble, 0)
    values, indices = sim_final.topk(5)
    top1 = (indices[:, 0] == target_final).tolist()
    top5 = ((indices == einops.repeat(target_final, 'b -> b k', k=5)).sum(-1)).tolist()

    top1ACC = np.array(top1).sum() / len(top1)
    top5ACC = np.array(top5).sum() / len(top5)

`

I see you split the validation datasets into $valEnsemble$ parts, and then get average from $valEnsemble$ parts. Finally,
calculate average part accuracy in validation datasets. Not 0:len(similarity). so I want to know what reasons u do this? To my knowledge, isn't it more convincing to calculate accuracy of 0:len(similarity).

I am looking forward your reply!
Thx