top-k multiclass macro accuracy is not calculated correctly
markussteindl opened this issue ยท 0 comments
markussteindl commented
๐ Bug
If some classes are not observed, the top-k multiclass macro accuracy is not calculated correctly.
To Reproduce
from torchmetrics import Accuracy
import torch
acc1 = Accuracy(task="multiclass", num_classes=3, average="macro")
acc2 = Accuracy(task="multiclass", num_classes=3, average="macro", top_k=2)
preds = torch.tensor([[0.9, 0.1, 0.0], [0.9, 0.1, 0.0], [0.9, 0.1, 0.0]])
targets = torch.tensor([0, 0, 2])
print(acc1(preds, targets)) # tensor(0.5000) --> ok
print(acc2(preds, targets)) # tensor(0.3333) --> inconsistent
Expected behavior
In the example, the top-1 and top-2 accuracy should be equal.
Note that class 1 does not occur in the targets.
The classes have the following top-1 and top-2 accuracies before averaging:
- class 0: 1.0
- class 1: nan
- class 2: 0.0
Thus, the macro aggregation of the top-1 and top-2 accuracy should be equal with a value of 0.5 (or 0.333, if we take the nan into account).
Environment
- TorchMetrics version: 1.3.0.post0, 1.3.1
- Python & PyTorch Version (e.g., 1.0): Python 3.11, torch 2.2.0
- Windows 10, 22H2, 19045.4046
- Environment managed by poetry