Lightning-AI/torchmetrics

top-k multiclass macro accuracy less than top-1 multiclass macro accuracy

markussteindl opened this issue · 2 comments

markussteindl commented

🐛 Bug

In some cases, the top-k (e.g. top-2) multiclass macro accuracy is less than the top-1 multiclass macro accuracy.

This seems to be the case when some classes were not observed in a batch.

To Reproduce

from torchmetrics import Accuracy
import torch

n_obs = 50
n_classes = 100

for i in range(1000):
    acc1 = Accuracy(task="multiclass", num_classes=n_classes, average="macro")
    acc2 = Accuracy(task="multiclass", num_classes=n_classes, average="macro", top_k=2)
    preds = torch.randn(n_obs, n_classes)
    targets = torch.randn(n_obs, n_classes).argmax(1)
    m1 = acc1(preds, targets)
    m2 = acc2(preds, targets)

    assert m1 <= m2

Expected behavior

In the example above, the assertion should never be violated. But it is.

Environment

  • TorchMetrics version: 1.3.0.post0, 1.3.1
  • Python & PyTorch Version (e.g., 1.0): Python 3.11, torch 2.2.0
  • Windows 10, 22H2, 19045.4046
  • Environment managed by poetry
github-actions commented

Hi! thanks for your contribution!, great first issue!

markussteindl commented

The issue seems to be a bug in the calculation of the top-k multiclass macro accuracy.
I created another issue with a minimal example: #2418

  • 👍1