Using MetaICL for multi-label classification

[thesofakillers]

I may have missed this in the paper, but from my understanding, the setup used for MetaICL doesn't lend itself too well for multi-label classification, where you can classify a given input as multiple labels (for example picture of cat -> mammal, cat, animal, given options mammal, cat, animal, reptile, plant, etc.).

I say this because I imagine that evaluating the log likelihood for every option combination sounds unfeasible. You would have to evaluate [mammal], [mammal, cat], [mammal, cat, animal], [reptile], [reptile, cat], etc... You can binarise this to getting 2^N completion options for a problem with N labels, which for N>10 is already in the thousands, running into prompt length limits

My question is, have the authors considered a way to circumvent this issue? Is this even an issue or is my understanding incorrect? What are your thoughts?

Thank you.

[shmsw25]

Hi, thanks for the question! You're right that we did not evaluate on multi-label classification, so not exactly sure how well it would perform. But one suggestion to make it work is something like thresholding, e.g., feed each class label independently, and take all labels whose log likelihood is above a particular threshold (which can be hyperparameter and will need tuning).

[thesofakillers]

Thanks for the swift response!