fine tuning RoBERTa-base with LoRA (ValueError: Classification metrics can't handle a mix of binary and multilabel-indicator targets)

[rozhix]

Hello, I'm trying to fine-tune RoBERTa-base with LoRA for a multi-classification task. My compute_metrics function is as follows:

def multi_label_metrics(predictions, labels, threshold=0.5):
# first, apply sigmoid on predictions which are of shape (batch_size, num_labels)
sigmoid = torch.nn.Sigmoid()
probs = sigmoid(torch.Tensor(predictions))
# next, use threshold to turn them into integer predictions
y_pred = np.zeros(probs.shape)
y_pred[np.where(probs >= threshold)] = 1
# finally, compute metrics
y_true = labels
f1_micro_average = f1_score(y_true=y_true, y_pred=y_pred, average='micro')
roc_auc = roc_auc_score(y_true, y_pred, average = 'micro')
accuracy = accuracy_score(y_true, y_pred)
# return as dictionary
metrics = {'f1': f1_micro_average,
'roc_auc': roc_auc,
'accuracy': accuracy}
return metrics

def compute_metrics(p: EvalPrediction):
preds = p.predictions[0] if isinstance(p.predictions,
tuple) else p.predictions
result = multi_label_metrics(
predictions=preds,
labels=p.label_ids)
return result

and this is my training part:

trainer = transformers.Trainer(
model=model,
train_dataset=encoded_dataset['train'],
eval_dataset=encoded_dataset['test'],
compute_metrics=compute_metrics,
args=transformers.TrainingArguments(
evaluation_strategy = "epoch",
save_strategy = "epoch",
per_device_train_batch_size=batch_size,
gradient_accumulation_steps=batch_size,
num_train_epochs=5,
weight_decay=0.01,
learning_rate=2e-4,
load_best_model_at_end=True,
metric_for_best_model=metric_name,
fp16=True,
logging_steps=1,
output_dir='outputs'
),
data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False)
)
model.config.use_cache = False
trainer.train()

When I run it I have this error:
ValueError: Classification metrics can't handle a mix of binary and multilabel-indicator targets

how should I change my multi_label_metrics function?