AxisError: axis 1 is out of bounds for array of dimension 1

[apethree]

Steps to reproduce:
Notebook: https://github.com/georgian-io/Multimodal-Toolkit/tree/master/notebooks
Run this notebook on evuluation phase you will see this error:

There is some issue with the way the numpy is handling the argmax() function.
With axis = 0 the error is

ValueError: could not broadcast input array from shape (2348,768) into shape (2348,)

With axis = 1 the error is

AxisError: axis 1 is out of bounds for array of dimension 1

The shape of the numpy array is (2,) so I am not sure how to resolve this issue.

------ Full evaluation error on train.evaluation() ----------

---------------------------------------------------------------------------
AxisError                                 Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_15804\2732109216.py in <module>
----> 1 trainer.evaluate()

~\anaconda3\envs\py-win-37\lib\site-packages\transformers\trainer.py in evaluate(self, eval_dataset, ignore_keys, metric_key_prefix)
   2937             prediction_loss_only=True if self.compute_metrics is None else None,
   2938             ignore_keys=ignore_keys,
-> 2939             metric_key_prefix=metric_key_prefix,
   2940         )
   2941 

~\anaconda3\envs\py-win-37\lib\site-packages\transformers\trainer.py in evaluation_loop(self, dataloader, description, prediction_loss_only, ignore_keys, metric_key_prefix)
   3218                 )
   3219             else:
-> 3220                 metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels))
   3221         else:
   3222             metrics = {}

~\AppData\Local\Temp\ipykernel_15804\165556549.py in calc_classification_metrics(p)
     22 #   pred_scores = softmax(p.predictions)
     23 
---> 24   pred_labels = np.argmax(p.predictions, axis=1)
     25   print('\nNot Executed\n')
     26   pred_scores = softmax(p.predictions, axis=1)[:, 1]

<__array_function__ internals> in argmax(*args, **kwargs)

~\anaconda3\envs\py-win-37\lib\site-packages\numpy\core\fromnumeric.py in argmax(a, axis, out)
   1193 
   1194     """
-> 1195     return _wrapfunc(a, 'argmax', axis=axis, out=out)
   1196 
   1197 

~\anaconda3\envs\py-win-37\lib\site-packages\numpy\core\fromnumeric.py in _wrapfunc(obj, method, *args, **kwds)
     52     bound = getattr(obj, method, None)
     53     if bound is None:
---> 54         return _wrapit(obj, method, *args, **kwds)
     55 
     56     try:

~\anaconda3\envs\py-win-37\lib\site-packages\numpy\core\fromnumeric.py in _wrapit(obj, method, *args, **kwds)
     41     except AttributeError:
     42         wrap = None
---> 43     result = getattr(asarray(obj), method)(*args, **kwds)
     44     if wrap:
     45         if not isinstance(result, mu.ndarray):

AxisError: axis 1 is out of bounds for array of dimension 1

[apethree]

I have made progress. The shapes of the arrays are:

p.predictions of the type:

p.predictions[0] -->. Array of size (1200,2)
p.predictions[1] --> List [0], [1]
Where [0] ---> (1200,768)
Where [1] ---> (1200,2)

What is the 768 wide row? Not sure how to use that in prediction and F1 score.

[apethree]

This the probably the correct way to calculate argmax():

  pred_labels = np.argmax(p.predictions[0], axis=1)
  pred_scores = softmax(p.predictions[0], axis=1)[:, 1]

Instead of

  pred_labels = np.argmax(p.predictions, axis=1)
  pred_scores = softmax(p.predictions, axis=1)[:, 1]

[akashsara]

Hey @apethree, you've hit the solution there. Due to a change in how EvalPrediction works, the calc_classification_metrics is bugged. The variable with the 768 refers to the embeddings before the final layer. Just use p.predictions[0] instead and you should be good.