Matrix Mismatch Error During DuoModalZeroShotModel Initialization in GATE Framework

[AntreasAntoniou]

Describe the bug
An execution error occurs when running a multi-modal learning model within the GATE framework, resulting in a shape mismatch between matrices during a linear operation. The specific error message is: RuntimeError: mat1 and mat2 shapes cannot be multiplied (2x768 and 512x2). This indicates that the tensor dimensions expected by the forward operation are incompatible.

To Reproduce

1. Initialize the environment with certain project configurations, as enumerated by the error trace.
2. Run the script gate/run.py while applying the given overrides related to the model and training configuration.
3. Observe that at some point when the DuoModalZeroShotModel is being instantiated, a forward pass is attempted with a dummy batch.
4. The process fails during this step, resulting in a RuntimeError, due to a shape mismatch between the tensors intended for matrix multiplication.

/disk/scratch_fast1/aantoni2/mambaforge/envs/gate/bin/python gate/run.py exp_name=alpha-flickr30k-SIGLIP-P16-224-7 encoder=timm encoder.image_size=224 encoder.pretrained=True encoder.timm_model_name=vit_base_patch16_siglip_224 encoder.clip_model_name=openai/clip-vit-base-patch16 adapter=duo-modal-zero-shot-classifier dataset=flickr30k optimizer.lr=1e-05 optimizer.weight_decay=0.01 trainer=image_to_text_zero_shot_classification evaluator=image_to_text_zero_shot_classification num_workers=32 seed=7 train_batch_size=64 eval_batch_size=64 train_iters=10000 learner.evaluate_every_n_steps=500

Expected behavior
The model should instantiate correctly, and tensor shapes should align appropriately to perform matrix multiplications during the linear algebra operations within the forward pass. The model's construction should not lead to this kind of shape mismatch error.

Error Logs

/disk/scratch_fast1/aantoni2/mambaforge/envs/gate/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py:755: FutureWarning: The class `PretrainedBartModel` has been depreciated, please use `BartPreTrainedModel` instead.
  warnings.warn(
...
RuntimeError: mat1 and mat2 shapes cannot be multiplied (2x768 and 512x2)
...

Desktop (please complete the following information):

• OS: The specific operating system on which the error occurred is not provided in the trace, but it appears to be a Unix-like system given the directory structure.
• Python Version: 3.10
• Framework Version: GATE environment specifics are not provided in the trace.

Additional context

• The warning about the deprecated PretrainedBartModel could be unrelated to the core runtime error but should be addressed as part of the code maintenance.
• The root of the error seems to be related to input dimensionality assumptions within the gate.models.task_adapters.duo_modal_zero_shot_classification.DuoModalZeroShotModel during model initialization.
• Further investigation is needed to ensure that all input feature dimensions are processed and projected correctly to match expected sizes for linear operations.

[AntreasAntoniou]

This was solved in commit
7fbb1dd