Multimodal Transformers
This code runs inference with the multimodal transformer models described in "Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers". Our models can be used to score if an image-text pair match. Please see our paper for more details. This code release consists of a colab to extract image and language features and input them into our transformer models. Transformer models are stored on tfhub.
Please see the tables below for details of models which we have released via tfhub.
| Name | Training Dataset | ITM | MRM | MLM | Heads | Layers | Att. Type | FineTuned | Notes |
|---|---|---|---|---|---|---|---|---|---|
| data_cc (base) | Conceptual Captions | Classification | Y | Y | 12 | 6 | Merged | N | |
| data_sbu | SBU | Classification | Y | Y | 12 | 6 | Merged | N | |
| data_vg | Visual Genome | Classification | Y | Y | 12 | 6 | Merged | N | |
| data_mscoco | MSCOCO | Classification | Y | Y | 12 | 6 | Merged | N | |
| data_mscoco-narratives | MSCOCO Narratives | Classification | Y | Y | 12 | 6 | Merged | N | |
| data_oi-narratives | OI Narratives | Classification | Y | Y | 12 | 6 | Merged | N | |
| data_combined-instance | All (instance sampling) | Classification | Y | Y | 12 | 6 | Merged | N | |
| data_combined-dataset | All (dataset sampling) | Classification | Y | Y | 12 | 6 | Merged | N | |
| data_uniter-instance | Uniter datasets (instance sampling) | Classification | Y | Y | 12 | 6 | Merged | N | |
| data_uniter-dataset | Uniter datasets (dataset sampling) | Classification | Y | Y | 12 | 6 | Merged | N | |
| data_cc-with-bert | Conceptual Captions | Classification | Y | Y | 12 | 6 | Merged | N | Language initialised with BERT |
| loss_itm_mrm | Conceptual Captions | Classification | Y | N | 12 | 6 | Merged | N | |
| loss_itm_mlm | Conceptual Captions | Classification | N | Y | 12 | 6 | Merged | N | |
| loss_single-modality-contrastive32 | Conceptual Captions | Contrastive | Y | Y | 12 | 6 | Sing. Modality | N | |
| loss_single-modality-contrastive1024 | Conceptual Captions | Contrastive | Y | Y | 12 | 6 | Sing. Modality | N | |
| loss_v1-contrastive32 | Conceptual Captions | Contrastive | Y | Y | 12 | 1 | Merged | N | |
| architecture_heads1-768 | Conceptual Captions | Classification | Y | Y | 1 | 6 | Merged | N | |
| architecture_heads3-256 | Conceptual Captions | Classification | Y | Y | 3 | 6 | Merged | N | |
| architecture_heads6-64 | Conceptual Captions | Classification | Y | Y | 6 | 6 | Merged | N | |
| architecture_heads18-64 | Conceptual Captions | Classification | Y | Y | 18 | 6 | Merged | N | |
| architecture_vilbert-1block | Conceptual Captions | Classification | Y | Y | 12 | 1 | Merged | N | |
| architecture_vilbert-2block | Conceptual Captions | Classification | Y | Y | 12 | 2 | Merged | N | |
| architecture_vilbert-4block | Conceptual Captions | Classification | Y | Y | 12 | 4 | Merged | N | |
| architecture_vilbert-12block | Conceptual Captions | Classification | Y | Y | 12 | 12 | Merged | N | |
| architecture_single-modality | Conceptual Captions | Classification | Y | Y | 12 | 6 | Sing. Modality | N | |
| architecture_mixed-modality | Conceptual Captions | Classification | Y | Y | 12 | 6 | Mix Modality | N | 5 single modality layers and 1 merged layer |
| architecture_single-stream | Conceptual Captions | Classification | Y | Y | 12 | 6 | Single Stream | N | |
| architecture_language-q-12 | Conceptual Captions | Classification | Y | Y | 12 | 6 | Asymmetric (language) | N | |
| architecture_image-q-12 | Conceptual Captions | Classification | Y | Y | 12 | 6 | Asymmetric (image) | N | |
| architecture_language-q-24 | Conceptual Captions | Classification | Y | Y | 24 | 6 | Asymmetric (language) | N | |
| architecture_image-q-24 | Conceptual Captions | Classification | Y | Y | 24 | 6 | Asymmetric (image) | N | |
| architecture_single-modality-hloss | Conceptual Captions | Classification | Y | Y | 12 | 6 | Single modality | N | Includes ITM loss after every layer |
| data-ft_sbu | SBU | Classification | Y | Y | 12 | 6 | Merged | Y | |
| data-ft_vg | Visual Genome | Classification | Y | Y | 12 | 6 | Merged | Y | |
| data-ft_mscoco | MSCOCO | Classification | Y | Y | 12 | 6 | Merged | Y | |
| data-ft_mscoco-narratives | MSCOCO Narratives | Classification | Y | Y | 12 | 6 | Merged | Y | |
| data-ft_oi-narratives | OI Narratives | Classification | Y | Y | 12 | 6 | Merged | Y | |
| data-ft_cc | Conceptual Captions | Classification | Y | Y | 12 | 6 | Merged | Y | |
| data-ft_combined-instance | All (instance sampling) | Classification | Y | Y | 12 | 6 | Merged | Y | |
| data-ft_combined-dataset | All (dataset sampling) | Classification | Y | Y | 12 | 6 | Merged | Y | |
| data-ft_uniter-instance | Uniter datasets (instance sampling) | Classification | Y | Y | 12 | 6 | Merged | Y | |
| data-ft_uniter-dataset | Uniter datasets (dataset sampling) | Classification | Y | Y | 12 | 6 | Merged | Y | |
| architecture-ft_single-modality | Conceptual Captions | Classification | Y | Y | 12 | 6 | Sing. Modality | Y | |
| architecture-ft_single-stream | Conceptual Captions | Classification | Y | Y | 12 | 6 | Single Stream | Y | |
| architecture-ft_language-q-12 | Conceptual Captions | Classification | Y | Y | 12 | 6 | Asymmetric (language) | Y | |
| architecture-ft_image-q-12 | Conceptual Captions | Classification | Y | Y | 12 | 6 | Asymmetric (image) | Y | |
| architecture-ft_language-q-24 | Conceptual Captions | Classification | Y | Y | 24 | 6 | Asymmetric (language) | Y | |
| architecture-ft_image-q-24 | Conceptual Captions | Classification | Y | Y | 24 | 6 | Asymmetric (image) | Y |
In addition to our transformer models, we also release our baseline models. See details of our baseline models in the chart below:
| Name | ITM | Bert Initialisation | FineTuned |
|---|---|---|---|
| baseline_baseline | Contrastive | Yes | N |
| baseline_baseline-cls | Classification | No | N |
| baseline_baseline-no-bert-transfer | Contrastive | No | N |
| baseline-ft_baseline | Contrastive | Yes | Y |
| baseline-ft_baseline-cls | Classification | No | Y |
| baseline-ft_baseline-no-bert-transfer | Contrastive | No |
Installation
You do not need to install anything! You should be able to run all code from our released colab.
Usage
You can run an image and text pair through our module and see if the image and text pair match.
import tensorflow.compat.v1 as tf import tensorflow_hub as hub
m =
hub.Module('https://tfhub.dev/deepmind/mmt/architecture-ft_image-q-12/1')Inference:
output = model.signatures['default'](**inputs)
score = tf.nn.softmax(output['output']).numpy()[0]where score indicates if an image-text pair match (1 indicates a perfect
match). Inputs is a dictionary with the following keys:
-
image/bboxes: Coordinates of detected image bounding boxes. -
image/detection_features: Features from image detector. -
image/padding_mask: Indicator if image features are padded. -
masked_tokens: Text tokens -
text/segment_ids: Indicates sentence segment. (Since we train with one sentencce this will always be 0.) -
text/token_ids: Indicates which words tokens belong to. (We use a tokenizer which can break one word into multiple tokens). -
text/padding_mask: Indicator if text features are padded.
Please see our colab linked for details on pre-processing. You will need to use the detector released in our colab for good results.
Citing this work
If you use this model in your research please cite:
[1] Lisa Anne Hendricks, John Mellor, Rosalia Schneider, Jean-Baptiste Alayrac, and Aida Nematzadeh. Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers, TACL 2021.
Disclaimer
This is not an official Google product.