This repository contains the code for an image captioning project that utilizes a combination of Convolutional Neural Networks (CNNs) and Transformers for generating captions from images. The goal of the project is to automatically generate descriptive and contextually relevant captions for input images.
The project consists of the following components:
Images are loaded using the OpenCV library, which allows for efficient image handling and manipulation. Preprocessing techniques such as resizing, normalization, and data augmentation can be applied to enhance the input images.
A pre-trained CNN model, such as ResNet50, is used to extract meaningful features from the input images. These features capture high-level representations of the visual content and serve as a foundation for generating accurate captions.
The extracted image features are passed through a Transformer-based language model, such as GPT-2, which generates captions based on the learned visual-textual associations. The language model is fine-tuned on captioning datasets to improve its caption generation capabilities.
Various evaluation metrics, such as BLEU (Bilingual Evaluation Understudy) and CIDEr (Consensus-based Image Description Evaluation), are employed to assess the quality and accuracy of the generated captions. These metrics provide quantitative measures of how well the model performs in capturing the essence of the images.
- https://huggingface.co/nlpconnect/vit-gpt2-image-captioning
- GPT-2: Language Models are Unsupervised Multitask Learners
- BLEU: A Method for Automatic Evaluation of Machine Translation
- CIDEr: Consensus-based Image Description Evaluation