This project contains an implementation of VSE++ losses by [Faghri, Fartash et al., 2017] that is a technique for learning visual-semantic embeddings for cross-modal retrieval, and an implementation of t-SNE by [van der Maaten et al., 2008] (school project, Signal Learning and Multimedia class, 2019).
It is applied on MSCOCO image captioning dataset by [Lin, Tsung-Yi et al., 2014], in particular with the val2014 data which contains a set of 40k images annotated with five captions each. We also used Resnet50 features by [He, Kaiming et al., 2016] and glove embeddings by [Pennington, Jeffrey et al., 2014].
A good introduction of Representation Learning would be [Bengio, Y. et al., 2013].
- Image retreival
- Caption retreival
- VSE++ losses (Loss-Sum-Hinge and Loss-Max-Hinge)
- t-SNE for captions in a 2D scatter
- t-SNE for captions in 3D scatter
- t-SNE for images in a 2D grid
- t-SNE for images in a 2D scatter
- t-SNE for both captions and images in a 2D scatter
It requires python3, python3-pip, the packages listed in requirements.txt and a recent version of git that supports git-lfs.
To install the required packages:
pip3 install -r requirements.txtA notebook is available, and each feature is illustrated in an example in test directory.
-
[Faghri, Fartash et al., 2017] Faghri, Fartash et al. “VSE++: Improving Visual-Semantic Embeddings with Hard Negatives.” BMVC (2017).
-
[van der Maaten et al., 2008] van der Maaten, L.J.P.; Hinton, G.E. (Nov 2008). "Visualizing Data Using t-SNE" (PDF). Journal of Machine Learning Research. 9: 2579–2605.
-
[Lin, Tsung-Yi et al., 2014] Lin, Tsung-Yi et al. “Microsoft COCO: Common Objects in Context.” Lecture Notes in Computer Science (2014): 740–755. Crossref. Web.
-
[Bengio, Y. et al., 2013] Bengio, Y., A. Courville, and P. Vincent. “Representation Learning: A Review and New Perspectives.” IEEE Transactions on Pattern Analysis and Machine Intelligence 35.8 (2013): 1798–1828. Crossref. Web.
-
[He, Kaiming et al., 2016] He, Kaiming et al. “Deep Residual Learning for Image Recognition.” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016): n. pag. Crossref. Web.
-
[Pennington, Jeffrey et al., 2014] Pennington, Jeffrey & Socher, Richard & Manning, Christoper. (2014). Glove: Global Vectors for Word Representation. EMNLP. 14. 1532-1543. 10.3115/v1/D14-1162.
- Charly Lamothe
- Guillaume Ollier
- Balthazar Casalé