Submitted: 14/02/2024
Created by Benjamin, Camilla and Tobias.
Create something innovative using SBERT and semantic search
The dataset used for this project is various GIF's and its description with 125k datapoints. Can be found here.
-
Feature Selection: Identify relevant features.
-
Feature Engineering: Conduct any necessary feature transformations. Reducing Dataset to 20k data rows.
-
Modelling/Embedding: Load the CLIP model and make embeddings on the dataset.
-
Defining the semantic search function: search function to return GIF url, description and cosinus score
-
Make Application: Making an application with the Gradio interface.
A functional Gradio application that gives the approapriate GIF to the text that you write. The app can be seen here.
-
Gradient Descent Exercise: Execute the process of updating weights for two examples using Stochastic Gradient Descent (SGD). Document each step, including input calculation, prediction, loss assessment, weight adjustments, and updates.
-
Attention Mechanism Exercise: Implement the attention mechanism on two distinct sentences. Choose sentences with polysmous words to demonstrate its functionality effectively.
The two distinct sentences presented are: sentence1 = "She saw a bat flying in the night sky" sentence2 = "He used a bat to hit the baseball during the game"
-
Load Model: Using SBERT to solve this task
-
Embedding: Compute embedding and encoding the sentences
-
Compute attention scores and context vector
-
Evaulating: Plotting a heatmap and evaulating the score
We see that the model finds a strong correlation between the words 'bat' and 'baseball' in sentence 2, where it struggles a bit more in sentence 1. Here the strongest corralations with the word 'bat' is in the words 'sky' and 'she'. So the model misunderstands the context a bit in sentence 1.