/VN_Multi_User_Video_Search

BED-AIO team code for AIChallenge2023

Primary LanguageJupyter Notebook

Multi-User Video Search: Bridging the Gap Between Text and Embedding Queries

Khai Trinh Xuan, Nguyen Nguyen Khoi, Huy Luong-Quang, Sang Hoa-Xuan, Anh Nguyen-Luong-Nam, Minh-Hung An

SOICT 2023 [Paper]

Pipeline

image

Dataset preparation

Dataset structure:

|- dict 
   |- ...
   |- faiss_clip_cosine.bin
   |- faiss_clipv2_cosine.bin
|- frontend
   |- ai
   |   |- public
   |   |   |- data
   |   |   |   |- KeyFrames
   |   |   |   |   |-L01
   |   |   |   |   |-L01_extra
   |   |   |   |   |-....

Dict

Download dict zip file: dict

Vector embeddings

Download bin file:

Keyframes

Download keyframes zip file and extract to folder frontend/ai/public/data.
Data part 1:

Data part 2:

Data part 3:

Raw video from AIChallenge 2023

Data part 1:

Data part 2:

Data part 3:

Dataset extraction

Detailed on dataset extraction: data

Installation

  • Backend

conda create -n AIChallenge2023
conda activate AIChallenge2023
pip install git+https://github.com/openai/CLIP.git
pip install -r requirements.txt

  • Frontend

Install nodejs: https://nodejs.org/en/download

npm install

  • DB Sever

pip install flask
pip install flask-cors
pip install flask-socketio
pip install pyngrok==4.1.1
ngrok authtoken your_token # Add your ngrok authentication

Usage

It is recommended to configure the environment using Anaconda. Linux support only.

  • Backend

Using local machine, from root of repo:

python3 app.py

Using colaboratory, run appNotebook (App section) for starting the backend.

  • Frontend

Change url in frontend/ai/src/helper/web_url.js.

cd frontend/ai/
npm run dev

  • DB Sever

Open 2 terminal and run:

python appStorage.py

ngrok http 5000

  • Interface

image