FrameNetBrasil/charon_gsoc2020

A tool for preprocessing videos for multimodal annotation

This project was bootstrapped with Create React App.

The repository contains code for a multimodal annotation preprocessing tool built using React and PHP.

GSoC report links:
First Evaluation Report: https://github.com/FrameNetBrasil/webtool/blob/gsoc2020_2/GSoC2020/weekly_reports/FirstEvaluationReport_idea2.md
Second Evaluation Report: https://github.com/FrameNetBrasil/webtool/blob/gsoc2020_2/GSoC2020/weekly_reports/SecondEvaluationReport_idea2.md
Final Report: https://medium.com/@prishita.r85/a-new-frame-based-image-and-video-annotation-pipeline-my-gsoc-20-project-6daa9fe5f391

This distribution is running as a Docker container. In order to run this container you'll need docker installed:

• Docker Windows
• Docker OS X
• Docker Linux

Usage

• Clone this repository at an accessible folder

$ git clone https://github.com/FrameNetBrasil/charon.git

• Download the trained model file from this google drive link, and save it in the public/server folder https://drive.google.com/file/d/1iw7VIoBspqgChl-4nDU809A6rvwZqKk5/view?usp=sharing

• Start the container

$ cd charon/devops/charon
$ docker-compose up

• Build frontend app

$ cd charon
$ docker exec -it fnbr-webserver-charon bash

// inside the container

# npm install
# npm run build

• Access the app at http://localhost:8600

• To start the flask server follow these steps inside the root directory of the container:

$ cd public/server
$ apt-get install python3.7
$ apt-get install python3-pip python3-dev python3-venv
$ python3 -m pip install --upgrade pip 
$ python3 -m pip install numpy matplotlib keras opencv-python scikit-learn scikit-image scipy argparse mysql-connector-python Pillow pickle-mixin glob3 flask opencv_contrib_python mpld3 moviepy
$ python3 -m pip install flask_cors, requests
$ apt update && apt install -y libsm6 libxext6
$ apt-get install -y libxrender-dev
$ python3 -m pip install --upgrade pip 
$ python3 -m pip install tensorflow==2.2.0 --no-cache-dir
$ python3 -m pip install elementpath
$ apt-get install python3-tk 
$ python3 track_objects.py

The server will start running on http://127.0.0.1:5000/

Container Parameters

Parameters are available to the container (mainly the webserver port) at

/charon/devops/charon/.env

Also adjust React env at (baseURL for API)

/charon/.env

Note: Since frame element updates are not being made currently in the ObjectMM table, to avoid referential integrity errors, temporarily remove the foreign keys by executing the following sql commands in the ObjectMM table from phpmyadmin:

ALTER TABLE objectmm DROP FOREIGN KEY fk_ObjectMM_AnnotationSetMM
ALTER TABLE objectmm DROP FOREIGN KEY fk_ObjectMM_FrameElement

Built With

• PHP 7.4
• MariaDb 10.4
• PhpMyAdmin 5.0.1
• React
• IBM Cloud Speech-to-Text API Service Credentials (User Account- Prishita Ray):

{ "apikey": "jrdLqCqvqz9JU8Eu8Ls7c40_uXTmCFrb3iWbLk77KgvJ", "iam_apikey_description": "Auto-generated for key d372a095-0998-4f7f-8557-afcf7301ca4a", "iam_apikey_name": "Service credentials-1", "iam_role_crn": "crn:v1:bluemix:public:iam::::serviceRole:Manager", "iam_serviceid_crn": "crn:v1:bluemix:public:iam-identity::a/6de8c018fee0fb1865ab45744f2f9483::serviceid:ServiceId-fac502ab-3ee9-44df-b3d5-bcb2158a69cd", "url": "https://api.us-south.speech-to-text.watson.cloud.ibm.com/instances/0c057c58-d790-44ba-97b5-2094825614fa" }

Service Access URL: https://stream.watsonplatform.net/speech-to-text/api/v1/recognize?end_of_phrase_silence_time=2.0&split_transcript_at_phrase_end=true&speaker_labels=true