Therapy Aid Tool

A Web App to aid in clinical and therapeutic decisions reducing analysis time for therapists

(The current version of this project aims to assist in decision making of Autism Spectrum Disorder (ASD) video sessions)

How this app can help in treatment?
Publications
Quick demonstration video
Current technologies and heuristics used
App Overview
Getting Started

💭 How this app can help in treatment?

The main goal is to provide a web aplication that is capable of filtering out relevant scenes of recorded therapy sessions to assist professionals in their analysis. With the use of Artificial Inteligence and Computer Vision this app is capable of:

Handle large amounts of therapy session videos, storing and retrieving them as needed.

⚡ Quickly store and access sessions with patient name and date.
Filter relevant scenes according to medical professionals.
- Point exactly in time where is happening an event of interest in the video.
  
  ⚡ YouTube-like timeline of interactions
- Reduce time for analysis of a whole video.
  
  ⚡ In this test set we got an analysis time reduction from 9% to 90%.
  ⚡ It means that a 60min therapy session can be converted to a 54min to 6min of manual analysis.
  ⚡ This can be achieved by balancing Precision x Recall through the changing of internal configurations.
Assistance in clinical and therapeutic decisions throughout interactive search and statistical reports.
- Summary of interactions in each session.
- Progress of interactions for all sessions.
  
  ⚡ Getting sense of the patient's progress through interactions can aid in decision making.

📰 Publications

Publication 1 (BRACIS Paper - Springer): Event Detection in Therapy Sessions for Children with Autism
Publication 2 (Computer on the Beach Periodical - Free/Open): Event detection in therapy sessions for children with Autism
Undergrad Thesis (Free/Open - pt-br): A Tool to aid Autism Spectrum Disorder therapies using Machine Learning

▶️ Quick demonstration video

https://www.youtube.com/watch?v=S3w9jncUsQc

📝 Current technologies and heuristics used

Core

For the task of object detection we use a YOLOv5 trained deep learning model. YOLO detects where the main actors of a therapy session are by generating bounding boxes (bboxes) around each one of them. For this current version, the main actors of a ASD therapy session are the Toddler, Caretaker and the interactive teddy bear called PlusMe.
These bboxes provide actors' location in the frame, and through the level of intersection (a flavor of IoU) of these bboxes we get a sense of how close they are to each other (closeness) and predict if an interaction is occuring.
The interactions for a therapy session generate quantitative results like:
- How many times happend a certain interaction.
- Duration of the interactions.
- The total time of interactions (also minimun, maximum and mean time).

App

Currently we use the Streamlit Framework to glue together this detections and processings, this database in SQLite and this CRUD operations. With Streamlit's easy to use aproach you can quicly spin up your own local server and render your app.

Trained models

We trained several YOLOv5 models in our sibling repo therapy-aid-nn. You can find the models and a sample video in the v1.0.0 release. The default for this app is the small model on a 256x256 image size.

🔎 App Overview

When the user uploads a video of a therapy session, YOLO do its magic and we generate statistical reports and a timeline of interactions in a "YouTube" style. Then the user can add the processed session to the database by providing a patient name and date. With more than one session added into the database for a patient, reports of progress are made.

👋 Getting Started

You're gonna need Python (3.8 or higher), Pip, Git (if you're gonna clone the repo) and a Chromium-based Web Browser.

⚠️ Use Chromium-based browsers. In this v1.0.0, the timeline rendering occurs bellow the video playback, and not inside above the loading bar like YouTube does. Streamlit in Chromium-based browsers create the loading bar accross the whole video window, so the rendered timeline can be nicelly stretched. This is a visualization issue for Firefox for example, where the loading bar is just a section of the whole window lenght.

⚡ To perform detection tasks we recommend a GPU with more than 4Gb VRAM

Installation

Clone this repo (or download the zip file):

git clone https://github.com/solisoares/therapy-aid-tool.git

Then install requirements:

cd therapy-aid-tool
python -m venv venv
source venv/bin/activate
pip install -e .

Default Usage

This will run the app with the small YOLO model that can be used with a decent cpu.

streamlit run 👋_Welcome.py

⚡ The first time you make a detection in the app, it will try to download the specified weights in the detect.cfg file from the therapy-aid-nn v1.0.0 release.

This will start a local Streamlit server and the app will open in a new tab in your default web browser (we recommend Chromium based browsers).

Custom Usage

To change which YOLOv5 model and image size you want to make detections with, adjust the detect.cfg file accordingly.

This file specifies the location of the model weights and the image size (it's recommended that the size is the same one YOLO was trained). You can generate new weights for your dataset or use the ones in the therapy-aid-nn v1.0.0 release (they are commented in the detect.cfg file)

In short, do this:

Put your weights in a folder of your choice (our pre trained models are here)

In the therapy_aid_tool/detect.cfg set:

[yolov5]
weights=<your-weights-path>
[model]
size=<img-size>

solisoares/therapy-aid-tool