/Convo_Sense_Public

This is the Senior Design Project. Convo-Sense is a machine learning model that analyses conversation (completely offline) in real time and picks out possible unknown words depending on user profession.

Primary LanguagePython

Team: Anthony Carneiro, Sohan Show, Jessica Peters, David Price, Professor Tilman Wolf

Problem Statement:

Everyone has been in a situation where either you or the person you are talking to needs to interrupt the conversation in order to look something up. It breaks the train of thought, causes you to divert attention away from the speaker, and in some cases can be rude. Or maybe you cannot keep up with a lecture, and end up losing valuable intuition about a subject simply because you do not understand a term.

Therefore, we decided to develop a system to mitigate this problem and automate the entire reference searching and note taking process during a conversation. Convo-sense listens to conversation between two peole and does offline analysis to find out relevant context and keywords of the conversation. This preserves user privacy and automates reference searching and note taking for the user.

Specifications:

The system should be able to:

  • Detect conversation from up to 6 feet away.
  • Audio signal converted to digital text with 95% accuracy from close range and 87% accuracy from 6 feet away.
  • Capture more than 110 words per minute.
  • Support four user professions (Engineer, Medical, Financial, and Law).
  • Display reference material in less than 6 seconds from the time of speech.
  • Is Compatible with any system so long as there is an internet connection and microphone support.
  • Run on a microcontroller.

How to run on any micro-controller:

There are a few things that you need to make sure before starting with the testing:

  • Make sure that the micro-controller runs Linux or any OS that supports python3.
  • Make sure you have your API keys ready for openai and GoogleImageSearch for the reference script.

Once you have all of these:

  1. git clone https://github.com/sohanshow/Convo_Sense_Public.git into a valid dir in your OS.
  2. cd into the dir and initiate python3 app.py

References and Acknowledgements: