/VIsION

This aims to provide visually impaired users with an affordable wearable AI solution for their day-to-day activities.

Primary LanguageHTMLGNU General Public License v3.0GPL-3.0

  

Animated VIsION Logo: This is an image of a closed camera lens that opens up to show the word VIsION and closes back again, in a loop.

  VIsION
Wearable AI for the Blind  







Winner Badge, Oct 2019, Awesome Foundation's Awesome Disability Chapter: This red and black badge says Grant Winner, Awesome Foundation and AwesomeFoundation.Org. This project is the October 2019 winner of the Awesome Foundation's Awesome Disability Chapter!

  OCT 2019 WINNER
Awesome Foundation's Awesome Disability Chapter  
( click photo to see this project in Awesome Foundation Disability Chapter's page )





  PROJECT SUMMARY

Latest 3D Printed Prototype: This photo shows VIsION's latest 3D printed prototype.

  This open source, do-it-yourself project aims to
  provide blind and visually impaired users with   
  a sustainable, fully customizable solution for their indoor and outdoor day-to-day activities ...  

  PROJECT STATUS

1st 3D Printed VIsION AI Glasses Prototype: This photo shows Marx Melencio wearing his first 3D printed prototype of the VIsION AI glasses for the blind. Click this photo to watch Marx using this prototype in action.
      Click This Photo to Watch Live Video Demo of Our 1st 3D Printed Prototype  

  I'm Marx, a completely blind developer. And I created this preliminary prototype.  
  This is a pair of 3D printed eyeglasses with electronic components. And it's powered by ML and DL models.  
  I call it VIsION AI Glasses for the Blind ... 

This photo shows blind user wearing raw VIsION AI glasses for the blind prototype with wireless clicker. Clicking this image link will open a new page and take you to a 90-second video presentation of the VIsION AI Glasses for the Blind on Youtube. This is for San Francisco Lighthouse for the Blind's global 2019 Holman Prize competition.
      Watch Live 90-Second Video Demo of Raw VIsION AI Glasses for the Blind

  This video is for San Francisco Lighthouse for the Blind's annual Holman Prize 2019 competition.  
  And this project was still in its infancy at that time ...  

This photo shows VIsION AI Glasses recognizing objects and sensing the distance of the central object. Clicking this image link will open a new page and take you to the concept video of the VIsION AI Glasses for the Blind on Youtube.
      Watch Concept Video of VIsION AI Glasses for the Blind

  This video shows the earliest target objectives of this project.  

  PROJECT REQUIREMENTS

  

  • SBC:  Tested on Intel Compute Stick (model with CORE M3 CPU), Liva Q, Raspberry Pi Zero WH, 3B+ and 4 model B
    •   
  • PLATFORM:  Tested on Microsoft Windows 10, Raspbian Stretch and Buster, Ubuntu 1809 & Android X86
    •   
  • PYTHON:  Tested on version 3.5 and 3.6
    •   
  • EXTERNAL HARDWARE:  Tested with USB 2.0 and 3.0 UVC 8Mp camera board with autoFocus Lens and built-in mic, Devantech SRF10 ultrasonic ranger, USB-ISS and GPIO-to-USB board, generic GPS module, bone-conduction speaker pads and stereo amplifier board, mini vibrating disk and DRV2605 microcontroller, USB soundcard, 4-port USB hub, generic programmable Bluetooth button and LiFePo4 batteries

  QUICK START

  

  1.  Install FFMPEG and Screen-Capture-Recorder (used for video and audio recording modes), ImageMagick for Linux or Microsoft Windows 10 (used for OCR image post processing), Skype v8.53.0.85 for Microsoft Windows 10 (used for manual visual assistance mode, currently working on video calling for Linux), Mozilla Firefox v68.2.0 and a few other tools for Linux (follow comments in vision.py file);
    2.   
  2.  Open Firefox. Go to about:config, search and set media > navigator > disabled to True, and toolkit > startup > max_resumed_crashes to a fairly large integer. Afterwards, install the Auto Fullscreen extension;
    3.   
  3.  Make sure the tools above are in the system path;
    4.   
  4.  Clone this repository ( git clone https://github.com/MXGray/VIsION.git );
    5.   
  5.  Go to cloned repo ( cd /path-where-you-cloned-this-repo/VIsION );
    6.   
  6.  Run python -m pip install -r requirements.txt;
    7.   
  7.  Inside your cloned repo's root directory, you'll see the freeware SeeingWithSound© vOICe offline Web app as a complete HTM directory with an HTML file. The vOICe Java applet / application is written by Peter B.L. Meijer. You can open Firefox and go here to install this freeware as an offline Web app. And when you want to use this freeware with this 3D printed eyeglasses, then don't forget to update the necessary directory and HTML filename in the vision.py file;
    8.   
  8.  Download a pre-trained Caffe model here, store it in multiobj/ade20k/ and rename the caffemodel and prototxt files ( should be dn.caffemodel and dn.prototxt );
    9.   
  9.  Download a pre-trained Mask-rCNN model here and store it inside multiobj, along with pre-trained Caffe models for face detection, age and gender estimation (folders inside multiobj are labeled);
    10.   
  10.  OR, just use re-trained models (via transfer learning) bundled with this repo, except for Ade20K and Mask-rCNN ( download here and here, as they're too big for Github, put dn.caffemodel inside multiobj/ade20k and navnet.pb in multiobj );
    11.   
  11.  NOTE: You can download here both re-trained models, along with FFMPEG 4.1.1 and Skype 8.53.0.85 ( already included in this repo are other required Windows tools ); and
    12.   
  12.  Finally, run python vision.py ... 

  NOTE:  3D printing source files are included in this repo ...  

  HOW IT WORKS

  

  1. 3D SOUNDSCAPE:  Freeware SeeingWithSound© vOICe using bone conduction speaker pads) for "visual" interpretation of scenes and objects through trained neural auditory synthesis for those who became blind from the age of 7 and above;
    2.   
  2. PRE-TRAINED ML & DL MODELS:  Local and server-hosted machine learning and deep learning models For describing visual scenes and objects with its clockface locations, for converting text in physical objects like books and signboards, for detecting persons and their current actions along with estimating their age and gender, and 10x much faster processing when online;
    3.   
  3. DISTANCE SENSING:  Audible & Haptic feedback when detecting distance of objects within user's central view (up to 20 feet);
    4.   
  4. MANUAL VISUAL ASSISTANCE:  On-demand access to a remote crowdsourced team of work-at-home sighted persons with disabilities like mobility-impaired individuals who as pro agents will provide manual visual assistance services for a reasonable per-minute fee (helpful during complex situations);
    5.   
  5. WEARABLE COMPUTER:  Use it for your productivity or recreational needs. Install any compatible software for your preferred platform, including a screenreader. Tested on full Windows 10, Linux & Android X86;
    6.   
  6. TURN-by-TURN GPS NAVIGATION (to-do):  Offline audible GPS location data;
    7.   
  7. DO-IT-YOURSELF:  Anybody can build, customize and extend the features of this tech; and
    8.   
  8. OPEN SOURCE (GNU GPLv3 license):  With the exception of packaging any work that derives or is adapted from this project as proprietary or closed source, anybody can modify, distribute, sell, freely give away or adapt from this tech for any other use case or for creating new solutions for other groups, i.e. For deaf users (convert camera input of hand gestures to text, convert voice input and camera input of lip movement of others around the deaf person into text and display by micro projector, manual remote interpreting assistance, etc.), or for non-disabled users (wearable computer with micro projector display or oLED lens, camera glasses, Bluetooth glasses for calls and audio playback, etc.) ...

  PROJECT PHOTOS

  
RAW PROTOTYPE 3D PRINTED PROTOTYPE
Front (Worn by User
Front and center photo of blind user wearing VIsION glasses with labels of already tested 8 megapixel camera, noise-cancelling mic and sonar sensor.		 Front & Center (Worn by User
Front and center photo of blind user wearing preliminary 3D printed VIsION AI glasses for the blind prototype with 8 megapixel camera, noise-cancelling mic, sonar sensor and bone-conduction speakers.
Front Right (Worn by User)
Front right photo of blind user wearing VIsION glasses with labels of already tested 8 megapixel camera, noise-cancelling mic, sonar sensor and wireless earbud.		 Right View (Worn by User)
Right view photo of blind user wearing preliminary 3D printed VIsION AI glasses for the blind with 8 megapixel camera, noise-cancelling mic, sonar sensor and bone-conducting speakers.
Top (RPi Version)
Full top photo of VIsION glasses (RPi version) with labels of already integrated 8 megapixel camera, noise-cancelling mic, sonar sensor, wireless earbud and tiny single board Raspberry Pi Zero W computer, along with labels of soon-to-be implemented haptic discs, additional sonar sensors and bone-conducting speaker modules.		 Back View (Worn by User)
Back view photo of preliminary 3D printed VIsION AI glasses for the blind with LiFePo4 battery-powered single board computer, GPS module, stereo amplifier and haptic microcontroller.
Top (ICS Version)
Full top photo of VIsION glasses (ICS version) with labels of already integrated 8 megapixel camera, noise-cancelling mic, sonar sensor, wireless earbud and small Intel Compute Stick pocket PC, along with labels of soon-to-be implemented haptic discs, additional sonar sensors and bone-conducting speaker modules.		        
Outer Front
Front and center photo of VIsION glasses with labelled electronic components.		        
Inner Front Side
Inner front side photo of VIsION glasses with labelled electronic components.		        
Outer Left Arm
Outer left arm photo of VIsION glasses with labelled electronic components.		        
Inner Left Arm
Inner left arm photo of VIsION glasses with labelled electronic components.		        
Outer Right Arm
Outer right arm photo of VIsION glasses with labelled electronic components.		        
Inner Right Arm
Inner right arm photo with labelled electronic components.		        
Bone-Conducting Speaker Placement
Left and right arm photo of VIsION glasses with labelled placement areas for pair of tiny bone-conducting speaker modules.		        

  VALIDATION PROCESS

  

  1. CONCEPT VALIDATION:  The Department of Science & Technology of the Philippines gave us a 2-year R&D software development grant for our project. This was after we presented our proof of concept prototype to their panel of scientists, engineers and business professionals in 2017;
    2.   
  2. TECH VALIDATION:  Entered the earliest version of our prototype to the 2019 Microsoft HackerEarth Azure AI Hackathon. Our project was hand-picked for the finals round, out of more than 760+ professional dev't teams in New Zealand, South Korea and the rest of South East Asia. This was after their panel of software engineers inspected our prototype; and
    3.   
  3. NICHE MARKET VALIDATION:  We entered our project to San Francisco Lighthouse for the Blind's 2019 Holman Prize competition. Our project was inspected by a panel of esteemed blind and visually impaired judges. They hand-picked our project for the semi-finals round, out of over 100 entries from across 6 continents ...

  Sept 15 2019 Update:  Mainly focused on hardware assembly and 3D printing for the past couple of months. Will update this repo with latest software code, hardware source files and circuitry schematics once hardware prototype design has been finalized for both plastic injection molding and 3D printing ...