hemanthbd/Morse-Diffusion

Morse Code meets Stable Diffusion via Pose Estimation

Morse-Diffusion

This project aims to help people with various hand-tremor disabilties as they wouldn't be able to utilize an entire keyboard, when instead one of the adaptive tools used is Morse Code as it's a single-switch alternative. In that respect, Morse Diffusion uses Morse Code to generate Art via the Keras Stable Diffusion API.

This is done in 3 ways-

1) Typing Morse Code

Here, the user can type the prompt with Morse Code and the script generates the corresponding images via the Diffusion API. For people with spinal muscle atrophy, they can't use the below methods well, so this would be better alternative.

2) Head Movements Morse Code

Generally, even the above way could prove to be cumbersome to many who can't use their hand-motor functions well, in that effect, they could make use of the movements of their head to convey the same morse code prompt, which will then generate images similarly.

Instructions:

* Moving your head- 

  * Left enters .(Dit)

  * Right enters .(Dah)

  * Up remove the previous Dit/Dah

  * Down (Short) enters the respective letter after completing the Morse Sequence for each letter

  * Down (Long) enters Space ('\') between words (There will be a countdown from 40 till 'Space Entered!')

  ** IMPORTANT AFTER FINISHING Morse Coding the Prompt : Make sure to do .-.-.- HeadDown (Left Right x3 HeadDown(Short/Long)) , which will close the webcam window and trigger the prompt to the Keras Stable Diffusion API. .-.-.- is a Full-Stop in Morse **
  
  * Click ESC to exit at any time if not satisfied with the prompt, it will exit the application without triggering the API

3) Eye Blinking + Head Movements Morse Code

Often, head movements can get tiring for longer sequences, which is why here, the user can use head movements combined with eye blinking to generate the morse code prompt

Instructions:

* Short Pause Blink enters .(Dit)

* Longer Pause Blink enters .(Dah)

* Head Up remove the previous Dit/Dah

* Head Down (Short) enters the respective letter after completing the Morse Sequence for each letter

* Head Down (Long) enters Space ('\') between words (There will be a countdown from 40 till 'Space Entered!')

* MPORTANT AFTER FINISHING Morse Coding the Prompt : Make sure to do .-.-.- HeadDown (ShortPauseBlink LongPauseBlink(x3) HeadDown(Short/Long)) , which will close the webcam window and trigger the prompt to the Keras Stable Diffusion API. .-.-.- is a Full-Stop in Morse

* Click ESC to exit at any time if not satisfied with the prompt, it will exit the application without triggering the API

0) Yes, Voice (But someone had already implemented a good Whisper in Hugging Face, so didn't want to do the same)

Examples!

(I apologize, the Keras Stable Diffusion time for generating 1 image is 5-7 minutes as I am working on my personal laptop which has only CPU, if you device has a GPU, it will take seconds to generate it) (I couldn't use Colab GPU as the Webcam + Mesh was causing the frame-rate to drop/lag, will work on that later on)

Head Movements Morse Code

a) dog eating spaghetti

Dog.Eating.Spaghetti.mp4

b) cat fighting dinosaur

Cat.fighting.a.dinosaur.mp4

Eye Blinking + Head Movements Morse Code

a) dolphin fighting an owl (This is funny because it seems that either an owl dressed by like a dolphin (wink) or that perhaps it needed more information about an owl)

dolphin_fighting_owl.mp4

Running the Scripts!

Prerequiste Libraries

 pip install mediapipe opencv-python
 pip install --upgrade keras-cv

Script Help

1) Typing Morse Code

 py main.py -t "-.. --- --. / . .- - .. -. --. / ... .--. .- --. .... . - - .." --head_morse False

2) Head Movements Morse Code

 py main.py --head_morse True

3) Eye Blinking + Head Movements Morse Code

py main.py -eyehead True --head_morse False

TO DO

• Can pop out previous dit/dah of current letter but not entire previous letter (as frame-rate drops), so need to fix that
• Automatic text-resizing on screen for longer message than frame width
• Head Movement sensitivity control w.r.t frame-rate
• Streamlit? this into a GUI with parameters to control on-fly
• Colab integration