Remed-E: Reminding Medicines Everyday

This project aims at providing an ecosystem of intelligent and user-friendly services based on modern samrt home devices and other technologies that can help patients adhere better to their medicines and become more self-aware about their side effects and usage.

The project also includes a subscriber-based mechanism which will provide relatives and doctors quick and easy access to patient's medical history in cas of emergency.

Technologies Used:

A Java/Kotlin based Android App as the frontend
Google Firebase Realtime Database for backend
Deep Learning based Optical Character Recognition with OpenCV & Java
Custom built Alexa Skill in python for Amazon Echo Device and other Amaxon Alexa based devices.

Data Flow Diagram:

The App:

The Alexa Skill:

The alexa skill is a custom-hosted skill built with python using ask-sdk-core and flask-ask-sdk(to route the requests sent by alexa to our skill using a flask app). The endpoint for the skill is obtained by deploying it with ngrok. The skill makes use of firebase_admin python module to get data from firebase backend. The interaction model for the alexa skill including all the intents, their utterances, slots, slot types and dialog model can be found in remede-alexa-skill/assets/interaction_model.json

Skill Setup Instructions:

Create a new skill on amazon developer console and use the assets/interaction_model.json to create the interaction model for the skill using the JSON Editor.
Create a new python virtual environment, navigate to remedE-alexa-skill directory and install all the dependencies with pip install -r requirements.txt
Run the remede-skill.py file which runs as a flask app on port 5000 on your localhost.
Download ngrok and run it with protocol set to http and port number 5000.
Copy the generated url and paste it in the https endpoint for the skill and then you are ready to test the skill.

How the Skill Works:

LaunchRequest Intent: The LaunchRequest Intent invokes when the user says "Launch Remedy Helper". This intent is handled by LaunchRequestHandler which returns a JSON response to alexa cloud with a welcome message including a brief description of the skill and how to begin.
LoginIntent: The LoginIntent invokes when the user asks the skill to "login for user USERNAME" or "provide medical adherence for USERNAME". This intent is handled by LoginIntentHandler which fetches the user key from the realtime database using the Utils.get_user_key() method. This method then stores the user name and key in json file for later reference. The handler also checks for existing medicine details using the Utils.get_user_medicine_details() method. If medicines are found in the user's data, then the skill returns JSON response to alexa cloud with a welcome message and a question asking permission to set reminders for all the medicines else the skill returns a response with just a welcome message.
AMAZON.YesIntent: The Amazon.YesIntent is built-in intent which is invoked when the user says "Yes" or its synonyms. This intent is handled by the CreateMedicineReminderHandler which makes an API call to the alexa Reminders API to set daily reminder for all the medicines at the times entered by the user. The skill then returns a JSON response with a confirmation message which then spoken by alexa device.
GetMedDataIntent: The GetMedDataIntent is invoked when the user asks the skill to "give a brief description of MEDANME" or "what is MEDNAME used for?". This intent is handled by the GetMedDataIntentHandler which recieves the medicine name as med_name slot value from the JSON request recieved by the skill. The handler then calls the Utils.get_med_json_data() to get the "generic_name" and "description" from the assets/med_data.json. The skill then prepares a speech output string for alexa and returns a JSON response to alexa cloud which is then spoken by alexa.
GetSideEffectsIntent: The GetSideEffectsIntent is invoked when the user asks the skill "what can be the side effects fo MEDNAME". This intent is handled by the GetSideEffectsIntentHandler which recieves the medicine name as med_name slot value from the JSON reques recieved by the skill. The handler then calls the Utils.get_med_json_data() method to get the "side_effects" from the assets/med_data.json for that medicine. The skil then prepares a speech output string for alexa using the side effects and returns a JSON response to alexa cloud which is then spoken by alexa.
GetNextDoseIntent: The GetNextDoseIntent is invoked when the user asks the skill "when is the my next dose of MEDNAME". This intent is handled by the GetNextDoseIntentHandler which recieves the medicine name as med_name slot and the calls the Utils.get_next_dose() method. This method then uses the Utils.get_user_med_data() method to get user medicine details from firebase realtime database. From the recieved medicine data, the method then finds closest dose time to the current time for which the user has not taken the medicine and returns it to the handler. The handler then creates a speech output string and returns the JSON response.
GetRemainingStockIntent: The GetRemainingStockIntent is invoked when the user asks the skill "which of my medicines are running out" or "which of my medicines are about to finish". This intent is handled by the GetRemainingStockIntentHandler which calls the Utils.get_remaining_stock() method which in turn uses the Utils.get_days_left() method to calculate the number of days in which a particular medicine will run out. The handler then creates an ouput speech which tells the user the number of days all the medicines will last and if any of the medicines will last for only 3 days or less, then those medicines are suggested for reordering to the user. The skill then uses "Intent Confirmation" using DelegateDirective to ask if alexa should place an order for the medicines. Depending on the user's response, if intent.confirmation_status of the request is equal to IntentConfirmationStatus.CONFIRMED, this intent is chained to ReorderMedicinesIntentHandler. Else if intent.confirmation_status in the request is equal to IntentConfirmationStatus.DENIED, this intent is chained to AMAZON.StopIntent. The skill then returns the appropriate intent as a DelegateDirective object with the help of add_directive() method in the response.
ReorderMedicinesIntent: This intent does not contain any sample utterances in the skill interaction model, but is chained to GetRemainingStockIntent and is invoked when the user replies in affirmation to the its Intent Confirmation. This intent is handled by the ReorderMedicinesIntentHandler which returns the order placement confirmation message and sends a card to the alexa mobile app of the user showing the deatils of order. (Currently this handler does not actually place any orders due to lack of availablity of any trusted online medicines ordering API)
AMAZON.StopIntent: This intent is invoked whenever the user says "stop" and is also chained to the GetRemainingStockHandler through which it is invoked when the user denies its Intent Confirmation. It is handled by the CancelOrStopIntentHandler which just returns a goodbye message to the alexa cloud which is then spoken by the alexa device.