Web application solution aimed to deal with multifaceted problems posed by the Covid-19 Crisis.
- Overview
- Video
- The idea
- How it works
- Documents
- Datasets
- Technology
- Running the App
- Getting started
- License
In times of crisis, there are multiple issues which affect the community and we're trying to solve a few of them. As the number of Covid-19 cases continue to rise, the number of people seeking treatment will overwhelm many hospitals. Having a handle on local hospitals’ capacity and resource availability could help balance the load of Covid-19 patients requiring hospitalization across a region by allowing medical staff to send a patient to a facility where they are more likely to be treated quickly. But many states lack real-time data on their current capacity to treat Covid-19 patients.
Another issue we'll be witnessing with the relaxation of lockdown restrictions is when people will be stepping out to go to work, purchase essentials and carry out other activities. But, normalcy has not returned and the hesitation and fear of safety will still affect people when they think about stepping out. A live monitoring of the safety status of locations around people will help manage crowd and the safety scenario in an efficient manner.
Technological advancement with a blend of machine learning, software development and connectivity through applications can equip the mankind with abilities to fare through this pandemic in a safer and smarter way.
A web application can be designed to help everyone - those who have been affected as well as those who haven't been yet. Providing the people essential statistics and services can make them aware of the current safety scenario at different locations. For the patients, real-time statistics of hospital capacity and other resources stored on the IBM Cloudant database can aid them in reaching to the best possible treatment as quickly as possible.
With the rapid spread of COVID-19 it has become increasingly difficult for the patients who test positive to be accommodated in the treatment centers. Often the treatment centers have no availability of beds and are already working at full capacity, which leads to inefficiency and confusion in communication while transferring patients from a COVID-testing facility to these treatment centres, leading to a risk of life.
Our application provides a platform for the hospitals to share real-time statistics like number of beds available and presence of testing facilities which can significantly help the patient reach the required resources as soon as possible. The data is stored on a hospital monitoring database hosted on IBM's cloud-based service, Cloudant which is used to setup a NoSQL Database which then can be used with a serverless web application.
For the people who have not been infected yet, the most worrying concern is the ground situation of the places that they are planning to visit. Once the lockdown gets lifted, huge numbers of people would take to the streets and the risk of the uncontrolled spread of virus is unfathomable.
The application provides the user with a close real-time approximate of the number of people present at any given location, which helps the user make an aware decision. With the help of deep learning, a mask detection model is also trained and integrated with the app which calculates the safety score of a place based on the percentage of people wearing masks. Mask detection is performed using the live footage from a video camera installed there (CCTV camera, drones or any video camera system feasible depending on the location). This ratio is extremely essential as it gives an idea to the user about the level of risk he/she might take. For publicly crowded places like markets and shopping centres, this application is furthermore crucial as the safety score can clearly convey the peril of visiting the place and allow efficient safety monitoring consequently avoiding overflow of people in places at any point of time.
App ID by IBM is used to monitor authentication for the login procedure in the app. Only hospitals will be authorised to input data into the form at dataentry.html. If an unauthorised person does so, an alert message shows up and the user is prompted to login. Their input will not be recorded until they are authorised. The workflow for the App ID looks as follows:-
The file app.js creates an Authentication Strategy based on App ID. On a call to /appid/login it goes on the authentication page, and in case of success redirects to /dataentry.html. From that page, a click on the logout button, initiates a call to /appid/logout and the user is again directed to /index.html. A successful login also sends a response via /api/user which contains the user's name and the login ID. The username and ID information is also stored along with hospital data to identify the user who filled the data, and hence enable the user to update the records for the same hospital in the future.(Update Feature - Partial Implementation).
IBM Cloud ID, Login via Google and Login via Facebook have been set as Identity Service Providers.
IBM's cloud-based service Cloudant is used to host the Hospital Monitoring Database containing real-time statistics on the presence of covid-19 test facility and number of beds available. Cloudant is used to setup a NoSQL Database which then can be used with a serverless web application.
The details filled by the hospital post login, are sent to the guestbook database (Hospital Monitoring Database) on click of the Submit Details button.
A GET request is also made to the database to retrieve the data regarding hospitals onboard which is displayed on the Currently Onboard tab of the web app. On clicking the More Details button, all the details for the particular hospital are retrieved and made visible to the user.
When the user inputs a specific location in Search for your Destination bar or clicks on the More Details button for a hospital, the location address string is retrieved and sent to the Geocoding API, which finds the latitude and longitude for the place. Then using the Directions API, the shortest route to the destination is computed and displayed on the map. The Distance Matrix API finds the distance and travel time required to reach the destination.
We used the structure of SSD (Single Shot Detector) Object Detector. However, to enable inference on the device integrated with the CCTV camera with minimum latency, the backbone network is light. It is easier to deploy the model to embedded systems (Raspberry Pi, Google Coral, Jetson, Nano, etc.).
The total model has 1.01M parameters. Input size of the model is 260x260, the backbone network has 8 conv layers. Overall, the model has 24 layers with the location and classification layers included. We merge the BatchNormalization to Conv layers in order to accelerate the inference speed.
We apply face detection to compute the bounding box location of the face in the image. Once the faces in the image are detected, we can extract the Regions of Interest (ROI). The image shown below depicts that facial landmarks are used to localize and represent salient regions of the face, such as - Eyes, Eyebrows, Nose, Mouth and Jawline.
Leveraging these facial landmarks, the model learns as to which facial features are covered by mask and which are not.
Thus, we have a classifier ready which adopts the following flow for mask detection:
- Take an input image -> Detect faces -> Detect ROI -> Use Facial Landmarks to Localize -> Classify as mask or no_mask
The mask detection model can be employed on the CCTV cameras installed throughout the city. For the locations with no CCTV cameras in place, the model can be integrated with drones. As it is a light model with less number of parameters, live video stream captured can be directly inferenced on the connected computer or system.
Instead of processing the video stream on cloud, we are leveraging the edge computing facilities already available with the camera and sending only the corresponding numeric metrics calculated. This saves us from the overhead of sending the entire video feed to the cloud and avoids any privacy concerns which may arise due to uploading of live video feeds. This makes the solution even more lightweight and easily deployable.
Safety score is calculated as the percentage of people wearing masks in a given frame. The live video feed is taken as input in intervals of 10 minutes and a corresponding safety score is output for each interval by averaging the safety score over all the frames. A 10 minute interval is selected to give the user a fair idea about the area he/she is planning to visit during that time. Thus, this will help us as a society, be prepared to handle the situation better by delaying outdoor visits if they are not urgent or taking necessary precautions and increasing Personal Protective equipment (PPE) otherwise. The workflow described is summarized in the flowchart below:-
In order to deploy the model on video camera systems (CCTV cameras/drones), we characterize the process into two distinct phases, each with its own respective sub-steps:
- Training: Here the face mask detection dataset is loaded from disk, the model is trained using Pytorch on this dataset, and then the face mask detector is serialized to disk
- Deployment: Once the face mask detector is trained, we can move on to loading the mask detector, performing face detection, and then classifying each face as with_mask or without_mask. Finally, we calculate the safety score based on this prediction and send data to IBM Cloudant
IBM Cloudant is used to setup a NoSQL Database which then can be used with a serverless web application. The Python script uploads data points such as no of people in the camera feed, safety score (based on % of people wearing masks), latitude and longitude of the camera location to the Cloudant mapbook database. The data is retrieved via a GET request, and then based on the user's location entry in the input bar, the database is searched for an entry. If a camera feed is found in the region, the score associated with it is retrieved and displayed to the user. The web app also keeps a track of the average score in all the places that are being monitored.
- IBM Cloudant Documentation
- python-cloudant documentation
- Watson Studio Documentation
- Directions Service-Google Maps API
- Creating and populating a simple IBM Cloudant database on IBM Cloud
- Build models in IBM Watson Studio
- Face Mask Dataset - WIDER Face and MAFA (MAsked FAces) datasets have been combined to form the Face Mask Dataset after correction of wrong annotations
npm install -> To install all the dependecies.
- express
- express-session
- passport
- ibmcloud-appid
node.exe app.js
Then open localhost:3000 in your browser.
pip install -r requirements.txt
- opencv-python
- pytorch
- cloudant
- numpy
- pillow
- python-googlegeocoder
python mask_model.py
Change video_path= variable in the script to use your own video file, or direct a camera feed directly to it using OpenCV.
Let's start by creating a IBM Cloudant. IBM Cloudant is a fully managed data layer designed for modern web and mobile applications that leverages a flexible JSON schema. IBM Cloudant is built upon and compatible with Apache CouchDB and accessible through a secure HTTPS API, which scales as your application grows.
- In the Catalog, select IBM Cloudant under Databases.
- Select a region.
- Pick a unique name for the service, such as
<yourinitials>-guestbook-db. - Select a resource group.
- Select Use both legacy credentials and IAM as authentication method.
- Click Create.
- Back in the IBM Cloud™ Resource List, click on the IBM Cloudant instance you created to open the instance full details page. Note: You may be required to wait until the status of the service changes to
Provisioned. - Under Manage, click on Launch Cloudant Dashboard which will open in a new browser tab. Note: You may be asked to log into your Cloudant instance.
- Click on Create Database and create a database named guestbook. Select Non-Partitioned under Partitioning.
- Back to the service dashboard page, under Service credentials
- Create New credential.
- Set name to for-guestbook.
- Accept the other defaults and click Add.
- Expand the newly created credentials. We will need these credentials later to allow Cloud Functions actions to read/write to your Cloudant service.
In this section, you will create serverless actions (commonly termed as Functions). IBM Cloud™ Functions (based on Apache OpenWhisk) is a Function-as-a-Service (FaaS) platform which executes functions in response to incoming events and costs nothing when not in use.
Sequence of actions to save the guestbook entry
You will create a sequence which is a chain of actions where output of one action acts as an input to the following action and so on. The first sequence you will create is used to persist a guest message. Provided a name, an emailID and a comment, the sequence will:
- Create a document to be persisted.
- Store the document in the IBM Cloudant database.
Start by creating the first action:
- Switch to Functions.
- Select or create a namespace from the namespace drop-down on the top right.
- On the left pane, click on Actions and then Create.
- Create Action with name
prepare-entry-for-saveunder Default Package and select Node.js as the Runtime (Note: Pick the latest version). Refer to IBMActions for code. - Click
SaveThen add the action to a sequence: - On the left pane, click on Enclosing Sequences and then Add To Sequence.
- Set the Sequence name to save-guestbook-entry-sequence. Leave the Default Package.
- Click Create and Add. Finally add a second action to the sequence:
- Click on save-guestbook-entry-sequence and then click Add.
- Select Use Public, Cloudant and then choose create-document under Actions.
- Create New Binding.
- Set Name to binding-for-guestbook.
- Set Instance to Input your own credentials.
- Set Username, Password, Host and IAM API Key from the values found in the Cloudant credentials for-guestbook created earlier.
- Set Database to guestbook.
- Set whiskoverwriteLabel to true.
- Click Save.
Sequence of actions to retrieve entries
The second sequence is used to retrieve the existing guestbook entries. This sequence will:
- List all documents from the database.
- Format the documents and returning them.
- Under Functions, click on Actions and then Create a new Node.js action under Default Package and name it
set-read-input.Refer to IBMActions for code. - Click Save. Add the action to a sequence:
- Click on Enclosing Sequences, Add to Sequence and Create New
- Enter
read-guestbook-entries-sequencefor the Action Name and click Create and Add. Complete the sequence: - Click on read-guestbook-entries-sequence sequence and then click Add to create and add the second action to get documents from Cloudant.
- Under Use Public, choose IBM Cloudant and then list-documents
- Under My Bindings, choose binding-for-guestbook and Add to create and add this public action to your sequence.
- Click Add again to create and add the third action which will format the documents from IBM Cloudant.
- Under Create New enter format-entries for name and then click Create and Add.
- Click on format-entries and refer to IBMActions for code.
- Click on Save.
- Choose the sequence by clicking on Actions and then read-guestbook-entries-sequence.
- Click on Save.
Create an API
- Go to Actions.
- Select the
read-guestbook-entries-sequence sequence. Next to the name, click on Web Action, check Enable as Web Action and Save. - Do the same for the
save-guestbook-entry-sequencesequence. - Go to APIs and Create API (or Create Managed API if you have existing APIs).
- Set API name to guestbook and base path to /guestbook
- Click on Create operation and create an operation to retrieve guestbook entries:
- Set path to
/entries - Set verb to
GET - Select the
read-guestbook-entries-sequenceaction
- Set path to
- Click on Create operation and create an operation to persist a guestbook entry:
- Set path to
/entries - Set verb to
PUT - Select the
save-guestbook-entry-sequenceaction
- Set path to
- Scroll to the end of the page to Create the API. Make note of the provided route, as you will use it from your web application.
- Obtain your credentials in one of the following ways.
- By navigating to the Applications tab of the App ID dashboard. If you don't have an application in the list, you can click Add application to create a one.
- By making a POST request to the /management/v4/{tenantId}/applications endpoint.
Request format:
curl -X POST \ https://us-south.appid.cloud.ibm.com/management/v4/39a37f57-a227-4bfe-a044-93b6e6060b61/applications/ \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer <IAM_TOKEN>' \ -d '{"name": "ApplicationName"}'
Example response:
{ "clientId": "111c22c3-38ea-4de8-b5d4-338744d83b0f", "tenantId": "39a37f57-a227-4bfe-a044-93b6e6060b61", "secret": "ZmE5ZDQ5ODctMmA1ZS00OGRiLWExZDMtZTA1MjkyZTc4MDB4", "name": "ApplicationName", "oAuthServerUrl": "https://us-south.appid.cloud.ibm.com/oauth/v4/39a37f57-a227-4bfe-a044-93b6e6060b61" }
- Refer to app.js for initialising the NodeJS SDK and setting up redirects and callback reqyests.
Google Maps Platform is a set of APIs and SDKs that are managed from the Google Cloud Platform Console (also referred to as the Cloud Console). To get started with Google Maps Platform you need to:
- Create a billing account
- Go to the Google Cloud Platform Console.
- Click the menu button and select Billing.
- Set up your billing account.
- If this is your first billing account, on the Billing page:
- Click the Add billing account button.
- Follow the steps to set up your first billing account.
- If you already have a billing account, on the Billing page:
- Click the billing drop-down and select Manage billing accounts.
- You'll see a list of your current billing accounts.
- Select Create account
- Follow the instructions to set up your next billing account.
- Create a project
- Go to the Google Cloud Platform Console.
- Click the menu button and select Home.
- Click the project drop-down and select NEW PROJECT.
- On the New Project page, fill in the required information.
- Project name: Accept the default or enter a customized name.
- Project ID: Accept the default or click EDIT to enter a customized ID.
- Billing account: Select a billing account for the project (if you only have one billing account, you won't see this option).
- Location: If you have an organization you want to link your project to, click Browse and select it, otherwise choose "No organization".
- Click CREATE
- Enable the following APIs
- Go to the Google Cloud Platform Console.
- Click the menu button and select APIs & Services > Library.
- On the API Library page, locate Maps and click VIEW ALL (15).
- On the Maps page, click the following APIs to enable.
- Maps JavaScript API
- Geocoding API
- Distance Matrix API
- Directions API
- Get, add, and restrict an API key
- Go to the Google Cloud Platform Console.
- Click the project drop-down and select or create the project for which you want to add an API key.
- Click the menu button and select APIs & Services > Credentials.
- On the Credentials page, click Create credentials > API key.
The API key created dialog displays your newly created API key. - Click Close.
The new API key is listed on the Credentials page under API keys. (Remember to restrict the API key before using it in production.)
IBM_video_script-mod.ipynb is the python script for deploying the trained face mask classifier.The Pytorch model(model360.pth) is stored in the models folder in our repository. The .ipynb notebook is made to run cell by cell:
- Import cloudant and other required python libraries
- Add the service credentials and establish a connection to the IBM Cloudant service instance
- Create a new database on cloudant
- Get the latitude and longitude of the video camera location by using python-googlegeocoder library
- Calculate the safety score (% of people wearing masks) using the inference function defined on the frame extracted from video
pytorch_inferenceperforms inference on the live video stream at intervals of 10 minutes by calling the functioninferencewhich calculates the safety score on individual frames- Finally, compile the required data including location information, no of people and safety score in the form of a JSON document and store it in the IBM Cloudant database created
Using the steps menioned above, another cloudant database maskbook is setup for recording the mask detection safety scores. Please refer to IBMAction for the code.
An API for a GET Request is setup in a similar fashion, and it is used to retrieve entries from the maskbook database based on the location entered by the user.
MIT License
Copyright (c) 2020 Team Regression