Order processing during pandemics for offline mode

How do we stop panic amongst people of hoarding essentials during lockdown? How do we maintain social distancing while procuring essentials?

Majority of Indian population (specially elderly population) do not have knowledge about digital services. Hence, they will not be able to use the available online portals to order daily essentials & medicines.

A toll free number (ex :- dial *222) based approach to contact control tower will be a better option. Here the user will be able to call toll free number and order the essentials. This will address the limitation mentioned above and will have wider reach with the population to make use of the facility. However,with majority of them using this toll free number based approach the traffic at the backend will be very high. To process of these calls manually will be tedious.

In this code pattern, we build an AI-powered backend system that can take the daily essentials orders through offline mode. The system processes the audio request by converting it to formatted orders list. A smart way of processing the information quickly.

This AI powered backend system can be later connected to the inventory database for optimising supply chain management aswell. This solution will be applicable into various domains such as ordering medicines and ordering daily essentials(groceries), etc.

When the reader has completed this Code Pattern, they will understand how to:

Use Speech to Text service.
Use Watson Language Translator.
Use and train model on Watson Knowledge Studio.
Deploy model on Watson Natural Language Understanding.

Flow

Feed the audio to Speech to Text service.
Convert the text into english using Watson Language Translator.
Feed the English text to Watson Knowledge Studio model which is deployed on Watson Natural Language Understanding.
The model deployed on Watson Natural Language Understanding will identify all the required attributes from the text.
These extracted attributes gets stored on Db2 on cloud.
Visualize the order and customer details from the recordings on a dashboard.

Watch the Video

Pre-requisites

IBM Cloud account: Create an IBM Cloud account.

Steps

Please follow the below to setup and run this code pattern.

Clone the repo
Setup Watson Speech to Text
Setup Watson Language Translator
Setup Watson Knowledge Studio and Natural Language Understanding
Setup IBM Db2
Add the Credentials to the Application
Deploy the Application to Cloud Foundry
Analyze the results

1. Clone the repo

Clone the order_processing_during_pandemics repo locally. In a terminal, run:

$ git clone https://github.com/IBM/offline_order_processing_during_pandemics

2. Setup Watson Speech to Text

Create Watson Speech to Text service
In Speech To Text Service Resource Page, Click on Services Credentials and create a New Credential as shown.
Once the credential is created, copy and save the credentials in a text file for using it in later steps in this code pattern.

3. Setup Watson Language Translator

Create Watson Language Translator

Make a note of the Watson Language Translator API Key and URL credentials as shown below,

Note: This service can be used incase you are using regional languages. Watson Language Translator supports these languages

4. Setup Watson knowledge Studio and Natural Language Understanding

Create the following services:

i. Launch Watson Knowledge Studio and create workspace

Launch the Watson Knowledge Studio tool.
click on Create workspace.

Enter a unique name and press Create.

ii. Upload Type System

A type system allows us to define things that are specific to review documents, such as product and brand names. The type system controls how content can be annotated by defining the types of entities that can be labeled and how relationships among different entities can be labeled.

To upload our pre-defined type system, from the Assets -> Entity Types panel, press the Upload button to import the Type System file TypeSystems.json found in the repository.

Press the Upload button. This will upload a set of Entity Types and Relation Types:

iii. Import Corpus Documents

Corpus documents are required to train our machine-learning annotator component. For this code pattern, the corpus documents will contain sample order processing converstation documents.

From the Assets -> Documents panel, press the Upload Document Sets button to import a Document Set file.

Use the corpus documents file training_Files.zip found in the local repository.

Once uploaded, you should see a set of documents:

iv. Create Custom Model

Since the corpus documents that were uploaded were already pre-annotated and included ground truth, it is possible to build the machine learning annotator directly without the need for performing human annotations.

Go to the Machine Learning Model -> Performance panel, and press the Train and Evaluate button.

Click on Edit Settings to ensure you are selecting the corpus document set which we have uploaded in previous step.

From the Document Set name list, select the annotation sets Import. Also, make sure that the option Run on existing training, test and blind sets is checked. Press the Train & Evaluate button.

This process may take few minutes to complete. Progress will be shown in the upper right corner of the panel.

You can view the log files of the process by clicking the View Log button.

Once complete, you will see the results of the train and evaluate process as successful.

v. Deploy the machine learning model to Natural Language Understanding

Now we can deploy our new model to the already created Natural Language Understanding service. Navigate to the Versions menu on the left and press Create Version.

Give a description on the version that you want to deploy.

The new version will now be available for deployment to Natural Language Understanding. To start the process, click the Deploy button associated with your version.

Select the option to deploy to Natural Language Understanding.

Enter your IBM Cloud account information to locate your Natural Language Understanding service to deploy to.

Once deployed, a Model ID will be created. Keep note of this value as it will be required later when configuring your credentials.

NOTE: You can also view this Model ID by clicking the Deployed Models link under the model version.

vi. Get the credentials of NLU service.

Go to IBM Cloud dashboard resource list to note down the NLU credentilas. Click on https://cloud.ibm.com/resources

On the left navigation bar, click on Manage and copy API key and the url to a notepad which will be used in the next section for UI integration.

5. Setup IBM Db2

Create a Db2 service.

Click on Service credentials and click on New Credentials to generate credentials and click on save credentials as shown.

6. Add the Credentials to the Application

Open the credentials.json file from the root directory and paste the Watson Speech to Text created in step 2 and save the file.
Open the credentials1.json file from the root directory and paste the Db2 Credentials created in step 5 and save the file.
Open app.py from the root directory, goto line number 33 and insert the natural language understanding API Key apikey, goto line number 34 and insert the natural language understanding URL nlu_url and lastly goto line number 35 and insert the knowledge studio model ID wks_model_id.

# Initialize WKS Model Credentials

apikey = 'YOUR-NLU-API-KEY-HERE'
nlu_url = 'YOUR-URL-HERE'
wks_model_id = 'YOUR-WKS-MODEL-ID-HERE'

7. Deploy the Application to Cloud Foundry

Make sure you have installed IBM Cloud CLI before you proceed.
Log in to your IBM Cloud account, and select an API endpoint.

$ ibmcloud login

NOTE: If you have a federated user ID, instead use the following command to log in with your single sign-on ID.

$ ibmcloud login --sso

Target a Cloud Foundry org and space:

$ ibmcloud target --cf

From within the cloned directory push your app to IBM Cloud.

$ ibmcloud cf push

Once Deployed You will see output on your terminal as shown, verify the state is running:

Invoking 'cf push'...

Pushing from manifest to org manoj.jahgirdar@in.ibm.com / space dev as manoj.jahgirdar@in.ibm.com...

...

Waiting for app to start...

name:              order-processing-pandemic
requested state:   started
routes:            order-processing-pandemic.xx-xx.mybluemix.net 
last uploaded:     Sat 16 May 18:05:16 IST 2020
stack:             cflinuxfs3
buildpacks:        python

type:            web
instances:       1/1
memory usage:    256M
start command:   python app.py
     state     since                  cpu     memory           disk           details
#0   running   2020-05-16T12:36:15Z   25.6%   116.5M of 256M   796.2M of 1

Once the app is deployed you can visit the routes to view the application.

8. Analyze the results

Once you visit the url order-processing-pandemic.xx-xx.mybluemix.net you will see the following screen.

Upload the Hindi regional telephone conversation 9192939495.wav to get the orders placed by the person as shown.

You can play the audio to verify that Watson Knowledge Studio is identifying the entities correctly.

Note: You can also upload English telephone conversations to get the orders.

License

This code pattern is licensed under the Apache License, Version 2. Separate third-party code objects invoked within this code pattern are licensed by their respective providers pursuant to their own separate licenses. Contributions are subject to the Developer Certificate of Origin, Version 1.1 and the Apache License, Version 2.

Apache License FAQ

salbertengo/offline_order_processing_during_pandemics