Create a Chatbot experience to help students learning remotely to find courses and develop curriculums

Description

It is the year 2020 and students are experiencing a new way of life when it comes to getting an education. Students are realizing they need to adopt a proactive and self-service mindset in to fulfill their academic needs. An intelligent chatbot that helps students find and access learning content supports this new self-service model. This pattern shows users how to build a self-service chatbot not only for education, but also for any other industries where users need to find information quickly and easily.

Using Watson Assistant, this pattern defines a dialog that a student and a course provider might experience as a student searches for learning content. Students can input grade-level and academic topics question, and the chatbot responds with course recommendations and learning content links. The conversation responses are further enhanced by using Watson Discovery and the Watson Assistant Search skill. Natural Language Understanding (NLU) is introduced in this pattern to complement Watson Discovery's accuracy by extracting custom fields for entities, concepts, and categories.

What is an Assistant Search Skill?

An Assistant search skill is a mechanism that allows you to directly query a Watson Discovery collection from your Assistant dialog. A search skill is triggered when the dialog reaches a node that has a search skill enabled. The user query is then passed to the Watson Discovery collection via the search skill, and the results are returned to the dialog for display to the user. Customizing how your documents are indexed into Discovery improves the answers returned from queries and what your users experience.

Click here for more information about the Watson Assistant search skill.

Why Natural Language Understanding (NLU)?

NLU performs text analysis to extract metadata such as concepts, entities, keywords, and other types of categories of words. Data sets are then enriched with NLU-detected entities, keywords, and concepts (for example, course names). Although Discovery provides great results, sometimes a developer finds that the results are not as relevant as they might be and that there is room for improvement. Discovery is built for "long-tail" use cases where the use case has many varied questions and results that you can't easily anticipate or optimize. Additionally, if the corpus of documents is relatively small (less than 1000), Discovery doesn't have enough information to distinguish important terms and unimportant terms. Discovery can work with a corpus this small - but it is less effective because it has less information about the relative frequency of terms in the domain.

Flow

Execute Python program to run data set through Natural Language Understanding to extract the meta-data (e.g. course name, desciption,etc) and enrich the .csv file
Run Node program to convert .csv to .json files (required for the Discovery collection)
Programmatially upload .json files into the Discovery Collection
The user interacts through the chatbot via a Watson Assistant Dialog Skill
When the student asks about course information, a search query is issued to the Watson Discovery service through a Watson Assistant search skill. Discovery returns the responses to the dialog

Included components

IBM Watson Assistant: Build, test and deploy a bot or virtual agent across mobile devices, messaging platforms, or even on a physical robot.
IBM Watson Discovery: A cognitive search and content analytics engine for applications to identify patterns, trends, and actionable insights.
IBM Watson Natural Language Understanding: Analyze text to extract metadata from content such as concepts, entities, keywords, categories, sentiment, emotion, relations, and semantic roles using natural language understanding.

Featured technologies

Node.js Versions >= 6: An asynchronous event driven JavaScript runtime, designed to build scalable applications.
Python V3.5+: Download the latest version of Python
Pandas: pandas is a fast, powerful, flexible, and easy-to-use open source data analysis and manipulation tool built on top of the Python programming language.

Pre-requsites

IBM Cloud Account.
Basic familiarity of IBM Cloud, Assistant service, Natural Language Understanding (NLU) service and Discovery service.

Steps

Follow these steps to setup and run this code pattern. The steps are described in detail below.

Clone the Rep
Create IBM Cloud services
Configure Watson Natural Language Understanding
Configure Watson Discovery
Configure Watson Assistant and Test the Chatbot

1. Clone the repo

git clone https://github.com/IBM/Education-SelfService-AI-Assistant

2. Create IBM Cloud services

Create the following services:

NOTE: use the Plus offering of Watson Assistant. You have access to a 30 day trial.

Create a copy of the env-sample file and call it .env Add in your NLU and Discovery credentials.

3. Configure Watson NLU

NLU enriches Discovery by creating the addition of metadata tags to your data sets. In other words, includes terms that overlap with words that users might actually provide in their queries.

The following instruction has the developer run the .csv files through NLU and extract entities and concepts. Do this by running the python program:

cd src
pip install watson-developer-cloud==1.5
pip install --upgrade ibm-watson
pip install pandas
sudo pip3 install -U python-dotenv
python NLUEntityExtraction.py

Note that this may take a few minutes. This will create 2 .csv files. Take a look at the format by exploring the files.

4. Configure Watson Discovery

Watson Discovery uses AI search technology to retrieve answers to questions. It contains language processessing capabilities and can be trained on both structured and unstructured data. The data that Discovery is trained on is contained within what is called a Collection (aka Database). You can learn more about Watson Discovery here

Create Discovery Collection

First create a Discovery Collection. This is the database that holds your response data.

Find the Discovery service in your IBM Cloud Dashboard.
Click on the service and then click on Launch Watson Discovery.
Create a new data collection by hitting the Upload your own data button. You will see that you have one collection created that comes with Discovery by default. That is the Watson Discovery News collection.
Provide a collection name - call it Courses
Select English language
Click Create

You will need to configure Discovery in 2 steps:

Create a set of .json files that Discovery will consume for its collection. The node program converts the .csv file to a set of .json files in a directory named manualdocs.

Install Node.js (Versions >= 6).
In the root directory of your repository, install the dependencies.

npm install

Run below command

node read-file.js

Verify the JSON files exists.

Programmatically upload the .json files into the discovery collection

ensure you have added your Discovery credentials into a .env file sitting in your root directory of your repo.

 npm install python-dotenv
 npm install ibm-watson
 node upload-file.js

This will take approximately 5 minutes or less. Documents are uploaded to your Courses Collection. Take a look and explore the collection. You should see about 135 documents.

NOTE: If using the Discovery Lite plan, you are limited to loading up to 1000 files into your discovery service. This limit is not per collection, but the combined number for all collections in your service.

5. Configure Watson Assistant and test the Chatbot

Create assistant

Find the Assistant service in your IBM Cloud Dashboard.
Click on the service and then click on Launch Watson Assistant.
Go to your Assistant tab and click create assistant.

Create Assistant dialog skill and test the chatbot

You will see that you now have the ability to add a:

Dialog skill
Search skill

Click on Add dialog skill

Note: You will import the Dialog skill that is saved within this repo: data/assistant/Student-skill-dialog.json.

Click on Import Skill
Click on Choose JSON File. Go to your cloned repo dir, and Open the JSON file in data/assistant/Student-skill-dialog.json
Click on Import

You will see that the dialog has been imported. Click on the words Student-skill and explore the dialog to see the intents, entities, dialog. You can also add to the dialog to extend the conversation.

Note: you can also create your dialog from scratch by selecting Create skill and adding in your own intents, entities and dialog.

Create Assistant search skill

From your Assistant panel, click on Add search skill.

Note: If you have provisioned Watson Assistant on IBM Cloud, the search skill is only offered on a paid plan, but a 30-day trial version is available if you click on the Plus button.

From the Add Search Skill panel, select the Create skill tab.

Give your search skill a unique name, then click Continue.

From the search skill panel, select the Discovery service instance and collection you created previously.

Note Assistant can only connect to 1 Discovery collection at a time. You will use either the elementary or high school collections and then discovery will respond appropriately.

Click Configure to continue.

From the Configure Search Response panel, select name as the field to use for the Title of the response. Select Description as the field for the Body and select URL as the filled to use foor URL. Click Save to complete the configuration.

Now when the dialog skill node invokes the search skill, the search skill will query the Discovery collection and display the text result to the user.

Learn more about the assistant search skill here

Enable search skill

From your Assistant panel, click on the three dots in the upper right-hand corner and select Settings.

Select the Search Skill tab and ensure that is in Enabled.

NOTE: The following feature is currently only available for Watson Assistant provisioned on IBM Cloud.

Test out the Chatbot

Normally, you can test the dialog skill be selecting the Try it button located at the top right side of the dialog skill panel, but when integrated with a search skill, a different method of testing must be used.

From your assistant panel, select Add Integrations.

From the list of available integration types, select Preview link.

From the Preview link integration panel, name your preview link and click Create.

If you click on the generated URL link, you will be able to interact with your dialog skill. Note that the input "Can you recommend a math class?" has triggered our Which grade do you want recommendation for dialog node and invoked our search skill.

Troubleshooting

Error: If you get an error that says you > Traceback (most recent call last): > File "NLUEntityExtraction.py", line 4, in > import pandas as pd > ImportError: No module named pandas
```
> you need to make sure you are using the same verion of pip and python.  We recommend you use Python 3 and Pip 3
```
Error: Unable to list workspaces for Watson Assistant: Forbidden: Access is denied due to invalid credentials.

This error occurs with Deploy to IBM Cloud button. Configure a runtime environment variable for ASSISTANT_APIKEY to allow automatic configuration of the default skill or configure SKILL_ID to use another skill.
Error: Only one free environment is allowed per organization

To work with a free trial, a small free Discovery environment is created. If you already have a Discovery environment, this will fail. If you are not using Discovery, check for an old service thay you may want to delete. Otherwise use the .env DISCOVERY_ENVIRONMENT_ID to tell the app which environment you want it to use. A collection will be created in this environment using the default configuration.

Learn more

Artificial Intelligence Code Patterns: Enjoyed this code pattern? Check out our other AI Code Patterns.
AI and Data Code Pattern Playlist: Bookmark our playlist with all of our code pattern videos
With Watson: Want to take your Watson app to the next level? Looking to utilize Watson Brand assets? Join the With Watson program to leverage exclusive brand, marketing, and tech resources to amplify and accelerate your Watson embedded commercial solution.

License

This code pattern is licensed under the Apache Software License, Version 2. Separate third party code objects invoked within this code pattern are licensed by their respective providers pursuant to their own separate licenses. Contributions are subject to the Developer Certificate of Origin, Version 1.1 (DCO) and the Apache Software License, Version 2.

Apache Software License (ASL) FAQ

quierati/Education-SelfService-AI-Chatbot