Answer simple questions using open language models and Wikipedia.
The high-level operation of the application is as follows:
- Accept a natural language query
- Pull proper nouns from the query
- For each proper noun, access its most relevant Wikipedia page
- Using sentence embeddings, select the sentences from Wikipedia most related to the query.
- Pass the query and the related knowledge to a language model fine-tuned for dialog question answering.
- Hopefully get an accurate response back from the model based on the knowledge.
Stop Word list from Terrier