You will make a chatbot that answers questions about physics. The aim is to be a teaching aid that can help students in their journey towards understanding physics. The domain of the knowledge is confined to those taught in the book attached here Download here.
The user interface does not need to be fancy, just a plain chat interface that allows users to chat. If you can handle multiple chat sessions, that would be great, but all I need from you is a chat interface that allows chatting and also one where the user can clear the chat if needed. The clearing can be done in different ways, including deleting the session.
You are only allowed to use open-source technologies, in addition to any code you write yourself. The expectation is that the user’s chat experience with your chatbot will be just-as-good-if-not-better when compared to existing widely used products like ChatGPT, Gemini, Meta Chat etc. albeit in the limited domain of the Physics book.
The Physics chatbot should run locally without requiring internet connection.
- Using Large Language Models (LLMs) and Small Language Models (SLMs)
- Using Retrieval Augmented Generation (RAG)
- Evaluating the responses of a LLM/SLM based chatbot
- Instruction fine-tuning - particularly how to curate data
Each group has to present their work for each of the following milestones in the prescribed week (unless announced otherwise).
Each group should make a presentation on:
- What they understood of the problem
- The UI they are proposing
- Results of some Retrieval Augmented Generation prototyping they have done (Demo)
- A comparative analysis of different Small/Large Language Models
- A comparative analysis of different embedding models
- A comparative analysis of different chunking strategies
A working version of the Physics chatbot that uses RAG to answer questions. It should be able to hold a conversation and answer correctly to factual questions.
A test plan document that details:
- Strategies that are widely used to test chatbots (with citations)
- Metrics used to calculate chatbot performance
- A test plan that includes:
- Test data collection/curation plan
- Automated testing plan with metric calculation
- Manual testing plan (incorporating more than 10 users)
An improved version of the Physics chatbot. In addition to the previous capabilities, it should be able to explain different concepts.
Automated test results with values of different test metrics, analysis of failure cases, and clear plan to address them.
A further improved version of the Physics chatbot with previously found shortcomings addressed. In addition to the previous capabilities, it should be able to answer exercise problems (and other problems of similar difficulties).
Automated test results with values of different test metrics, results of manual testing, analysis of failure cases, and clear plan to address them.
Final Report Due.
Use the CSE499 Capstone Report Template prepared by the Department. You will need to modify sections to match your 299 section details. Additionally, Chapter 7 is not needed for CSE299.