From the December 2021 update for sponsors:
- The two points above are next steps on the general track I'm following besides following a bachelor's degree. However, I'm really excited about this course on reinforcement learning I'm following which gives us students a lot of freedom in what project we'd like to focus on. Of course, I plan on using the available resources (expert feedback and compute) in order to do something thoughtware-related using RL. Concretely, I was thinking of developing a virtual sandbox environment for training conversational virtual assistants to collaborate with users effectively, helping users achieve certain goals like come up with new solutions or better understand a concept. Crucially, the most interesting part of this project is that during training, the RL agent is interacting not with the user (not enough time to generate enough data that way), but with a user simulation hallucinated by a language model like GPT-3. The RL agent might eventually get to something like: "Huh, so I've noticed that when the user appears to be in a state of confusion, it helps if I ask them to explain their reasoning to a child, so I'll take this action now..." After thousands of dreamed up conversations with the user simulation (hence the really tentative name: oneironomicon), my hypothesis is that the RL agent will be able to transfer its effectiveness to the real world, with real human users, a bit like OpenAI trained a robot hand to solve a Rubik's cube in a virtual game engine for efficiency before applying it in the real world. I'll get back with a lot more details in the upcoming weeks.
From an exchange with my lecturer:
I want to train an RL agent to help a person answer a given question through a conversation. Just like a good teacher might recognize different obstacles a student is facing while reasoning through a question (e.g. getting stuck, confusion) and provide appropriate guidance (e.g. How could you approach this differently?, What if you tried to break this down into smaller pieces?), I want the trained RL agent to be able to perform those interventions (i.e. actions) strategically, getting the person to answer the question well. But collecting experience from interacting with a person is extremely slow, so I want to simulate the student using a language model like GPT-2 with a clever prompt, which would essentially provide the RL agent's (conversational) environment. The actions would be discrete and finite (e.g. a list of 200 question prompts). The state space would be based on the distribution of the student's replies in semantic space, but discretized for simplicity (e.g. k-means with k=1000 "states of the student's mind"). The objective would be to get the student to answer a given question (e.g. using a model which recognizes whether an answer actually answers a question). As for the RL algorithm, maybe Q-learning with a softmax policy which decreases in temperature over training, slowly going from exploration to exploitation. Hopefully, the experience of the RL agent with the student simulation would transfer to a real human person.