agential-ai/agential

[Feature Request]: SVAMP for CRITIC

Closed this issue · 1 comments

alckasoc commented

Feature Description

Familiarize yourself with the repository and take a look the CRITIC repo, paper, and SVAMP.

There are 3 math reasoning tasks tested in the paper: GSM8k, SVAMP, TabMWP.

Currently, the CRITIC implementation only has prompts for HotpotQA and TriviaQA.

Add the SVAMP prompts and relevant logic to the current CRITIC implementation. You'll see that an agent's current structure is divided into cog/agent, cog/modules, and cog/functional/. This task will require you to make modifications in cog/prompts, but will also require you to test your code in all the other relevant modules cog/functional and cog/agent. CRITIC does not have any cog/modules.

What to submit:

  • Set up your environment via the CONTRIBUTING.md
  • Make a Pull Request (PR)
  • Add the prompts for SVAMP
  • Write a short notebook tmp.ipynb in cog/agent showcasing the agent ran on a sample question from the benchmark
    • Add print statements for all calls to the LLM for easier debugging + I can easily verify the outputs
  • Request a review from @alckasoc

If there is any additional logic for how CRITIC is tested on SVAMP (refer to the paper's repo), include these specifications in the PR description.

Feel free to ask me questions on slack if you're confused! Good luck!

SteveZ-Cal commented

I'll take this issue :)

  • 👍1