the-crypt-keeper/can-ai-code

Too few questions for senior category

rodion-m opened this issue · 2 comments

rodion-m commented

Hi! Thanks for a useful benchmark. The separation of quants is great.

Am I right in thinking that we currently have only one question in the senior category? I'm afraid it might not be representative.

rodion-m commented

Also, if the question is only one, what does "Passed" column mean here?
image

the-crypt-keeper commented

@rodion-m There's 3 questions in total, the meat of this test is in vm.yaml which asks the LLM to build an assembler. The score is based on how well the answer performs on the Checks.. the model can get up to weight points for each check and partial marks are possible: for example if 3 of 8 returned list entries are correct the model will still get 3 points, otherwise the scores would all be too low.