openai/evals

How to dynamically change API parameters through `oaieval` CLI tool?

leodiegues opened this issue · 2 comments

leodiegues commented

I'm trying to run an eval based on evals.elsuite.basic.match:Match class but I can't find a way to modify API parameters such as temperature and max_tokens through the CLI. What is the current solution that the framework offers to this problem?

andrew-openai commented

It's a good idea! I added a --modelspec_extra_options that would support setting those values from the CLI, e.g.

oaieval gpt-3.5-turbo test-match --modelspec_extra_options max_tokens=50,temperature=1

Edit: Although that said, I am curious why you want nonzero temperature for evals, usually we want to set that value to 0 for determinism across runs.

  • 🎉3
leodiegues commented

Wow! Thank you so much for your help @andrew-openai . About the temperature, it doesn't make any sense changing it at all for match evals. You're completely right. Actually, I was just trying to mess around with API parameters in general to better understand how to work with Portuguese prompts and responses. For a moment I thought temperature could be something to that I should change but I wasn't thinking straight anymore I guess.

Again, thank you so much @andrew-openai !