Where Can I Submit my Models for evaluation?

[ajinkya123-robo]

Hello,
This is a great initiative! Where can i submit my models for evals?
Thanks

[wkx228]

Hello, This is a great initiative! Where can i submit my models for evals? Thanks

We provide inference.py script to generate class-level code outputs. If your model is available on hugging face and not in the default model list, you can set the argument --model to 10, and set the model’s path by argument --checkpoint to run the script. You can follow the README for more details and the specific description for different arguments can be obtained by using --help. Note that we use API AutoModelForCausalLM.from_pretrained and model.generate to load and run most models, so if your model is unavailable on hugging face, you might need to customize the code for model loading and inference (see get_model_tokenizer_and_config and model_generate methods in inference_pipeline.py). Feel free to contact us if you have any other questions.