leetcode dataset not found

[jonastuttle]

I noticed that this repository has already added its own dataset for leetcode, but it is not yet enabled to be one of the "available" datasets.

Command:

python3 automodel.py --name codellama/CodeLlama-7b-Instruct-hf --root-dataset leetcode --lang py --temperature 0.8 --batch-size 5 --completion-limit 20 --output-

Error:

ValueError: BuilderConfig 'leetcode-py' not found. Available: ['humaneval-cpp-keep', 'humaneval-cpp-transform', 'humaneval-cpp', 'humaneval-cpp-remove', 'humaneval-cs-keep', 'humaneval-cs-transform', 'humaneval-cs', 'humaneval-cs-remove', 'humaneval-d-keep', 'humaneval-d-transform', 'humaneval-d', 'humaneval-d-remove', 'humaneval-go-keep', 'humaneval-go-transform', 'humaneval-go', 'humaneval-go-remove', 'humaneval-java-keep', 'humaneval-java-transform', 'humaneval-java', 'humaneval-java-remove', 'humaneval-jl-keep', 'humaneval-jl-transform', 'humaneval-jl', 'humaneval-jl-remove', 'humaneval-js-keep', 'humaneval-js-transform', 'humaneval-js', 'humaneval-js-remove', 'humaneval-lua-keep', 'humaneval-lua-transform', 'humaneval-lua', 'humaneval-lua-remove', 'humaneval-php-keep', 'humaneval-php-transform', 'humaneval-php', 'humaneval-php-remove', 'humaneval-pl-keep', 'humaneval-pl-transform', 'humaneval-pl', 'humaneval-pl-remove', 'humaneval-py-keep', 'humaneval-py-transform', 'humaneval-py', 'humaneval-py-remove', 'humaneval-r-keep', 'humaneval-r-transform', 'humaneval-r', 'humaneval-r-remove', 'humaneval-rb-keep', 'humaneval-rb-transform', 'humaneval-rb', 'humaneval-rb-remove', 'humaneval-rkt-keep', 'humaneval-rkt-transform', 'humaneval-rkt', 'humaneval-rkt-remove', 'humaneval-rs-keep', 'humaneval-rs-transform', 'humaneval-rs', 'humaneval-rs-remove', 'humaneval-scala-keep', 'humaneval-scala-transform', 'humaneval-scala', 'humaneval-scala-remove', 'humaneval-sh-keep', 'humaneval-sh-transform', 'humaneval-sh', 'humaneval-sh-remove', 'humaneval-swift-keep', 'humaneval-swift-transform', 'humaneval-swift', 'humaneval-swift-remove', 'humaneval-ts-keep', 'humaneval-ts-transform', 'humaneval-ts', 'humaneval-ts-remove', 'mbpp-cpp-keep', 'mbpp-cpp', 'mbpp-cs-keep', 'mbpp-cs', 'mbpp-d-keep', 'mbpp-d', 'mbpp-go-keep', 'mbpp-go', 'mbpp-java-keep', 'mbpp-java', 'mbpp-jl-keep', 'mbpp-jl', 'mbpp-js-keep', 'mbpp-js', 'mbpp-lua-keep', 'mbpp-lua', 'mbpp-php-keep', 'mbpp-php', 'mbpp-pl-keep', 'mbpp-pl', 'mbpp-py-keep', 'mbpp-py', 'mbpp-r-keep', 'mbpp-r', 'mbpp-rb-keep', 'mbpp-rb', 'mbpp-rkt-keep', 'mbpp-rkt', 'mbpp-rs-keep', 'mbpp-rs', 'mbpp-scala-keep', 'mbpp-scala', 'mbpp-sh-keep', 'mbpp-sh', 'mbpp-swift-keep', 'mbpp-swift', 'mbpp-ts-keep', 'mbpp-ts']

[cassanof]

Hi @way2swaggy, the leetcode datasets are not stable yet, so you will have to build them on your own. To do so, you can use the prepare_prompts_json.py in the dataset_builder folder. Here is how you build a Racket LeetCode-hard dataset for instance:
python3 prepare_prompts_json.py --lang humaneval_to_rkt.py --originals ../datasets/leetcode-hard --output ../prompts/leetcode-hard-rkt.jsonl --prompt-terminology reworded

Then, to run this dataset you will have to pass the --use-local and --dataset path/to/translated/dataset flags in automodel_vllm.py.

Let me know if I can further help.