FudanSELab/ClassEval

Several minor issues when running tests on ClassEval.

ztwater opened this issue · 3 comments

Thanks for your contribution with a class-level context-dependent method generation benchmark. Here are some minor problems I found when I inspected the test results. I downloaded this benchmark in August and maybe some of them have been fixed up to now.

  1. 测试方法中所依赖的python包需要提前安装,建议可以给一个list,否则会导致运行测试时大量的NameError。

  2. ClassEval_5_display
    返回值应该是string而非None

  3. ClassEval_44_format_line_html_text
    测试代码中多了from benchmark_code.HtmlUtil import HtmlUtil

  4. ClassEval_49_search_jobs
    :param criteria: The requirements of the position,list.
    实际上输入是字符串,e.g., 'skill1'

  5. ClassEval_50
    类名是JSONProcessor,实际文件命名为JSONHandler

  6. ClassEval_52_lemmatize_sentence
    "solution_code"删去nltk download语句,下载nltk对应包,放到环境中,防止测试超时的问题

  7. ClassEval_60_create_table
    方法描述中的字段名写错,'author name'应该是'theater_name'

  8. ClassEval_69_extract_text_from_pdfs
    方法的测试代码少了一个测试类

  9. ClassEval_94_display_items
    :return: If the vending machine is empty, returns False, otherwise, returns a list of the products in the vending machine, list.
    实际测试用例写的是返回 string

非常感谢您的反馈!我们会仔细检查这些问题并且重新发布升级后的benchmark以及所需的list

我们已经修复您提到的问题,重新发布了1.1.0版本的classeval数据,并且更新了requirements

感谢你们的迅速回复和处理!ClassEval是一个对领域很有价值的benchmark,期待它的影响力越来越大。