xlang-ai/UnifiedSKG

BLEC metric for SQL-to-Text

awasthiabhijeet opened this issue · 3 comments

Dear Authors,

Thanks for releasing this code.

I have a question regarding the BLEC metric for SQL-to-Text:

Does BLEC utilize a gold reference? Or is it just based on the predicted text and the input SQL?
https://github.com/HKUNLP/UnifiedSKG/blob/0d47cd9382f0a3d17c70701056f399706ee3b698/third_party/BLEC/Spider/spider.py#L22

Thanks

Hi Abhijeet,

Thanks for double-checking! The short answer is: the SQL2Text BLEC does not utilize a gold reference.
Here is a longer one: BLEC is a metric defined by Shu et al. (2021), and it requires hand-crafted templates for the logic matching process. As of Feb 5th, 2022, they have released the Logic2Text BLEC at https://github.com/chatc/BLEC, but not yet the SQL2Text BLEC. We contacted the authors last year, and they gave us the code generously, explaining that this code can only be used for the dev. and test splits proposed in their paper because currently they only have templates for these splits. Hope this information is helpful!

Thanks

I see. Thank you for the clarification.

Thanks for double-checking!