Help wanted! What is the significance of extracting the skeleton?

[kanseaveg]

I am very interested in your articles and code. When I read the code, I found that you use sentence-transformers to calculate Euclidean distance to extract examples. At the same time, I also noticed that you have extracted the question skeleton and SQL skeleton and they were used when generating question.json.

    def __init__(self, tokenizer: str, *args, **kwargs):
        self.tokenizer = get_tokenizer(tokenizer)
        self.example_qualities = []
        self.pattern_similarities = []

    def record_example_quality(self, examples, target):
        quality_list = []
        for example in examples:
            quality_list.append(jaccard_similarity(example["query_skeleton"], target["query_skeleton"]))
        self.example_qualities.append(quality_list)

    def get_example_quality(self):
        if self.example_qualities:
            return np.mean([num for row in self.example_qualities for num in row])
        else:
            return 1

    def get_example_quality_for_each(self):
        if self.example_qualities:
            return [np.mean(row) for row in self.example_qualities]
        else:
            return []

    def record_pattern_similarity(self, examples, target):
        similarity_list = []
        for example in examples:
            similarity_list.append(jaccard_similarity(example["question_pattern"], target["question_pattern"]))
        self.pattern_similarities.append(similarity_list)

    def get_pattern_similarity(self):
        if self.pattern_similarities:
            return np.mean([num for row in self.pattern_similarities for num in row])
        else:
            return 1

However, to my surprise, you only used the skeleton to calculate the average for output and did not use them in ask_llm.py.

So I'm curious about what is the purpose of extracting skeletons?

I am very interested in your work, could you please help me clarify my doubts? Thank you.

[BeachWang]

Hi,

We calculate example_quality and pattern_similarity as the "query similarity" and "question similarity" respectively mentioned in Table 2 of our paper. They imply the quality of examples we have selected. Since the examples have been organized to the prompts in question.json, the two similarity scores are not used in ask_llm.py.