Appendix B, listings 2,3,4, and 5?

Question

Appendix B, listings 2,3,4, and 5?

rodrigob opened this issue a year ago · 5 comments

I am looking for the API examples provided per-dataset as hinted in listings 2,3,4, and 5, of the appendix.
From a quick search I could not find them in the codebase. Are they available somewhere?

Answer 1 · 2023-03-22T20:50:20.000Z

Hi, thanks for your interest! We will release benchmark-specific details upon paper acceptance.

Answer 2 · 2023-03-22T22:30:52.000Z

In the meantime:
a) Can you confirm that he prompt includes: the API (with docstring documentation) + dataset specific examples of how to use the API + the image & text question.
b) Could you indicate (roughly) how many dataset specific examples are provided in the prompt ?

Thanks for your reply!

Answer 3 · 2023-03-22T22:53:38.000Z

Hi Rodrigo,

a) The prompt for the benchmarks includes the API + dataset-specific query-code example pairs, but it does not contain the image or video (Codex does not take visual input). It also does not contain the ground truth answer.

b) We use around 8 examples per dataset.

Answer 4 · 2023-03-22T23:49:16.000Z

Got it, thanks for the information.
I think sections 3.1 and 3.2 of the paper would benefit from making more explicit the usage of per-dataset query-code pairs beyond the ones included in the docstring.
We were quite surprised when we noticed the mention at the very end of the appendix.

Answer 5 · 2023-03-23T00:27:12.000Z

Thanks for the suggestion!
The use of examples is mentioned in Section 3 (not only at the end of the appendix), but we agree it could be clearer that dataset-specific optional examples can be included; we follow Flamingo (Appendix A2) [1] in primarily discussing this in the Appendix.
We also want to point out that in the wild, these are not necessary — results like those in Figure 1 don’t use any examples.