google-research/FLAN

Is Inversion used for the Few-Shot case?

AadSah opened this issue · 3 comments

Hi @shayne-longpre do you use inversion for the few-shot data? If yes, can you provide the different permutations you use for it?

Thanks!

@AadSah Yes we do! There are many permutations built into the templates.

For instance, if you look at templates.py and cmd-F for "_input_inversion" you will see the explicitly added input inversion templates for CoT, Dialog, and Program Synthesis tasks.

For Natural Instructions and T0, the specific tasks come pre-defined in normal and inverted formats, so we don't need to create inverted templates.

And for Flan2021 tasks, the input inversions were already written into some of their templates options for the same task, rather than creating new input inverted tasks. For instance, look at this version of Few Shot RTE.

@shayne-longpre Thanks for the information. One more quick question - how were the T_NAME_TO_NUM_CASES (https://github.com/google-research/FLAN/blob/main/flan/v2/constants.py#L48) values computed for each of the datasets?

@AadSah this should be the size of each dataset as it already exists on Tensorflow Datasets. It's a quirk of seqio that we needed to calculate this in advance.