zihaohe123/wiki-enhanced-stance-detection

Regarding VAST dataset

Opened this issue · 5 comments

kgarg8 commented

Hi,

I am curious why you used lowercased processed text (with punctuations and stopwords removed) for VAST dataset. This could be a bit unnatural for transformer models.

https://github.com/zihaohe123/wiki-enhanced-stance-detection/blob/983decae7ae6165351d4168e05cdeca446c22a84/src/datasets.py#L194C22-L194C28

E.g.,
text_s column looks like:

absolutely 's needs defined regulated use currently word natural used food products totally confusing meaningless. clearly trying imply item healthy possibly organic see food manufacturers like frito lay campbell 's products labelled natural alone set alarms seems. ;-)

whereas the original tweet (Post column) was:

Absolutely it's needs to be defined and regulated in its use, as currently the word 'natural' when used on food products is totally confusing and meaningless. Clearly they are trying to imply the item is 'healthy' or possibly 'organic', but when you see food 'manufacturers' like Frito-Lay or Campbell's with products labelled 'natural', that alone should set off alarms that all is not what it seems. ;-)
zihaohe123 commented

Hi, thanks for your interest in the paper.
I can't remember exactly what I chose to use "text_s". You can try both the processed text and original text and compare which one leads to better performance (and share your findings with me if possible). Thanks!

kgarg8 commented

Regarding the use of the two columns, here are my findings:
image

The results vary slightly but there is no clear winner it seems.

Another followup question,
My reproduced results are little lower than yours. Do you have any suggestions?
FYI, I tuned the learning rate little bit as you see above, but it doesn't help much.

Reporting the results with the default lr for VAST:

VAST | Zero-shot | Few-shot | Overall
Original | 75.3 | 73.6 | 74.5
Reproduced | 70.53 | 74.03 | 72.36

Any comments?

zihaohe123 commented

Thanks for sharing the results!

About your reproduced results on VAST, could you share with me your experimental setup?

kgarg8 commented

Hey, sorry for not being to share my setup since I am working in a private repo and the work is unpublished still.

I did more hyperparameter tuning and got closer to the original results.

VAST | Zero-shot | Few-shot | Overall
WS-BERT (original) | 75.3 | 73.6 | 74.5
WS-BERT (reproduced) | 72.16 | 75.02 | 73.62

One thing I observe in the experiments is that Few-Shot > Zero-Shot for this repository. I am curious if you could double-check whether you swapped zero-shot and few-shot by mistake.

Thanks.

zihaohe123 commented

Thanks for sharing the results. I'm quite sure that I didn't swap the results of zero-shot and few-shot.