[encoding,Â] top1000.dev.tsv

[Albert-Ma]

I read the content from top1000.dev.tsv, but met many encoding problems.

For example:

Right text in queries.tsv:what complication is a potential danger associated with continuous iv infusions?

Problem text in top1000.dev.tsv: what complication is a potential danger associated with continuous iv infusions?

There are many places appear this problem which text contain  in top1000.dev.tsv.

[Albert-Ma]

@dfcf93

[spacemanidol]

Hey,

You are correct. We seem to have a bug in the topfiles. I will go ahead and update the files at some point but in the meantime each line also includes the QID and PID(Question ID) and (Passage ID) which you can join with the collection.tsv and the query files. Sorry for the delay.

[Albert-Ma]

Hi,
And also found this problem in train.triples file, but not in collection. I think it's a encoding problem while parsing web document.
Thanks for the reply.

[spacemanidol]

Just not to leave you in the dark I am going to update all the documents to fix this issue I am just currently away on paternity leave. I should be able to get to it mid may

[Albert-Ma]

Just not to leave you in the dark I am going to update all the documents to fix this issue I am just currently away on paternity leave. I should be able to get to it mid may

That's very kind of you.

[spacemanidol]

This has been fixed and uploaded for the top1000 dev and eval. The top1000 train and the train triples will follow shortly.

for context I used https://ftfy.readthedocs.io/en/latest/

import ftfy
import sys

filename = sys.argv[1]
with open(filename, 'r') as f:
with open('cleaned' + filename, 'w') as w:
for l in f:
w.write(ftfy.fix_text(l))