source and target not written the same way in BERT mode
Opened this issue · 2 comments
sbAsma commented
Not sure if this is an issue, but it does pose a problem when wanting to write general functions that handle target
, pred
, and source
. What I mean by this is that data in target is accessed via:
target[f"{field}/sample={sample:05d}/ml={level:05d}"]
But data in source is accessed via:
source[f"{field}/sample={sample:05d}"]
Where you have to access the ml by index and not by the actual level number. ml in source can be read by doing:
for key in source[f"{field}/sample=00000/ml"]:
print(key)
but you cannot access level-wise data in source by doing source[f"{field}/sample=00000/ml/96"]
. Instead you have to use hard coded indexes
clessig commented
source should be changed to match target. Can you please propose a fix and open a PR for it.
sbAsma commented
Sure, I'll add it to my todo list, and get on it soon