clessig/atmorep

source and target not written the same way in BERT mode

Opened this issue · 2 comments

Not sure if this is an issue, but it does pose a problem when wanting to write general functions that handle target, pred, and source. What I mean by this is that data in target is accessed via:

target[f"{field}/sample={sample:05d}/ml={level:05d}"] 

But data in source is accessed via:

source[f"{field}/sample={sample:05d}"] 

Where you have to access the ml by index and not by the actual level number. ml in source can be read by doing:

for key in source[f"{field}/sample=00000/ml"]:
    print(key)

but you cannot access level-wise data in source by doing source[f"{field}/sample=00000/ml/96"]. Instead you have to use hard coded indexes

source should be changed to match target. Can you please propose a fix and open a PR for it.

Sure, I'll add it to my todo list, and get on it soon