lofar-astron/RMextract

Allow for local path to IONEX data

Closed this issue · 4 comments

Hi @maaijke,

It'd be really useful to be able to handle local (or other exotic server) versions of IONEX data. This is often required on HPCs where internet access is locked down - especially for FTP connections! Thankfully, urllib can already handle this with e.g. server=file://path/to/the/data.

Along side this, it'd be handy to be able to pass a formatter for the file naming convention used. This is currently somewhat hard-coded in the get_urllib_IONEXfile function. My suggestion would be to have an optional formatter argument that is called to appropriately format the URL strings, with a lookup table of known formatters e.g.

KNOWN_FORMATTERS = {
    "ftp.aiub.unibe.ch": 
        lambda server, year, dayofyear, prefix, yy: 
            f"{server}/CODE/{year:4d}/{prefix.upper()}{dayofyear:03d}0.{yy:02d}I.Z",
    "cddis.gsfc.nasa.gov": 
        lambda server, year, dayofyear, prefix, yy: 
            f"{server}/gnss/products/ionex/{year:4d}/{dayofyear:03d}/{prefix}{dayofyear:03d}0.{yy:02d}i.Z",
    "igsiono.uwm.edu.pl":
        lambda server, year, dayofyear, prefix, yy: 
            f"{server}/data/ilt/{year:4d}/igrg{dayofyear:03d}0.{yy:02d}i",
}

@tjgalvin and myself have been playing with implementing these changes and along the way we also found that, due to the speed of a local cache, some race conditions can occur. We've put in some checks to catch these.

Would you be open to a PR that brings in these changes? All being well, the current behaviour should be completely preserved.

Hi @maaijke - would you be interest this at all?

Hi Alec, I think something went wrong with my earlier reply, sorr for that. YEs I am interested, especially in the formatter!
Local IONEX files is already supported, but if you have a better way of doing that I am happy to have a look at that

Ok cool - I'd be curious to see what is currently supported! I'll open a draft PR so you can inspect the modifications I've been using locally. I'm keen to see what you think :)

Hey @maaijke - thanks for the merge! Just wondering, will you push a new release of RMextract along with this?