pangeo-forge/pangeo-forge-recipes

`full_target.fs.protocol` returns multiple protocols

norlandrhagen opened this issue · 6 comments

I'm trying to run a reference recipe and get a failure in WriteCombinedReference. In my recipe, full_target.fs.protocol returns a list ['s3', 's3a'], which causes a TypeError TypeError: unhashable type: .

Wondering how this should be handled if fsspec returns multiple protocols.

Happy to open up a PR that adds the optional kwarg remote_protocol that is infered from fsspec if not explicitly stated, but not sure if that's a good solution.

Would love to hear your thoughts @cisaacstern!

@norlandrhagen seems like fsspec takes the first protocol in case of multiple protocols, so we could do the same, like this:

https://github.com/fsspec/filesystem_spec/blob/b8aeb13361e89f22f323bbc93c8308ff2ffede19/fsspec/spec.py#L1410-L1414

Sounds great! Where should that logic live?

I would just put it right above where we currently have full_target.fs.protocol, since IIUC we only use that once in the code currently.

@norlandrhagen did this get fixed?

I think once this is merged, this can be closed

Closed by #632 💫