protocol stripping problem
Closed this issue · 1 comments
rabernat commented
I am trying to read a zarr store over plain http
This file exists:
curl http://s3.wasabisys.com/era5-single-reanalysis/.zmetadata
However, I can't access it from an HTTP filesystem
import fsspec
http_url = 'http://s3.wasabisys.com/era5-single-reanalysis'
mapper = fsspec.get_mapper(http_url)
mapper['.zmetadata']
MissingSchema: Invalid URL 's3.wasabisys.com/era5-single-reanalysis/.zmetadata': No schema supplied. Perhaps you meant http://s3.wasabisys.com/era5-single-reanalysis/.zmetadata?
During handling of the above exception, another exception occurred:
KeyError Traceback (most recent call last)
<ipython-input-16-690c397eba4f> in <module>
2 http_url = 'http://s3.wasabisys.com/era5-single-reanalysis'
3 mapper = fsspec.get_mapper(http_url)
----> 4 mapper['.zmetadata']
/srv/conda/envs/notebook/lib/python3.7/site-packages/fsspec/mapping.py in __getitem__(self, key, default)
76 if default is not None:
77 return default
---> 78 raise KeyError(key)
79 return result
80
KeyError: 's3.wasabisys.com/era5-single-reanalysis/.zmetadata'
According to @martindurant on gitter
fs.get_mapper() works as expected, but get_mapper does not, it strips the protocol instead of deferring to the class implementation of what to do with the protocol - this is only an issue for HTTP, I think.
--- a/fsspec/mapping.py
+++ b/fsspec/mapping.py
@@ -157,4 +157,4 @@ def get_mapper(url, check=False, create=False, **kwargs):
cls = get_filesystem_class(protocol)
fs = cls(**kwargs)
# Removing protocol here - could defer to each open() on the backend
- return FSMap(path, fs, check, create)
+ return FSMap(url, fs, check, create)
This feels like the sort of thing that should be covered by the test suite.
rabernat commented
Sorry this should have been an fsspec issue, not intake. Could someone transfer?