fsspec/filesystem_spec

protocol stripping problem

Closed this issue · 1 comments

I am trying to read a zarr store over plain http

This file exists:

curl http://s3.wasabisys.com/era5-single-reanalysis/.zmetadata

However, I can't access it from an HTTP filesystem

import fsspec
http_url = 'http://s3.wasabisys.com/era5-single-reanalysis'
mapper = fsspec.get_mapper(http_url)
mapper['.zmetadata']
MissingSchema: Invalid URL 's3.wasabisys.com/era5-single-reanalysis/.zmetadata': No schema supplied. Perhaps you meant http://s3.wasabisys.com/era5-single-reanalysis/.zmetadata?

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
<ipython-input-16-690c397eba4f> in <module>
      2 http_url = 'http://s3.wasabisys.com/era5-single-reanalysis'
      3 mapper = fsspec.get_mapper(http_url)
----> 4 mapper['.zmetadata']

/srv/conda/envs/notebook/lib/python3.7/site-packages/fsspec/mapping.py in __getitem__(self, key, default)
     76             if default is not None:
     77                 return default
---> 78             raise KeyError(key)
     79         return result
     80 

KeyError: 's3.wasabisys.com/era5-single-reanalysis/.zmetadata'

According to @martindurant on gitter

fs.get_mapper() works as expected, but get_mapper does not, it strips the protocol instead of deferring to the class implementation of what to do with the protocol - this is only an issue for HTTP, I think.

--- a/fsspec/mapping.py
+++ b/fsspec/mapping.py
@@ -157,4 +157,4 @@ def get_mapper(url, check=False, create=False, **kwargs):
     cls = get_filesystem_class(protocol)
     fs = cls(**kwargs)
     # Removing protocol here - could defer to each open() on the backend
-    return FSMap(path, fs, check, create)
+    return FSMap(url, fs, check, create)

This feels like the sort of thing that should be covered by the test suite.

Sorry this should have been an fsspec issue, not intake. Could someone transfer?