Duplicated layers loaded with s3fs & fsspec 2023.12.2
jluethi opened this issue · 2 comments
I'm running into an issue when setting up fresh environments with the napari-ome-zarr plugin and could trace it back to an interaction with fsspec==2023.12.2 & s3fs==2023.12.2.
Using different OME-Zarr datasets (for example this public dataset: https://zenodo.org/records/10257149), I see the same layers being loaded multiple times. Sometimes a layer is loaded 3x, sometimes 46x.
The same dataset has different behaviors whether I load it from my local SSD (more duplicates) or slower remote storage.
The issue is reproducible for local storage and remote shares, on Windows & Mac, with napari 0.4.18, 0.4.19 & 0.5.0dev versions from the main branch.
The thing that fixes it is reverting to an older version of fsspec & s3fs, e.g. by doing: pip install s3fs==2023.9.2
I couldn't trace the underlying issue yet. I don't know if it's a bug in fsspec, s3fs or the way ome-zarr py is using the libraries. Just thought I'd document it here for the moment in case other napari-ome-zarr users come across it. Downgrading the fsspec & s3fs versions does solve it (that was a wasted afternoon to figure out, so maybe it saves someone else time to find this here ;))
I've downloaded the dataset linked, and it seems to open fine for me locally. I'm using napari 0.4.19, fsspec 2024.5.0. Could you update your package versions and see if this is still an issue?
Hey @dstansby
Thanks for pinging me. I can actually not reproduce this issue anymore with new environments. Neither with fsspec/s3fs 2024.5.0, the newer 2024.6.1. But to my surprise, neither with the older 2023.12.2 that was making an issue before either.
I had months where only the older versions like 2023.9.2 were avoiding this issue repeatedly. But given that I can't reproduce this now anymore, I'll be closing this issue.