LoadDataset needs cache directory option

[klacansky]

I think it would be useful to have an optional keyword argument cache_dir to specify directory where to cache files (if they are accessed remotely).

[scrgiorgio]

'LoadDataset' is cache_dir agnostic and I think it must remain this way.
There are users/services loading the same datasets with different access policies i.e .'LoadDataset' is ununware of how binary data is retrieved.
I think this issue is more related/connected to Access/DiskAccess/CloudAccess and must be changed accordingly.

[klacansky]

I am thinking what a user in Python may do. I would probably want to specify the cache directory per dataset, not per access or read. I would not have default cache directory because if somebody does regular backups, they may not want to backup cache directory in their home directory. There could also be issues with global cache if two datasets have the same name and different content (how is it resolved currently?).

import OpenVisus as ov
dataset = ov.LoadDataset('test.idx', cache_dir='.')
data = dataset.read(x=[100,101])
data = dataset.read(y=[101,102])
...

[scrgiorgio]

I am OK with something like this if internally the dataset uses the given value to set a "*default_cache_dir*": void Dataset::setDefaultCacheDir(String value) {.... this->default_cache_dir=value;} This way: - DiskAccess or IdxDiskAccess can use the Dataset::*default_cache_dir* *IIF *there is no value specified in their XML visus.config file (there are cases where OpenVisus cache data in different places depending on some multiplex logic). - it's clear it's a default value for other classes. The dataset is not in charge of how/if the data is cached. What do you think?

…

On Sun, Dec 4, 2022 at 2:58 PM Pavol Klacansky ***@***.***> wrote: I am thinking what a user in Python may do. I would probably want to specify the cache directory per dataset, not per access or read. import OpenVisus as ov dataset = ov.LoadDataset('test.idx', cache_dir='.') data = dataset.read(x=[100,101]) data = dataset.read(y=[101,102]) ... — Reply to this email directly, view it on GitHub <#220 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJTZP7KR7F6FWABHJRHCPTWLUHYBANCNFSM6AAAAAASTDAFOI> . You are receiving this because you commented.Message ID: ***@***.***>

-- Giorgio Scorzelli Director of Software Development - Center for Extreme Data Management, Analysis and Visualization (CEDMAV) Director of Software Development - National Science Data Fabric (NSDF)