`LD_PRELOAD`-based virtual s3 FS for systems without FUSE module installed
Closed this issue · 4 comments
Tell us more about this new feature.
Yes, LD_PRELOAD-based fs-related libc/stdio call interception is not ideal (and breaks wherever syscalls invocations are compiled in directly), but for many situations of simple data processing e.g. from Python, it could be sufficient
The need for this is for situations where one does not have FUSE kernel module available or does not have sudo permissions
Hi, thanks for making this feature request. If I'm understanding your issue correctly, you're looking for something that can be injected into existing software that intercepts file read requests and redirects them to S3?
If so, it's unlikely that we will be wanting to include this in our repositories, but if there are any small features we could add to unblock your usecase, feel free to open another feature request for them.
you're looking for something that can be injected into existing software that intercepts file read requests and redirects them to S3?
Precisely - mainly for adopting simple legacy scripts for S3 data placement in restricted environments (without FUSE kernel module / without root permissions) and without code modifications. LD_PRELOAD-based interceptions of open/read/seek calls can often be sufficient for this goal
If so, it's unlikely that we will be wanting to include this in our repositories, but if there are any small features we could add to unblock your usecase, feel free to open another feature request for them.
It appears that the FUSE aspect is well-separated from the S3FS impl itself, which is nice. Maybe examples of using the S3FS without FUSE would be helpful, but that should be sufficient for implementing the LD_PRELOAD-based interceptions as out-of-tree.
Maybe examples of using the S3FS without FUSE would be helpful
Sure, as a matter of fact, some of our tests use S3Filesystem directly which implements methods which are convenient to use to serve fuse requests. For example, this is how you read a file.
Though, it's important to note that this API is internal and is subject to change, so please keep that in mind. Thanks for the interest in our project!