Support for Pydio Cells (MinIO based) server
Closed this issue · 17 comments
Does S3Drive support MinIO?
I got this error message: "The specified key does not exist".
My config: bucket name = data; s3 endpoint = mydomain.ext
The MinIO server log reports:
2023-04-13T08:51:31.764-0300 ERROR pydio.gateway.data Error while putting object:{"Id":"workspace.not.found","Code":404,"Detail":"Workspace .aainit.txt is not found","Status":"Not Found"} {"RemoteAddress": "X.Y.Z.0", "UserAgent": "MinIO (Unknown; Unknown) minio-dart/2.0.0", "HttpProtocol": "HTTP/1.1"}
Hi @jaimedelano,
Thanks for your report. Yes, S3Drive does support MinIO.
Can you let me know what platform/OS are you using an what's your client version (located at the bottom of the Login or About pages)?
@jaimedelano
Indeed there was an issue with our latest 1.1.5 release on Android. We've changed the S3 credentials validation method from write to read-only and that's failing on MinIO. We're pending 1.1.6 release which is going to address this issue, but now needs approval from Play Store which will likely take few hours from now.
I am only confused by the fact that you've: .aainit.txt in logs, which refers to a write validation method which was present until 1.1.4 version and replaced in 1.1.5 with read-only.
I will let you know once Android patch is live. In the meantime feel free to try out one of our desktop clients which being still 1.1.4 shall connect to MinIO without issue: https://s3drive.app/desktop
I have already tried the desktop app as well. That is why I get ".aainit.txt is not found"
Android 12 app version 1.1.5, here the error message is about failed authentication, something like "jwt rawIdToken verify: failed, trying next"
Hi @jaimedelano,
Android version 1.1.6 is now available.
I was wondering if your MinIO enforces region? In either case S3Drive shall connect correctly, it's just with region enforced S3Drive performs additional request to detect the region.
If region is enforced the login / validation workflow shall look like follows:
First request - region detection
Second request - read test on non-existent key
Third request - bucket listing
I am not exactly sure why are you getting JWT error message. If you're connecting directly to MinIO instance, there is no JWT involved at all.
If you long tap on version number (at the bottom) you will open app internal logs, perhaps there is more logs that we could analyze. I would appreciate if you could send more info / screenshots, so we can help.
If possible, can you please send some test credentials to the MinIO endpoint, so I can look at it directly? If you decide to do so, please send it to: tom@s3drive.app
Thanks,
Tom
App updated. Now I can list files on Android.
But I can neither upload nor download files.
I'm getting http 403 forbidden error code.
I have emailed you.
Thanks, I've received an e-mail and tried to use the details with the CLI clients first. It does seem that something is not entirely intact with the responses that this endpoint returns.
aws s3 ls
AWS_ACCESS_KEY_ID=<yourKeyId> AWS_SECRET_ACCESS_KEY=<yourSecret> aws --endpoint https://<yourEndpoint> s3 ls s3://data
# maximum recursion depth exceeded
aws s3api list-objects-v2
AWS_ACCESS_KEY_ID=<yourKeyId> AWS_SECRET_ACCESS_KEY=<yourSecret> aws --endpoint https://<yourEndpoint> s3api list-objects-v2 --bucket data
# maximum recursion depth exceeded while calling a Python object
On S3Drive I could list below location (personal-files/Fp1...-J.jpeg) which in fact causes a recursive error for native AWS client, as "personal-files" prefix is recursively nested within itself.
When trying to copy single key out of it, this is what I get:
aws s3 cp
AWS_ACCESS_KEY_ID=<yourKeyId> AWS_SECRET_ACCESS_KEY=<yourSecret> aws --endpoint https://<yourEndpoint> s3 cp s3://data/personal-files/Fp1...-J.jpeg /tmp/test.jpeg
# An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.
... also the odd thing is that when I query the results from S3Drive which handles the initial ls on root prefix correctly I get inconsistent ordering back.
I would never thought that MinIO can response such gibberish. Can you please let me know what MinIO version it is? Did you manage to work successfully with this back-end with any other client?
This is a custom MinIO version used by Pydio Cells server.
It works with (Windows 11) Cyberduck desktop client, Mountain Duck mapped drive, /n Software S3 Drive 2022 (mapped drive), and Android app Cloudly - S3 Manager.
Thanks for letting me know that's indeed interesting.
If you don't mind leaving these credentials valid for a few more days I will get back to this when I have little bit more time.
No problem.
Cyberduck S3 (HTTPS) plugin works perfectly with Pydio's MinIO S3 server.
https://github.com/iterate-ch/cyberduck/tree/master/s3/src/main/java/ch/cyberduck/core/s3
I think I have found the cause of the problem:
- Your app is not sending the authorization headers, instead it is sending the authorization as a query string;
- The HTTP header X-Amz-Content-Sha256 is not being sent. Is this necessary?
***** CORRECT REQUEST by Cyberduck *****
2023/04/29 17:25:57.617 info http.log.access.log0 handled request {"request": {"remote_ip": "HOST_IP_ADDRESS", "remote_port": "46574", "proto": "HTTP/1.1", "method": "GET", "host": "HOST_IP_ADDRESS:8008", "uri": "/data/personal-files/VNC-Viewer-7.1.0-Windows-64bit.exe", "headers": {"X-Real-Ip": ["172.71.227.137"], "X-Forwarded-For": ["CLIENT_IP_ADDRESS, 172.71.227.137"], "X-Forwarded-Proto": ["https"], "User-Agent": ["Cyberduck/8.5.8.39606 (Windows 10/10.0.22000.0) (amd64)"], "Connection": ["Upgrade"], "Range": ["bytes=8756224-10945279"], "Cf-Visitor": ["{"scheme":"https"}"], "Cf-Ipcountry": ["BR"], "Accept-Encoding": ["gzip"], "Date": ["Sat, 29 Apr 2023 17:24:53 GMT"], "X-Amz-Content-Sha256": ["e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"], "X-Amz-Date": ["20230429T172453Z"], "Cf-Ray": ["7bf93f44381316ee-REC"], "Authorization": ["AWS4-HMAC-SHA256 Credential=P8vaxNhVes264RNoZVArY61cZtWSfEKkjkQ6qtygWRU.a2XHKCUyHumRLq3vxqQLzZi3MpgUppNC9_2fQ6Ov7n8/20230429/us-east-1/s3/aws4_request,SignedHeaders=date;host;range;x-amz-content-sha256;x-amz-date,Signature=7e9a04f518603d74bda33519699202e894ec86003826c81a1ce98008b1364bdb"], "Cf-Connecting-Ip": ["CLIENT_IP_ADDRESS"], "Cdn-Loop": ["cloudflare"]}, "tls": {"resumed": true, "version": 771, "cipher_suite": 49199, "proto": "", "server_name": ""}}, "user_id": "", "duration": 0.036400583, "size": 2189056, "status": 206, "resp_headers": {"X-Xss-Protection": ["1; mode=block"], "Content-Length": ["2189056"], "Last-Modified": ["Sat, 29 Apr 2023 15:39:16 GMT"], "Accept-Ranges": ["bytes"], "Server": ["Caddy", "MinIO"], "Content-Type": ["application/vnd.microsoft.portable-executable"], "Etag": [""cbe34eaf904d28b6f17d39ce96c4ddbf""], "Vary": ["Origin"], "X-Amz-Request-Id": ["175A77D2B1821357"], "Content-Range": ["bytes 8756224-10945279/10945280"], "Date": ["Sat, 29 Apr 2023 17:25:57 GMT"], "Content-Security-Policy": ["block-all-mixed-content"]}}
***** FAILED REQUEST by S3Drive *****
2023/04/29 17:27:36.201 error http.log.access.log0 handled request {"request": {"remote_ip": "HOST_IP_ADDRESS", "remote_port": "56832", "proto": "HTTP/1.1", "method": "GET", "host": "HOST_IP_ADDRESS:8008", "uri": "/data/personal-files/VNC-Viewer-7.1.0-Windows-64bit.exe?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=P8vaxNhVes264RNoZVArY61cZtWSfEKkjkQ6qtygWRU.a2XHKCUyHumRLq3vxqQLzZi3MpgUppNC9_2fQ6Ov7n8%2F20230429%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230429T165355Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=3c6acd35387cb89a9e057e8467577e6469a62f2108f3908d60129bfdfbd09834", "headers": {"X-Real-Ip": ["172.71.227.142"], "X-Forwarded-For": ["CLIENT_IP_ADDRESS, 172.71.227.142"], "X-Forwarded-Proto": ["https"], "Cdn-Loop": ["cloudflare"], "Accept-Encoding": ["gzip"], "Cf-Connecting-Ip": ["CLIENT_IP_ADDRESS"], "Connection": ["Upgrade"], "Cf-Ipcountry": ["BR"], "Cf-Ray": ["7bf941b1fa6516ed-REC"], "Cf-Visitor": ["{"scheme":"https"}"]}, "tls": {"resumed": true, "version": 771, "cipher_suite": 49199, "proto": "", "server_name": ""}}, "user_id": "", "duration": 0.002873172, "size": 385, "status": 403, "resp_headers": {"X-Amz-Request-Id": ["175A77E9A7967054"], "Content-Length": ["385"], "Date": ["Sat, 29 Apr 2023 17:27:36 GMT"], "Content-Security-Policy": ["block-all-mixed-content"], "Content-Type": ["application/xml"], "X-Xss-Protection": ["1; mode=block"], "Accept-Ranges": ["bytes"], "Server": ["Caddy", "MinIO"], "Vary": ["Origin"]}}
Hi @jaimedelano,
Sorry for not getting back to you earlier, but we're slammed recently.
Thank you for attaching these raw requests, it's helpful to analyze the potential issue.
As such the: X-Amz-Content-Sha256 header is required for Signature V4: https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html
We're using V4 Signatures for most of the operations (listing, copy, delete, version fetch etc.), except download/upload.
The main difference between these two requests that you've attached are that Cyberduck seem to be using native AWS GetObject call: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html to fetch the object contents.
In S3Drive we're using presigned URLs: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html for maximum compatibility with native upload/download clients on all major platforms. If we haven't done it that way we would be dependent on AWS SDK which wasn't working for us on all platforms.
Presigned URLs are supported fully by all major S3 providers including self-hosted ones e.g. MinIO.
I am not sure why Pydio Cells doesn't seem to support it. If I have the right sources they seem to be using very early MinIO version from June 2017: https://github.com/pydio/minio-srv
I am saying this all based on the requested that you've attached. Personally I was finding few other problems with this S3 endpoint (recursive loops and non-deterministic order being one of them)... and if native AWS S3 tooling doesn't cooperate with it, then it's not a great indicator.
In theory we could support native Get/Put (without presigned URLs) available as a Settings option. Our goal is to support as many S3 providers as possible, but given our limited resources we won't be available to satisfy every edge case. Our roadmap for 2023 (https://s3drive.canny.io/) is quite ambitious and I am sorry to say that, but that's not something we would be able to fit in this year. Since we're feedback driven project... if there are more people requesting we can definitely revisit our priorities.
We're handling custom implementations outside of our priority list through our Enterprise plan: https://s3drive.app/pricing, if it's something that you would be interested in then we could discuss details.
I hope to look at this endpoint again to confirm all that, but I am failing to find time.
Can't promise anything, but perhaps I will be able to revisit this topic after Early May holiday.
Thanks !
I managed to bypass that limitation.
My Pydio installation is running behind a local Nginx proxy.
So I can edit query strings and HTTP headers.
My solution: convert X-Amz query strings into http authentication headers.
http {
# .....
map $args $arg_amz_credential {
~*&?X-Amz-Credential=([^&]*)%2F(\d+)%2Fus-east-1%2Fs3%2Faws4_request $1/$2/us-east-1/s3/aws4_request;
default "";
}
map $args $arg_amz_signature {
~*&?X-Amz-Signature=([^&]*) $1;
default "";
}
map $args $arg_amz_date {
~*&?X-Amz-Date=([^&]*) $1;
default $http_x_amz_date;
}
map $request_uri $request_uri_path {
"~^(?P<path>[^?]*)(\?.*)?$" $path;
default $uri;
}
}
server {
# .....
location / {
#The next (4) lines maintain compatibility with other clients, e.g. Cyberduck
set $authorization $http_authorization;
set $x_amz_content_sha256 $http_x_amz_content_sha256;
set $date_time $http_date;
set $query_strings "$is_args$args";
#Workaround for Pydio's broken MinIO server
if ( $args ~ "^X-Amz" ) {
set $authorization "AWS4-HMAC-SHA256 Credential=$arg_amz_credential,SignedHeaders=host,Signature=$arg_amz_signature";
set $query_strings "";
set $date_time $date_gmt;
set $x_amz_content_sha256 "UNSIGNED-PAYLOAD";
#set $x_amz_content_sha256 "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855";
}
#proxy_set_header Auth-Debug-Info "$arg_amz_date *** $date_time *** $x_amz_content_sha256 **** $args *** $authorization";
proxy_set_header Authorization $authorization;
proxy_set_header X-Amz-Content-Sha256 $x_amz_content_sha256;
proxy_set_header Date $date_time;
proxy_set_header X-Amz-Date $arg_amz_date;
#Strip query strings from presigned URL
proxy_pass "https://127.0.0.1:8008$request_uri_path$query_strings";
}
}
That's quite impressive chunk of work.
I need to say I wasn't aware that it's "just" a matter of rewriting params to headers.
Thank you for documenting and sharing this. It will definitely be useful for us if we ever were to maintain compatibility with clients not supporting presigned URLs.
If you have any other issues / feature requests regarding S3Drive I would be happy to help out. Thanks !
hi guys,
Just jumping in the conversation, as a pydio developer :-) We did have to fork minio for licensing issue, but we are using a decently recent version, and maintaining it.
I'm surprised to read that we do not support presigned URLs, as our own Javascript frontend is using exactly that for uploads and downloads, as you can see for example here https://github.com/pydio/cells/blob/main/frontend/front-srv/assets/gui.ajax/res/js/core/http/PydioApi.js#L365
But maybe we do not support a specific presigned format (maybe presigned v2 vs. presigned v4?) any hints on this? Our own javascript-generated presigned do not contain 'X-Amz-' parameters indeed.
It's hard for me to tell exactly without verifying it. V2 signatures are deprecated by AWS around ~June 24, 2020: https://aws.amazon.com/blogs/aws/amazon-s3-update-sigv2-deprecation-period-extended-modified/
According to doc: https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingAWSSDK.html#UsingAWSSDK-move-to-Sig4
the minimum JS version that supports the V4 is: 2.68.0 and you seem to be using newer version 2.847.0 anyway:
https://github.com/pydio/cells/blob/b0e73cfb659107b9578b9efa9ac84caaf99c335c/frontend/front-srv/assets/gui.ajax/package.json#LL18C1-L18C1
however you might be required to enable it specifically: https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingAWSSDK.html#specify-signature-version
var s3 = new AWS.S3({signatureVersion: 'v4'});It may be possible that you're supporting presigned URLs but with V2 signatures only (hence that's why JS code works fine), but at this stage it's a blind guess.
According to AWS doc, the V4 signature might be passed as headers or query string.
https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html
Authentication Methods
You can express authentication information by using one of the following methods:
HTTP Authorization header – Using the HTTP Authorization header is the most common method of authenticating an Amazon S3 request.... Query string parameters – You can use a query string to express a request entirely in a URL. In this case, you use query parameters to provide request information, including the authentication information. Because the request signature is part of the URL, this type of URL is often referred to as a presigned URL. You can use presigned URLs to embed clickable links, which can be valid for up to seven days, in HTML. For more information, see [Authenticating Requests: Using Query Parameters (AWS Signature Version 4)](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html).
V4 Signature format for query parameters: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html
===
@cdujeu
Regardless of the above you may benefit from supporting fully V4 presigned URLs anyway and drop V2 due to security problems (e.g. V2 token might be permanent / non-expirable).
Speaking about licenses I think MinIO forked from Apache2 to AGPLv3 not that long ago, it's hard for me to believe that their latest Apache2 version wouldn't support V4 presigned URLs.
In principle it also looks like Pydio Cells does support V4 signatures, it's just it doesn't seem to accept them in the query string, which is somewhat strange.
I know nothing about Pydio Cells, but Is there any layer before it reaches MinIO that might deliberately or by mistake strip some of the query parameters?
Hey @tomekit
I agree that our MinIO version is very probably supporting presigned v4, but we do have some middlewares that may pollute the signal ;-)
I'll look into that more deeply, thanks for all the details!
Best
I consider this resolved now. Thank you all for participating.