MinIO Rock-on web-UI cannot be accessed
Closed this issue · 20 comments
As originally reported by forum user benrw67, and further confirmed/detailed by @Hooverdan96, we have a failure to access the webUI of the MinIO Rock-on:
https://forum.rockstor.com/t/minio-rock-on-issue/8141
It seems that Docker image has changed how it works so we will probably need to adjust our Rock-on definition. The documentation reads:
MinIO Server comes with an embedded web based object browser. Point your web browser to http://127.0.0.1:9000 to ensure your server has started successfully.
NOTE: MinIO runs console on random port by default if you wish choose a specific port use --console-address to pick a specific interface and port.
... with a docker run example as follows:
docker run \
-p 9000:9000 \
-p 9001:9001 \
minio/minio server /data --console-address ":9001"
It seems we need to bind a second port and add the new --console-address ":9001" command argument.
Maybe @MikeMcPherson can also help clarify what's needed and what is not, considering all of the changes, since he was the original author of the Rockon json file.
The one challenge I see is that the --console-address is a command line parameter, which means that a port would have to be "fixed" to correspond to the embedded parameter. It seems to me that the environment variable names have changed, too? So, not sure what would have to be done (needs to be done) to make this work.
I’ll look in to this as soon as I can. It may be tomorrow; today is pretty full…
I think this will be a simple fix. It will take me longer to reconstruct my test environment than to make the fix... :-) Busy week, so probably tomorrow.
Much appreciated Mike!
I'm finally getting back to this. It turned out to be a little more complicated than expected. I believe the fix requires specifying four arguments to be passed into the container. The command as it arrives in the container should look like "server --console-address :9001 /data". The Rock-on Profile documentation seems to say that the cmd_arguments array can consist of only a single pair of strings. In fact, more than one pair will be processed, but the order isn't preserved.
Another complication is that "9001" is a port number that the user can specify at Rock-on install time. I need to figure out how to substitute the port that was specified into the cmd_arguments array.
I'm looking at the rockstor-core code, but if someone knows the answer that would be great.
Thanks a lot for all the effort, @MikeMcPherson!
I'm unfortunately short on time but wanted to provide some feedback:
Another complication is that "9001" is a port number that the user can specify at Rock-on install time. I need to figure out how to substitute the port that was specified into the cmd_arguments array.
I'm looking at the rockstor-core code, but if someone knows the answer that would be great.
That is a difficult one to implement, indeed, and I fear we do not have a way to do this automatically at the moment. The only thing I can think of is to explicitly mention in the webUI port description (tooltip) that the port MUST be set to 9001 for it to work. If he MinIO image supports it, we could also choose a very exotic port number unlikely to interfere with any other Rock-on we have (past, present, and future), and set that port as well in the server --console-address:XXXX argument.
This is similar to our current limitation on providing a valid webUI link for the Rock-ons who use host networking and have conflicting port numbers (see rockstor/rockstor-core#2209 for details, if interested).
There may be a better option but none that comes to mind for the moment.
Note if your interested in understanding how the Rock-ons framework is implemented, we have a wiki for that on our forum:
https://forum.rockstor.com/t/rock-on-framework-implementation/6288
The command as it arrives in the container should look like "server --console-address :9001 /data". The Rock-on Profile documentation seems to say that the cmd_arguments array can consist of only a single pair of strings. In fact, more than one pair will be processed, but the order isn't preserved.
Thanks for bringing out the issue of the order... that's something I do not remember, for instance. If you have a reproducer, feel free to create an issue for that on our rockstor-core repo as this would represent a bug, indeed. In the meantime, you should be able to circumvent this by putting everything as just one argument:
"cmd_arguments": [ ["server --console-address :9001 /data", ""] ],This is from memory, but I believe this should help... let us know if you find time to give it a try... I will on my end once I can.
I had a similar thought right after I wrote my previous post. [ ["server --console-address :9001 /data", ""] ] doesn't work; MinIO complains about an invalid command. [ ["server", "--console-address :9001 /data"] ] does get passed into the container and accepted by MinIO, but MinIO doesn't recognize the option for some reason. So, cmd_arguments problem solved (I think), but another one to chase.
I'll create an issue for the out-of-order thing after I get this problem fixed.
Regarding what I'm thinking of as variable substitution in Rock-on Profiles... Bummer, that would be useful, but I can see how it ratchets up the complexity. On a related note, is it possible to set the port numbers without prompting the user doing the Rock-on install? If that were possible, then I could just force the standard ports. If not, I might be able to accomplish the same thing using the "opts" section of the profile.
On a related note, is it possible to set the port numbers without prompting the user doing the Rock-on install? If that were possible, then I could just force the standard ports. If not, I might be able to accomplish the same thing using the "opts" section of the profile.
Yes, you definitely could put the -p XXXX:XXXX in the "opts", but the problem is that Rockstor would not know that it needs to create a link to the webUI. We indeed use the ports key of the JSON for that:
https://github.com/rockstor/rockon-registry#port-object
{
"description": "<A detailed description of this port mapping, why it's for etc..>",
"label": "<A short label for this mapping. eg: Web-UI port>",
"host_default": <integer: suggested port number on the host. eg: 8080>,
(optional)"protocol": "<tcp or udp>",
(optional)"ui":true, not needed if false
}
That's why the solution I can think of at the moment is to define a port "exotic" enough not to conflict with another one so that the default we set will not be changed by the user (and Rockstor will not need to suggest a change to the user). It really is a workaround but all I can think of for the moment.
MinIO complains about an invalid command. [ ["server", "--console-address :9001 /data"] ] does get passed into the container and accepted by MinIO, but MinIO doesn't recognize the option for some reason.
I just had a quick look at their doc and noticed a detail that could be useful; please accept my apologies if this is something you've already tried:
Their example uses the following arguments:
https://docs.min.io/docs/minio-docker-quickstart-guide.html
server /data --console-address ":9001"
Besides the different order between the /data and --console-address ":9001" bits, I noticed the double quotes for the value of the --console-address option. Maybe you could try adding them to your json and escaping them? I can't remember the proper way to do so unfortunately from memory, but something like the following maybe? (or try with single quote for ':9001'
"cmd_arguments": [["server", "--console-address \":9001\" /data"] ],Thanks, I had seen that, and no apologies necessary. I've tried it both ways, doesn't seem to make a difference. And the position of "/data" either after "server" or at the end doesn't seem to make a difference. (The MinIO documentation shows it both ways in different places.) I have fired up a MinIO container on a different, non-Rockstor, server and everything works perfectly.
The really puzzling thing is that when I exec into the running container and do a "ps", the command that started MinIO is:
minio server --console-address ":9001" /data
And yet, it doesn't see the flag. Baffled.
Curious indeed... I'll try to see if I can find some time to give that a try and report.
In the meantime, did you know you can turn on DEBUG mode if you are using an ISO or rpm Rockstor install?
/opt/rockstor/bin/debug-mode onand then you should be able to see DEBUG log statements in:
/opt/rockstor/var/log/rockstor.log
Among these DEBUG statement will be the docker run ... command when you install a Rock-on as was parsed/understood by Rockstor based on the JSON file and the user input during the Rock-on install wizard.
Hope this helps, and thanks again for all your time and effort here, it is more than greatly appreciated!
Didn't know how to turn on debug, but was wondering. Thanks!
More data points. I wasn't getting anywhere with ["server" , "--console-address :9001 /data" ]. (As it turns out, MinIO will accept the port either quoted or not; it insists on the colon, oddly.) It almost worked, but something funny is happening with string processing in between the Rock-on profile and the container. I'll try to characterize it well enough to create an issue.
I decided to have another try at:
"cmd_arguments": [
[ ":9001" , "/data" ],
[ "server" , "--console-address" ]
],
It worked on the first try! And on the second. But then not on the third, fourth, and fifth. As far as I can tell, the pairs of strings are being processed in random order. Sometimes they result in the command "server --console-address :9001 /data" and sometimes in ":9001 /data server --console-address".
So, I'm kinda stuck. Do you want me to create an issue for all of this? How do we let people know that the MinIO Rock-on is unavailable until we figure this out?
Mike
I had a similar thought right after I wrote my previous post. [ ["server --console-address :9001 /data", ""] ] doesn't work; MinIO complains about an invalid command. [ ["server", "--console-address :9001 /data"] ] does get passed into the container and accepted by MinIO, but MinIO doesn't recognize the option for some reason. So, cmd_arguments problem solved (I think), but another one to chase.
yeah, it sees the command argument as one long string with a ' at the beginning and the end ', so it interprets it differently (i.e. not a valid sub-command for MinIO message), compared to you breaking it out (but with the problem of inconsistent sequencing).
@MikeMcPherson Thanks from me also for all your fathoming here.
Re:
How do we let people know that the MinIO Rock-on is unavailable until we figure this out?
This issue itself does that currently.
Do you want me to create an issue for all of this?
That would be great actually. And given it looks to be a short fall in our Rock-on json capability/parsing such an issue would belong in our rockstor-core repo:
https://github.com/rockstor/rockstor-core
Thanks again for pushing through this one. We definitely have some work to do in the Rock-on capabilities although @FroggyFlox has made some impressive headway on that front with the likes of rocknets. So any improvements you can offer up would be much appreciated. And exactly defining a short fall via a focused issue or two would be great. You can link back to this issue for a concrete example of where such improvements will play-out.
Incidentally we do have a little used facility to treat some Rock-ons in a special way. That may provide a work around in the interim. Take a look at for example the following outdated treatment needed for our now removed discourse Rock-on:
And here is an example of how such custom substitutes are resourced, if they are found to exist:
I'm less familiar with the Rock-on code currently than @FroggyFlox but I just wanted to pop this in as a potential work around in rockstor-core for a given named Rock-on that may need special treatment. Always better to extend generic capability though, as long as it doesn't add unreasonable complexity for definite edge cases.
Thanks! I'll get started on a simple test and an issue.
My apologies for taking so long to get this fixed. It turned out to be more complex than expected, and life threw me some family emergencies that consumed my attention for a while. I think it is fixed now.
Good news, the official MinIO Docker image is now multi-arch. I don't have a free Raspberry Pi 4 to test, and they're scarce as hen's teeth right now. Perhaps someone with a Pi 4 sitting around can confirm that it works.
Confirmed that the MinIO Rock-on also works properly on the Raspberry Pi 4.
Mike
@MikeMcPherson Thanks for the Pi4 / ARM64 proof of function. Good to know we are working as expected on that front.