Unable to run s3ninja behind a reverse proxy
mcayland opened this issue · 8 comments
I've been testing s3ninja (docker tag scireum/s3-ninja:7.0
) and attempting to run it behind a reverse proxy on a non-root URL, but I can't get the UI to appear correctly.
I've tried updating application.conf
so that product.wondergemRoot
points to /my/new/uri/ui
but from what I can see there are still hardcoded references to the /ui
uri within the application.
Is there a configuration option that can be used to allow s3ninja to run correctly in a reverse proxy setup?
Hello @mcayland 👋 Many thanks for getting in touch. In fact, the /ui
portion is just the auxiliary user frontend. The real S3 interaction (i.e. s3cmd
or similar tools, and API implementations) happens directly on the root level. We strictly follow the scheme used by AWS, and there is no way to change this (currently).
Please consider that S3 Ninja is intended for development purposes only. It is not able to handle huge amounts of data, and lacks the usual means required to operate a server for wider/frequent access in terms of speed, reliability and security. That is, it is not made for settings where a reverse proxy is usually required.
We generally recommend to use S3 Ninja within a Docker container for simpler operation, and only during development for cost-effective testing. In a production system, please use a different system or web service.
Sorry for not having a more satisfactory answer… 😔
@jakobvogel thanks for the response!
I do understand that S3 Ninja is for development purposes only: let me explain my use case. I am currently working on a prototype website that allows users to upload content to be made available to subscribers via S3. The rest of the site sits behind a reverse proxy with authentication so that the client can see the current work in progress.
Since S3 Ninja isn't covered by the same authentication domain then someone who knows the domain and port of the instance would be able to browse the copyrighted content directly via /ui
without requiring a password.
In regards to s3cmd
i'm currently using the AWS PHP SDK to access the S3 storage: for development sites I used the use_path_style_endpoint
option which allows you to specify an arbitrary endpoint url instead of using the AWS host-based naming scheme. It is this feature that would enable me to S3 ninja behind a reverse proxy in this way.
I do understand that this may not be your primary use case, but I thought it would be worth mentioning why I think such a feature would be useful in future for developers in a similar situation.
Hello @mcayland! Many thanks for clarifying this. I have asked our IT administrator about his thoughts on this problem and will get back if we have an alternative suggestion. In any case, we will consider your idea for future development. 👍
@jakobvogel I thought it was worth adding an update here for anyone else who finds themselves in a similar situation: my realisation was that I could easily switch the application over to use a test configuration which doesn't require use_path_style_endpoint
so in fact all I needed was to be able to allow access to the UI through the frontend proxy to allow developers to check access logs and the bucket contents.
Unfortunately the templates included with s3ninja have hard-coded absolute URLs causing the links and stylesheets to be incorrect, but I was able to get a basic UI working by using the following nginx configuration on the frontend proxy:
#
# s3ninja at /s3ninja/
#
# s3ninja assets
location /s3ninja/assets/ {
proxy_pass http://container-s3ninja:9000/assets/;
proxy_set_header Accept-Encoding "";
sub_filter_once off;
sub_filter_types text/css;
sub_filter '/assets/' '/s3ninja/assets/';
}
# s3ninja UI
location /s3ninja/ui {
proxy_pass http://container-s3ninja:9000/ui;
proxy_set_header Accept-Encoding "";
sub_filter_once off;
sub_filter_types text/html;
sub_filter '"/assets/' '"/s3ninja/assets/';
sub_filter '"/ui' '"/s3ninja/ui';
}
To summarise: it is possible to use s3ninja on a container network whilst forwarding access to the UI through a frontend reverse proxy using this configuration. However the use of sub_filter
to fix up the absolute URLs seems quite heavy-handed, so would it be possible to use relative URLs in the UI so that this extra workaround is no longer required?
Hello @mcayland — Many thanks for sharing this. Please excuse my late reply, these are busy release times here. In theory, it is possible, of course. However, that would require quite a lot of time to do. The routing is done via our Sirius framework which basically knows multiple methods for dispatching data given a requested URL. In particular, re-routing everything below assets/
with relative paths would be challenging, and it would require changes in the library as well. Considering the special nature of your use case, and the effort required to change the paths, I would rather prefer to leave the paths as they are and stay with your rerouting solution via the configuration file. It looks to be the more solid solution.
Are there still paths not covered by your configuration?
The routing is done via our Sirius framework which basically knows multiple methods for dispatching data given a requested URL. In particular, re-routing everything below
assets/
with relative paths would be challenging, and it would require changes in the library as well.
That makes sense. Presumably this is the root URL controlled by product.wondergemRoot
? I'm just thinking that the easy fix here would be to relocate the assets from /assets
to /ui/assets
using a dispatcher that runs text/html pages through the existing templating engine as is already done in src/main/resources
instead of src/assets
. It would then be a case of simply doing a find/replace on /assets
to @wondergemRoot/ui/assets
so that the correct path gets injected into the page.
Are there still paths not covered by your configuration?
I did a quick check and the only thing I spotted that doesn't work so far is adding a new bucket through the UI. However this is not important for our setup here, since when deploying to our test environment, the buckets (and content) are pre-seeded with data from a .tar.gz file.
Hello @mcayland! Unfortunately, the structure of Sirius is more complicated. Effectively, Sirius wraps netty. Whenever a request comes in, a sorted list of WebDispatcher instances is traversed, and asked to handle it. One of them is the AssetsDispatcher which handles everything beneath /assets
. Moving the assets to a different directory requires to change the existing dispatcher, or a new one. This is feasible, of course, see the S3Dispatcher in this project, but I am reluctant to do it as it bloats the code only to support a not-so-common setting.
wondergem comes in at a later stage. Another dispatcher is the ControllerDispatcher that is checked with lower priority. Effectively, this dispatcher handles routes that are defined via methods marked with @Routed
annotations residing in instances of Controller. See, for instance, the NinjaController in this project. Within such a method, the template engine tagliatelle/pasta/noodle can be used to generate output of any sort, but mostly HTML pages. In order to serve pages in a unified manner, we have two template libraries, the older wondergem with table-based layout, and the more recent tycho with card-based layout.
Bucket creation works via /ui/<bucket>?create
. If you can open a bucket, you should be able to create one as well. 🤔 Do you encounter an error when attempting to create a bucket?
Closing this due to inactivity.