SetPort always setting port to 443
Opened this issue · 31 comments
Hi ;
When setting RPAF_SetPort On , apache always sets port to 443, even if you're requesting directly apache on port 80 with HTTP, and no headers added, from an IP which isn't in RPAF_ProxyIPs.
this is very confusing for PHP apps, as they rely on different informations to guess if you're coming in HTTP or HTTPS.
Are you setting the X-Port or X-Forwarded-Port header on the reverse proxy? please show your configuration.
yes, from an nginx front, serving only HTTPS requests.
here are some headers received while requesting HTTPS (real client IP is applied):
HTTPS = on
HTTP_X_FORWARDED_PROTO = https
HTTP_X_FORWARDED_FOR = xx.xx.xx.xx
HTTP_X_FORWARDED_HTTPS = on
REMOTE_ADDR = xx.xx.xx.xx
SERVER_PORT = 443
SERVER_PROTOCOL = HTTP/1.0
SERVER_SIGNATURE =
Apache Server at test.foo.com Port 443
the same while requesting directly apache with HTTP (real client IP is applied too):
HTTPS =
HTTP_X_FORWARDED_PROTO =
HTTP_X_FORWARDED_FOR =
HTTP_X_FORWARDED_HTTPS =
REMOTE_ADDR = xx.xx.xx.xx
SERVER_PORT = 443
SERVER_PROTOCOL = HTTP/1.1
SERVER_SIGNATURE =
Apache Server at test.foo.com Port 443
As suspected, you have not set X-Port or X-Forwarded-Port, you need to tell mod_rpaf what port the request came in on.
ok I restart :
my arch : an nginx (listening 443) acting as HTTPS reverse proxy for apache (listening 80)
so I've 2 use cases :
- #1 : HTTPS request : browser talks with nginx, which forwards the request to apache through HTTP
- #2 : HTTP request : browser talks directly to apache
so the deal is for apache to distinguish correctly request coming from browsers and others coming from nginx using RPAF.
here are results for use case #1 :
HTTPS = on
HTTP_X_FORWARDED_PROTO = https
HTTP_X_FORWARDED_FOR = xx.xx.xx.xx
HTTP_X_FORWARDED_HTTPS = on
HTTP_X_FORWARDED_PORT = 443
HTTP_X_PORT = 443
REMOTE_ADDR = xx.xx.xx.xx
SERVER_PORT = 443
SERVER_PROTOCOL = HTTP/1.0
SERVER_SIGNATURE =
Apache Server at test.foo.org Port 443
and use case #2 :
HTTPS =
HTTP_X_FORWARDED_PROTO =
HTTP_X_FORWARDED_FOR =
HTTP_X_FORWARDED_HTTPS =
HTTP_X_FORWARDED_PORT =
HTTP_X_PORT =
REMOTE_ADDR = xx.xx.xx.xx
SERVER_PORT = 443
SERVER_PROTOCOL = HTTP/1.1
SERVER_SIGNATURE =
Apache Server at test.foo.org Port 443
coming directly with a browser, there is no additional headers. but why does RPAF believes it's an HTTPS request an sets port to 443 ?
my rpaf.conf :
RPAF_Enable On
RPAF_SetHostName On
RPAF_ProxyIPs 127.0.0.1
RPAF_Header X-Forwarded-For
RPAF_SetHTTPS On
RPAF_SetPort On
Hi ;
I've done some tests with many VM, with the config provided in the README.
It seems that this happen only on Apache 2.2 (ubuntu 12.04).
On Apache 2.4 (ubuntu 14.04 and later), it works like it should : setting port to 443 only when request coming from HTTPS reverse proxy.
Is it supposed to work with Apache 2.2 ?
correction : it doesn't work with apache 2.4 either.
it seems that RPAF is supposed to work only with reverse proxies, no direct connection from clients.
- first connection : direct HTTP : port is 80
- second connection : proxified HTTPS : port is 443
- third connection : direct HTTP : port is 443
any further direct HTTP connection will always have port affected to 443.
it seems like Apache keeps values from previous connections, and with the many conditional "return DECLINED;" it doesn't affect port correctly.
@olaulau - It should work with direct connections also, I see the PR that looks to address this but I believe it can be done without adding the additional overhead that this fix will include. Thanks for reporting the issue and concise information on how to replicate it.
it should, but it doesn't work.
what overhead in my PR ? can you explain me ? I've just replaced a return with an assignment, you think executing the rest of the module code is heavy ?
Executing the rest of the code is not particularly heavy, but unnecessary.
With Apache 2.4, I noticed that RPAF_SetHTTPS works as expected, however RPAF_SetPort gives inconsistent results. For example, I can request the same URL via https, and eventually the $_SERVER[SERVER_PORT] variable will report 80, while the following all display correctly:
$_SERVER[HTTP_X_PORT] = 443
$_SERVER[HTTP_X_HTTPS] = on
$_SERVER[HTTPS] => on
HTTPS is offloaded via Nginx, which proxies via HTTP through Varnish to the backend which runs Apache 2.4.18.
I set each of the following in Varnish after doing a 'strings' on the module to see what it checks for:
HTTPS
X-Forwarded-For
X-Forwarded-Host
X-Host
Host
X-Forwarded-HTTPS
X-HTTPS
X-Forwarded-Proto
X-Forwarded-Port
X-Port
However, even when they're all in place as expected, the inconsistency still shows if you make the requests rather quickly. The only way around this that I found was by changing the Host header in Varnish before passing it to Apache to include the port, then set UseCanonicalName Off in Apache. This results in an expected SERVER_PORT based on the port from the Host header, however issues with apps can likely arise. I would love to avoid piggybacking on the Host header and UseCanonicalName Off instead of RPAF_SetPort giving consistent results.
@gnif - Are the SERVER_PORT and env HTTPS values somehow being cached or reference incorrect pointers (or something else possibly along those lines)? I have a relatively straight forward stack offloading SSL through Nginx, which then goes through Varnish via HTTP, and finally Apache 2.4.18 backend. The setup is along these lines:
(client/HTTPS) -> (SSL offload - Nginx) -> (balancer - Varnish) -> (http server - Apache)
(client/HTTP) -> (balancer - Varnish) -> (http server - Apache)
In my tests all requests were being passed directly to Apache by Varnish, and no caching was involved along the way. The following headers were all set accordingly before the request ever reached Apache:
X-Forwarded-Proto
X-Forwarded-For
X-Port
X-HTTPS
The tests I ran were rather simple, and consisted of two concurrent while loops to quickly see the issues side by side. For example
screen1 (https request loop):
while true;do date;curl -s https://domain.tld/looptest/headers.php|egrep -i "X-Forwarded|X-Https|server.*https|SERVER_PROTOCOL|SERVER_PORT|REMOTE_ADDR|SERVER_ADDR|SERVER_NAME|HTTP_X_UA_DEVICE|HTTP_X_HTTPS|attack";echo;sleep 0.5;done
screen2 (http request loop):
while true;do date;curl -s http://domain.tld/looptest/headers.php|egrep -i "X-Forwarded|X-Https|server.*https|SERVER_PROTOCOL|SERVER_PORT|REMOTE_ADDR|SERVER_ADDR|SERVER_NAME|HTTP_X_UA_DEVICE|HTTP_X_HTTPS|attack";echo;sleep 0.5;done
The headers.php file is just a simple headers dump script
While running the loops, the following inconsistencies would appear:
Test sample #1 - Showing both https and http forwarded requests reporting SERVER_PORT 80/HTTPS off/REQUEST_SCHEME https
screen1 (https URL requests with invalid SERVER_PORT/HTTPS env):
Sat Aug 13 18:43:06 CEST 2016
X-Forwarded-For: 1.1.1.1<br>
X-Forwarded-Proto: https<br>
X-Forwarded-For-Src: Trusted<br>
X-Forwarded-Https: on<br>
X-Forwarded-Port: 443<br>
X-Https: on<br>
$_SERVER[HTTP_X_FORWARDED_PROTO]; => https<br>
$_SERVER[HTTP_X_SCHEME]; => https<br>
$_SERVER[HTTP_X_FORWARDED_HTTPS]; => on<br>
$_SERVER[HTTP_X_HTTPS]; => on<br>
$_SERVER[HTTP_X_UA_DEVICE]; => pc<br>
$_SERVER[HTTPS]; => off<br>
$_SERVER[SERVER_NAME]; => domain.tld<br>
$_SERVER[SERVER_ADDR]; => 2.2.2.2<br>
$_SERVER[SERVER_PORT]; => 80<br>
$_SERVER[REMOTE_ADDR]; => 2.2.2.2<br>
$_SERVER[REQUEST_SCHEME]; => https<br>
$_SERVER[SERVER_PROTOCOL]; => HTTP/1.1<br>
screen2 (http URL requests which maintained expected SERVER_PORT and HTTPS headers, however there's an invalid REQUEST_SCHEME set):
Sat Aug 13 18:43:06 CEST 2016
X-Forwarded-For: 1.1.1.1<br>
X-Forwarded-For-Src: Direct<br>
X-Forwarded-Proto: http<br>
X-Forwarded-Https: off<br>
X-Forwarded-Port: 80<br>
X-Https: off<br>
$_SERVER[HTTP_X_FORWARDED_HTTPS]; => off<br>
$_SERVER[HTTP_X_HTTPS]; => off<br>
$_SERVER[HTTP_X_UA_DEVICE]; => pc<br>
$_SERVER[HTTPS]; => off<br>
$_SERVER[SERVER_NAME]; => domain.tld<br>
$_SERVER[SERVER_ADDR]; => 2.2.2.2<br>
$_SERVER[SERVER_PORT]; => 80<br>
$_SERVER[REMOTE_ADDR]; => 2.2.2.2<br>
$_SERVER[REQUEST_SCHEME]; => https<br>
$_SERVER[SERVER_PROTOCOL]; => HTTP/1.1<br>
Test sample #2 - Showing both https and http forwarded requests reporting SERVER_PORT 443/HTTPS on/REQUEST_SCHEME https
screen1 (https URL requests which maintained SERVER_PORT, but failed to set HTTPS while REQUEST_SCHEME seems to of been set to https):
Sat Aug 13 18:40:44 CEST 2016
X-Forwarded-For: 1.1.1.1<br>
X-Forwarded-Proto: https<br>
X-Forwarded-For-Src: Trusted<br>
X-Forwarded-Https: on<br>
X-Forwarded-Port: 443<br>
X-Https: on<br>
$_SERVER[HTTP_X_FORWARDED_PROTO]; => https<br>
$_SERVER[HTTP_X_SCHEME]; => https<br>
$_SERVER[HTTP_X_FORWARDED_HTTPS]; => on<br>
$_SERVER[HTTP_X_HTTPS]; => on<br>
$_SERVER[HTTP_X_UA_DEVICE]; => pc<br>
$_SERVER[HTTPS]; => off<br>
$_SERVER[SERVER_NAME]; => domain.tld<br>
$_SERVER[SERVER_ADDR]; => 2.2.2.2<br>
$_SERVER[SERVER_PORT]; => 443<br>
$_SERVER[REMOTE_ADDR]; => 2.2.2.2<br>
$_SERVER[REQUEST_SCHEME]; => https<br>
$_SERVER[SERVER_PROTOCOL]; => HTTP/1.1<br>
screen2 (http URL requests with invalid SERVER_PORT/REQUEST_SCHEME variables):
Sat Aug 13 18:40:43 CEST 2016
X-Forwarded-For: 1.1.1.1<br>
X-Forwarded-For-Src: Direct<br>
X-Forwarded-Proto: http<br>
X-Forwarded-Https: off<br>
X-Forwarded-Port: 80<br>
X-Https: off<br>
$_SERVER[HTTP_X_FORWARDED_HTTPS]; => off<br>
$_SERVER[HTTP_X_HTTPS]; => off<br>
$_SERVER[HTTP_X_UA_DEVICE]; => pc<br>
$_SERVER[HTTPS]; => off<br>
$_SERVER[SERVER_NAME]; => domain.tld<br>
$_SERVER[SERVER_ADDR]; => 2.2.2.2<br>
$_SERVER[SERVER_PORT]; => 443<br>
$_SERVER[REMOTE_ADDR]; => 2.2.2.2<br>
$_SERVER[REQUEST_SCHEME]; => https<br>
$_SERVER[SERVER_PROTOCOL]; => HTTP/1.1<br>
Worth noting:
- Both loops start showing the same values at the same time, for the same duration suggesting a common internal value is being shared/referenced
- $_SERVER[REQUEST_SCHEME] is consistently set as 'https', even when SERVER_PORT and HTTPS are both set to 80/off
- Gracefully restarting Apache causes the SERVER_PORT value to reset in both screens. However they continue showing the same duplicate/invalid SERVER_PORT/HTTPS values for a while after.
- Delays between the dupe SERVER_PORT value changes are random. Sometimes both values change within 1 second, other times they remain "stuck" for as long as 1min
- There are however short/long periods of time where all headers except for REQUEST_SCHEME via HTTP are set correctly for both HTTPS/HTTP requests.
After disabling KeepAlive in Apache, and implementing @olaulau's pull (https://github.com/gnif/mod_rpaf/pull/44/files) there don't appear to be any more discrepancies, other than with REQUEST_SCHEME which is still set to https regardless of SERVER_PORT/HTTP referencing non-https. However, disabling KeepAlive is not a proper fix as it hinders performance. Please consider actually fixing this issue sooner than later.
bump - Any progress or intentions to fix?
There are intentions to fix, but like most unpaid open source projects, it has to wait until I can find the time to debug the problem, or someone else graciously does so and provides a fix.
Totally understandable. Where do we send our coffee tips to? and how much would it take :D
I've fallen in same issue.
I have nginx listening on 443 and 80 and an apache with rpaf on 8081.
It's working ok, but sometimes when mod_rewrite is generating a 301-redirect browser gets urls like
http://domain.tld:443/some/other/location. Nginx responds with error 400 on such requests
After disabling KeepAlive in Apache this issue is gone, but now some redirects change scheme from http to https. This is not so terrible, but i'm not sure that it cannot change https to http.
I also ran into that same issue with http://<domain.tld>:443 redirects, and ended up just stripping the :443 from http:// links. If anyone's interested in some dev time.. I'll pay :)
Looks like i have found reasons. Digging deeper now to be sure.
After a couple of hours of digging around i have found two reasons:
- My nginx configuration says
proxy_pass http://localhost:8081;
localhost resolves to ip4 and ipv6 addresses:
$host localhost
localhost has address 127.0.0.1
localhost has IPv6 address ::1
My apache RPAF_ProxyIps don't include ::1
So, sometimes nginx decides to use ipv6 address, RPAF checks allowed proxies list and stop processing, because ::1 is not allowed. The problem is in server port/scheme, which are changed by previous RPAF-processed request. If previous request was https, and current - http , mod_rewrite will produce redirect to https://
- Second problem is in forgetting X-HTTPS header for ssl frontend, but setting X-Port to 443.
When RPAF processes such request it set server port to 443, and scheme to http. If request produces normal "200 OK" response you will not notice any problem, but apache internal variables a set to http/443.
If next request will trigger the first problem (disallowed ::1 proxy) and mod_rewrite will produce a redirect, mod_rewrite will use bad values for port/scheme (http://<>:443) from previous request.
SOLUTION:
- For ssl-frontend set
proxy_set_header X-Port 443;
proxy_set_header X-HTTPS on; - For non-ssl-fronted set
proxy_set_header X-Port 80; - Either use ip address as backend url:
proxy_pass http://127.0.0.1:8081
or if you use hostname like 'localhost' check if it can resolve to ipv6-address and put it to RPAF_ProxyIps:
RPAF_ProxyIps 127.0.0.1 ::1
@elik-ru What happened with your fixes from earlier and fork at https://github.com/elik-ru/mod_rpaf? Regardless of setting all of the expected X headers, if Apache has KeepAlive on, you will get inconsistencies with the HTTPS env as shown above. Do you still have that patch you were working on? Did you get a chance to test @gnif's suggestion with apr on the NULLs?
It was wrong, so i've deleted my fork. I still have a local copy (with @gnif 's suggestions), but it's not helping really.
The only thing affected by
apr_table_set(r->subprocess_env , "HTTPS" , "on");
is environment variable HTTPS, passed to CGI application. It does not affect mod_rewrite behavior.
I think KeepAlive option only make worse the problem above, but is not the source itself.
I tried to integrate latest mod_rpaf functionality into mod_remoteip addressing the stuff with rewrites.
Alas, it cannot be easily compiled as separate module 'cause it is a full scale Apache patch, affecting mod_ssl, mod_remoteip and mod_nw_ssl modules.
You can find patch along with the description here: https://alex-at.net/blog/apache-mod_remoteip-mod_rpaf
If you don't need instructions, here is just the patch: https://alex-at.net/media/blogs/blog/quick-uploads/p10/httpd-2.4-remoteip-rpaf.patch
The only thing preventing compilation as module is impossibility to hook ssl_is_https globally though. Any suggestions on how to do it are welcome, I would be very grateful if I can get rid of this mod_ssl patching part.
@AlexAT Awesome news! I'll give it a shot over the weekend. Any chances in getting it pushed upstream?
@AndreiG6 Well, regarding upstream, it needs to be tuned to make this ssl_is_https function a hook first (for now it's patched directly) and then tested enough (I mean deep enough conformity testing to clearly describe what and how it affects in detail). Will probably make a few tweaks to it and try to offer it, but wonder (and heavily doubt) if HTTPS state enforcing via specific hook will be accepted.
Regarding applicability, I'm already running it in production for some time to facilitate HTTPS-capable distributed shared hosting behind haproxy (haproxy is providing HTTPS and Apache is HTTP only) and it's doing its job fine (all the features used).
Please share your experience if possible as well. Not only if you find some drawbacks or something that can be improved, but also if it runs just fine.
@AlexAT My apologies for not replying sooner, my schedule got a bit hectic. I currently run CloudLinux with EasyApache4, which has an RPM based Apache/PHP deployment and unfortunately manually patching would raise concerns with updates. Seeing as this fixes a long standing issue for lots of users, including redirect loops for those using Cloudflare's Flexible SSL option, I submitted your patch for CloudLinux to review and hopefully deploy as they tend to see the bigger picture. Thanks again for the contribution!
Thanks @AndreiG6
To be honest, run on production environment (and possibly distro) besides my one company hosting is the best experience I can expect some opinion/feedback/possible acceptance from. Please tell once you have some news :)
Also eager to check and fix stuff if any funny case goes.
Their L3 support escalated it up to the developers for review! (case id EA4D-26) So far so good :)
For anyone following, Cloudlinux ended up taking my suggestion and added the mod_remoteip patch from https://alex-at.net/blog/apache-mod_remoteip-mod_rpaf in to their EasyApache builds (EA4D-26) - https://www.cloudlinux.com/cloudlinux-os-blog/entry/beta-easyapache-4-updated-1-31
Cool! Please check if the latest revision of the patch is used, one nasty bug specific for simultaneous trusted/untrusted proxy access was fixed. Otherwise, I've found no more outstanding issues and atm running it in production for thousands of shared hosts since end of 2017 already.
2018-01-09: important bugfix: fixed possible port and HTTPS flag changes propagation between requests if server is accessed by both trusted internal proxies and untrusted proxies or clients directly (thanks to vladimir.umnov for reporting it and providing a test case)
@AndreiG6
By glancing over the code, I can confirm ea-apache24-2.4.33-5.el7.cloudlinux.3.src.rpm is indeed using the latest version of the patch with the bugfix from 2018-01-09 as a base one. So no worries there.
Awesome. I asked them earlier to confirm as well. Thanks for the follow-up!