Gunicorn Server Refuses Connections

Question

Gunicorn Server Refuses Connections

Closed this issue 4 years ago · 6 comments

Hi, I am attempting to set up UW SAML authentication for a project at my research lab at the UW. I decided to try to use this repo as it seems a lot simpler than setting up the SP myself. I was wondering if someone who helped create the repo could give me a few pointers. We are using an amazon ec2 instance to run a lab-wide DBMS platform and we want to authenticate that our users are part of UW. We are running ubuntu 18.04.

I am unsure what the typical setup process would be for this use case as I have never done it before.

I have cloned the repo onto the ec2 instance, and used docker-compose up --build to set everything up, but have run into several issues. I have not set up a SECRET_KEY variable as of yet but I assume that I should be able to run everything without that. I have also not registered my service with UW yet as I am unsure at what step in the process I should do this.

The output of docker-compose up --build:
Starting nginxsamlproxy_nginx_1 ...
Starting nginxsamlproxy_saml_1 ...
Starting nginxsamlproxy_nginx_1
Starting nginxsamlproxy_saml_1 ... done
Attaching to nginxsamlproxy_nginx_1, nginxsamlproxy_saml_1
nginx_1 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
nginx_1 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
nginx_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
nginx_1 | 10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
nginx_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
nginx_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
nginx_1 | /docker-entrypoint.sh: Configuration complete; ready for start up
saml_1 | [2021-03-09 21:07:11 +0000] [1] [INFO] Starting gunicorn 20.0.4
saml_1 | [2021-03-09 21:07:11 +0000] [1] [INFO] Listening at: http://0.0.0.0:6000 (1)
saml_1 | [2021-03-09 21:07:11 +0000] [1] [INFO] Using worker: eventlet
saml_1 | [2021-03-09 21:07:11 +0000] [8] [INFO] Booting worker with pid: 8
saml_1 | [2021-03-09 21:07:11,349] ERROR in app: Generating burner SECRET_KEY for demo purposes

First of all I am confused about several of the nginx lines in the example nginx configuration files:
auth_request /saml/status;
auth_request_set $auth_user $upstream_http_x_saml_user;
error_page 401 = @login_required;
proxy_pass http://saml:5000/status;

I am new to working with web servers, so the http://saml:5000 threw me off. I assumed that this url is meant to point at the flask app and is somehow aliased to saml, so I replaced it with http://0.0.0.0:5000, but I am unsure if that is intended.

My other issue is with the actual flask app / gunicorn server. Part of the service I am working on uses port 5000 so I expected the gunicorn server to be unable to start due to this. However, when I ran 'docker-compose up --build', it started but threw a 302 found error whenever I tried to access the nginx proxy at port 443, 'curl localhost:443/'.

To fix this I changed the port from 5000 to 6000 and this avoids the 302 error but the nginx proxy cannot connect to the flask app so it throws an internal server error:
nginx_1 | 2021/03/09 21:15:04 [error] 21#21: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", subrequest: "/saml/status", upstream: "http://0.0.0.0:6000/status", host: "localhost:443"
nginx_1 | 2021/03/09 21:15:04 [error] 21#21: *1 auth request unexpected status: 502 while sending to client, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", host: "localhost:443"
nginx_1 | 172.18.0.1 - - [09/Mar/2021:21:15:04 +0000] "GET / HTTP/1.1" 500 177 "-" "curl/7.58.0" "-"

I am sure these issues are due to me missing some crucial step in the setup process and any pointers to deal with them would be much appreciated.

Answer 1 · 2021-03-09T22:32:21.000Z

Hi!
I have not used the Docker compose rig before but I think I can give some pointers.

If you want to change the flask app to use port 6000 it needs to be changed on line 19 in the Dockerfile so that flask itself knows to listen on a different port. And then it should also change in the nginx config.
The "saml" in "http://saml:5000" refers to the Docker compose name. The name "saml" resolves to the address of of the container named "saml". As long as you're running under Docker compose you can use that name.
In your config snippet it says proxy_pass http://saml:5000/status but that should be proxy_pass http://saml:5000/ (or :6000) as shown in the example config in the repo.
And for completeness I'll mention that the other proxy_pass reference in the example http://secure:5000/ refers to whatever your target app is. This should be changed to point to whatever you are protecting and not pointed to the nginx-saml-proxy flask app.

I think in general changing the port to 6000 in the Dockerfile and in the nginx config you should be able to get past this current error.

--Eric

Answer 2 · 2021-03-09T23:58:12.000Z

HI!

Thanks so much responding and for clarifying the Docker compose name stuff. As for editing the docker file I believe that I have done so already, it shows in the output I included that the flask app is listening on port 6000.

The core issue is that the gunicorn output states that it is listening on 6000 but refuses all connections both from something like curl 0.0.0.0:6000 or when I cause the nginx server to try to connect with curl localhost:443/.

When I do use the nginx server, it gives this output:

nginx_1 | 172.18.0.1 - - [09/Mar/2021:23:58:57 +0000] "GET /login HTTP/1.1" 500 177 "-" "curl/7.58.0" "-"
nginx_1 | 2021/03/09 23:58:57 [error] 21#21: *1 connect() failed (111: Connection refused) while connecting to upstream,
client: 172.18.0.1, server: , request: "GET /login HTTP/1.1", subrequest: "/saml/status", upstream: "http://0.0.0.0:6000/status", host: "localhost:443"
nginx_1 | 2021/03/09 23:58:57 [error] 21#21: *1 auth request unexpected status: 502 while sending to client, client:
172.18.0.1, server: , request: "GET /login HTTP/1.1", host: "localhost:443"

Answer 3 · 2021-03-10T01:27:21.000Z

Yes, I see now, that you had it listening on 6000 correctly.

Does it have the same problem using http://saml:6000/ in the location /saml/ configuration block?

Would you mind sharing your whole nginx.conf? I feel like I don't have the whole picture.

-Eric

Answer 4 · 2021-03-10T02:17:53.000Z

Sure, I basically copied the config in the test folder. I just want to get something working before I configure it for my specific purposes. All I've changed are the 2 proxy_pass lines which are 0.0.0.0 instead of saml because I get 302 found errors when I switch back.

When I use /saml I get a 301 moved permanently error regardless of the proxy_pass line, either http://saml:6000 or http:/0.0.0.0:6000. With / It throws a 302 error using http://saml:6000 but the internal server error copied above occurs with http:/0.0.0.0:6000.

server {
listen 443;
ssl_certificate /etc/nginx/ssl/cert.pem;`
ssl_certificate_key /etc/nginx/ssl/key.pem;
root /usr/share/nginx/html;

location / {
auth_request /saml/status;
auth_request_set $auth_user $upstream_http_x_saml_user;
error_page 401 = @login_required;
proxy_pass http://0.0.0.0:6000/status;
}

location /secure {
auth_request /saml/status/group/uw_it_all;
error_page 401 = @login_required;
alias /usr/share/nginx/html;
}

location /2fa {
auth_request /saml/status/2fa;
error_page 401 = @2fa_required;
alias /usr/share/nginx/html;
}

location /saml/ {
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Prefix /saml/;
proxy_pass http://0.0.0.0:6000/;
}

location @login_required {
return 302 https://$http_host/saml/login$request_uri;
}

location @2fa_required {
return 302 https://$http_host/saml/2fa$request_uri;
}
}

But also even if my server.conf is bad wouldn't I be able to connect on 0.0.0.0:6000 with curl or some other method anyways?

Answer 5 · 2021-03-10T17:52:54.000Z

Thanks for the full config. I'm also better rested. :)

http://0.0.0.0 is meaningless. That isn't an IP address, its a network. One can listen on 0.0.0.0 which means listen on every interface but it isn't valid IP to specify a target. So you'll need to use "saml" or otherwise specify the address of the other container. I think this explains why you get an internal server error.

Now when using http://saml:6000 you get a 302. That is normal. The 302 isn't an error, its a redirect. That's what you should get back in the SAML flow. The 302 target is defined in the login_required location at the bottom of the config. Getting a 302 means you're probably as working as you can be until you've gotten your SP registered with the Idp.

I will attempt to describe the flow through this config file:

Client asks for https://localhost:443
Client request is routed into the "location /" block
Server checks to see if client is allowed by making sub-request to /saml/status (defined by auth_request)
If /saml/status returns 200 then client is authorized so proceed through the block and forward the request to the address specified in proxy_pass. DONE. (This would be your app but for testing could be http://saml:6000/status)
if /saml/status returns error 401 then client is NOT authorized. In that case the "error_page" directive sends the client request to @login_required block.
In the @login_required block the client is issued a 302 redirect to the login URL.
Client responds to redirect and asks for https://localhost:443/saml/login/
Client request is routed into the "location /saml/" block
Headers are added and the client request is passed to http://saml:6000/login (the flask app)
The flask app decodes and infers some things and then sends another 302 redirect to the client, this time redirecting to idp.u.washington.edu
Client visits idp.u.washington.edu, logs in and on success sends the client a 302 redirect back to https://localhost:443/
Loop back to step 2 and this time the user should be authorized.

Hopefully this is helpful in understanding the config.

-Eric

Answer 6 · 2021-03-10T19:33:39.000Z

Thanks so much. I knew it must be something super simple I was missing. So the 302 is expected since I haven't completed registration for this as of yet.

Above I mention that when I use localhost:443/saml myself I get a 301 moved permanently but I assume thats because in the flow you describe, I would never actually use localhost:443/saml myself it would just be used as part of the process I start up by going to localhost:443 and entering the / location block.