github/janky

Enabling authentication - infinite OAuth Redirect Loop

Closed this issue · 6 comments

Iristyle commented

I'm sure I'm missing something obvious here, but I'm seeing a redirect loop when I enable organizational security for my Janky install.

Per the docs

To restrict access to members of a GitHub organization, register a new OAuth application on GitHub with the callback set to $JANKY_BASE_URL/auth/github/callback then set a few extra settings:

JANKY_SESSION_SECRET: Random session cookie secret. Typically generated by a tool like pwgen.
JANKY_AUTH_CLIENT_ID: The client ID of the OAuth application.
JANKY_AUTH_CLIENT_SECRET: The client secret of the OAuth application.
JANKY_AUTH_ORGANIZATION: The organization name. Example: "github".

So I register the OAuth app for my org, and plug in the the JANKY_AUTH_CLIENT_ID and JANKY_AUTH_CLIENT_SECRET for the app. JANKY_AUTH_ORGANIZATION is of course my org name (EastPoint) in this case. I randomly generate 16 characters for JANKY_SESSION_SECRET.

If I try to hit the url of my Janky install, it looks like it's working. The app immediately redirects to

https://github.com/login/oauth/authorize?response_type=code&client_id=JANKY_AUTH_CLIENT_ID&scope=repo&redirect_uri=https%3A%2F%2FMY.JANKY.SITE%2Fauth%2Fgithub%2Fcallback

I click Allow to authorize and the OAuth flow continues with a 302 to

https://MY.JANKY.SITE/auth/github/callback?code=THECODE

When the request goes to my server, I have one rack.session and I'm assigned a new one, and a 302 is sent which redirects me to the root of the site

https://MY.JANKY.SITE/

At this point, a request is sent back to https://github.com/login/oauth/authorize? and it loops. As far as I can tell, it would appear that Janky is either unable to lookup THECODE passed back from Github, or is having some sort of issue properly managing / associating the given rack.session with the OAuth flow that just occurred.

There's nothing in the logs (is there a way to turn on something more verbose?)

Is there some system-level dependency that I might be missing that could be preventing the calls back to GitHub by the Janky server from succeeding?

Some notes about my environment, for completeness - everything here checks out:

  • Not running on Heroku - running on EC2 - Ubuntu 12.04.1 LTS - have the following packages installed - sudo apt-get -y install curl ssh git-core git build-essential zlib1g zlib1g-dev libyaml-dev libxml2-dev libc6-dev autoconf automake libtool make postgresql postgresql-client postgresql-contrib libpq-dev ruby1.9.3 ruby1.9.1-dev rubygems1.9.1 ruby-bundler, and I've also done an install rubygems-update
  • Ruby 1.9.3 (yes, I have made sure that default Ruby is not 1.8 -- see here)
  • Postgres 9.1 -- I can connect to and see the relevant tables in the database that I've called janky_production - I've tested my connection configuration in irb and it's fine.
  • Running under Upstart - using Foreman to generate the Upstart scripts - RAILS_ENV and RACK_ENV are both set to production
  • I created an authorization for my JANKY_GITHUB_USER and have configured JANKY_GITHUB_STATUS_TOKEN with the token
  • JANKY_SESSION_SECRET is a randomly generated 16 character string from pwgen per the docs suggestion. Similarly JANKY_GITHUB_HOOK_SECRET has been generated in the same manner.
  • Hubot is configured properly with hubot-scripts and a HUBOT_JANKY_URL. I can verify in HipChat that his part is working. I can issue Hubot commands like ci? and the Janky result is properly parsed from https://MY.JANKY.SITE/_hubot/help as an example. However, if I issue a command like ci setup github/janky janky I get the response Error F7U12: Can't Setup.
  • The Url configured for JANKY_BUILDER_DEFAULT is using basic auth over HTTPS. This same basic url we've configured as a WebHook Url under the Service Hooks for the GitHub repo. The webhook is using the Jenkins token style like https://JANKY_GITHUB_USER:JENKINS_TOKEN@JENKINS..DOMAINcom/github-webhook/. The JANKY_BUILDER_DEFAULT looks identical, except that it doesn't have the github-webhook/ part.
  • I can see the user count increase for the GitHub application assigned to JANKY_AUTH_CLIENT_ID, and I can see the application listed as approved under my user account -- so I know that GitHub side has properly handled the OAuth request from Janky ... it's just that Janky seems to be missing something.

I've double and triple checked all my environment variables to make sure they match up, but it's possible I've done something stupid here.

On the double-checking front, I've also discovered an undocumented JANKY_AUTH_TEAM_ID at https://github.com/github/janky/blob/master/lib/janky.rb#L155 I have configured this to the integer id for the Developers team of my organization, and I am a member. I will add a new issue for updating the docs with a description of this.

A couple other last gory, related details -- again, only mentioned for completeness really.

  • I am running two other apps hosted on EC2 that have GitHub OAuth enabled - Jenkins and ErrBit. Both of these apps work totally fine, so I'm familiar with the process of setting up OAuth. All of our users are automatically logged into these apps if they are logged into GitHub.
  • Technically speaking, at the network infrastructure level, these apps (Jenkins, Errbit and Janky) are running behind a reverse proxy setup. NGinx performs SSL termination, then forwards the request to HAProxy, which reverse proxies based on host header and forwards the request to an internal server -- so directly at the Janky instances listening thin app, default port 5000. I hesitated to mention this, because I don't want to confuse the issue here, as we do have the proxy_set_header configurations in place. I mention this only because maybe, just maybe, there's something different going on in the headers (than all the other apps we reverse proxy that work fine). This piece of setup in nginx.conf has worked fine for years --
location / {
                #proxy to haproxy
                proxy_pass              http://localhost:80;
                proxy_set_header        Host $host;
                proxy_set_header        X-Real-IP $remote_addr;
                proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header        X-Forwarded-Proto https;

                proxy_redirect          off;
        }

I think that covers it... lots of detail here -- hopefully someone can point me to what I'm missing!

Iristyle commented

Issue re: JANKY_AUTH_TEAM_ID - #112

Iristyle commented

Ok, so last bit of diagnosis for the night.

If I take the code returned by GitHub, I'm having a problem getting a token

Note that the original request from Janky to GitHub contains

I see in the OAuth docs that state is optional, and Janky does not specify it

So if I take the code from auth/github/callback?code=THECODE, and try to Curl for the token... I don't have any luck.

Per the docs, state is optional on the initial request for a code, but mandatory on the request for the token, so I'm not sure what to do with it. (I'm just trying to verify that I can use the production machine to get the token and verify that this is an application issue)

I've tried specifying state as empty like this

curl -v -H "Accept: application/json" -H "Content-type: application/json" -X POST -d '{"client_id":"JANKY_AUTH_CLIENT_ID", "client_secret":"JANKY_AUTH_CLIENT_SECRET","code":"THECODE", "state":""}' https://github.com/login/oauth/access_token

Or simply not in the request like this

curl -v -H "Accept: application/json" -H "Content-type: application/json" -X POST -d '{"client_id":"JANKY_AUTH_CLIENT_ID", "client_secret":"JANKY_AUTH_CLIENT_SECRET","code":"THECODE" }' https://github.com/login/oauth/access_token

Neither work... in both instances I get a

{"error":"bad_verification_code"}

Per the docs

If the user accepts your request, GitHub redirects back to your site with a temporary code in a code parameter
as well as the state you provided in the previous step in a state parameter. If the states don’t match, 
the request has been created by a third party and the process should be aborted.

If I watch the traffic from my Errbit install, I see that the state parameter is filled in for the request to github.com/login/oauth. The callback to Errbit includes that state, which is then presumably used to acquire a token.

Iristyle commented

/cc @atmos not sure if this is a sinatra_auth_github issue or not?

Iristyle commented

So I've disabled the GitHub OAuth for now, to try and determine why the Janky commands that configure / talk to Jenkins aren't working.

Finally I have something useful in a log. I kind of half wondered if there might be some issue using SSL. It turns out that there is...

When Janky tries to talk to Jenkins, I get:

OpenSSL::SSL::SSLError - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed:

Relvant portion of stack

/usr/lib/ruby/1.9.1/net/http.rb:799:in `connect'
/usr/lib/ruby/1.9.1/net/http.rb:799:in `block in connect'
/usr/lib/ruby/1.9.1/timeout.rb:54:in `timeout'
/usr/lib/ruby/1.9.1/timeout.rb:99:in `timeout'
/usr/lib/ruby/1.9.1/net/http.rb:799:in `connect'
/usr/lib/ruby/1.9.1/net/http.rb:755:in `do_start'
/usr/lib/ruby/1.9.1/net/http.rb:744:in `start'
/usr/lib/ruby/1.9.1/net/http.rb:1284:in `request'
/var/lib/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/net_http_ext.rb:51:in
/var/lib/gems/1.9.1/gems/janky-0.9.13/lib/janky/builder/http.rb:21:in `run'
/var/lib/gems/1.9.1/gems/janky-0.9.13/lib/janky/builder/runner.rb:12:in `run'
/var/lib/gems/1.9.1/gems/janky-0.9.13/lib/janky/builder/client.rb:21:in `run'

This seems to be a pretty well-documented issue where OpenSSL can't find the certs on the machine (or similar)... but I'm still have some issues getting it worked out.

I was hoping to do something like this, where I just export a variable.

export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

However, this doesn't seem to work.

There are a number of other solutions involving monkey-patching, etc... For instance, there are suggestions of setting ca_file -- this would be here https://github.com/github/janky/blob/master/lib/janky/github/api.rb#L88 for the GitHub api. But this isn't really a great global solution.

        http.ca_file     = "/etc/ssl/certs/ca-certificates.crt"

I also tried this and it didn't seem to work -- but that might be because it only affects the calls to GitHub, and not the calls to Janky.

So I'm looking for the simplest, least-intrusive manner of getting this working.

If anyone has any suggestions, I'm all ears.

Iristyle commented

Also posted to SO in hopes that someone more Ruby savvy than I can figure this out!
http://stackoverflow.com/questions/13290759/ruby-opensslsslsslerror-on-ubuntu-12-04-1-ruby-1-9-3p0

rsanheim commented

Can you just disable SSL certificate verification and see if that gets you going?