Can't get authorization

We're getting lost here, and I'm not sure where we are going wrong. Ideas please?

>>> url = 'https://www.example.org'
>>> my_headers = {'Content-Type': 'application/json'}
>>> 
>>> requests.get(url, auth=HTTPKerberosAuth(mutual_authentication=OPTIONAL,service="HTTP"), headers=my_headers, verify=False)

/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:730: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html (This warning will only appear once by default.)
  InsecureRequestWarning)
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 60, in get
    return request('get', url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 49, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 457, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 575, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/hooks.py", line 41, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests_kerberos/kerberos_.py", line 252, in handle_response
    _r = self.handle_401(response, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests_kerberos/kerberos_.py", line 164, in handle_401
    _r.raise_for_status()
  File "/usr/local/lib/python2.7/site-packages/requests/models.py", line 825, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Authorization Required

We are using py-kerberos-1.1.1_5, requests-kerberos-0.6.1_1, and requests-2.4.3

The code you have is different to the code we released in v0.6.1. It seems you have a downstream patch on top of your code that added a raise_for_status call. Where did you get requests-kerberos from?

We are using FreeBSD and I created this port https://www.freshports.org/security/py-requests-kerberos/ based on https://github.com/requests/requests-kerberos

I don't know how familiar you are with FreeBSD. A port is [more or less] just a package.

To explicitly answer your question, it's from https://github.com/requests/requests-kerberos version 0.6.1

Does that help?

I'm reasonably familiar with the concept of a port. =)

What you'll need to do is track down where that patch was added and why. As you can see in our tree, we do not have a raise_for_status line, either on the line you have it or in the function you have it. To work out what's going on you really need to understand where that line came from and why it was added.

Ahh, I know now. It came from here: https://github.com/requests/requests-kerberos/pull/47/files

Aha, ok then. That debug log contains the important facts then: you're failing to authenticate. I recommend turning on debug logging to see what it's doing.

OK, I think this has led us back to an error I saw earlier:

'Server (krbtgt/www.example.org@auth.example.org) unknown'

I think it's that krbtgt which is wrong for our setup. It should be HTTP for what we're doing and we don't know how to change that.

Do you know where that log is coming from?

This is from yesterday, and my test code has changed since then:

015-03-31 11:08:00,166 kerberos_.py - ERROR - generate_request_header(): authGSSClientStep() failed:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/requests_kerberos/kerberos_.py", line 113, in generate_request_header
    _negotiate_value(response))
GSSError: ((' Miscellaneous failure (see text)', 851968), ('Server (krbtgt/www.example.org@auth.example.org) unknown', -1765328377))

Interesting. I don't actually know how to change that either: kerberos is not an area I know that much about, I'm afraid!

Ugh. You and me both.

I thank you for your help though. It has helped my focus on the issue. Perhaps my next port of call should be: https://www.calendarserver.org/browser/PyKerberos/trunk

No problem, happy to help. Let's keep this open for now, in case it turns out to be a problem on our end.

Agreed. Thanks again.

See https://www.calendarserver.org/ticket/905

FYI, I now believe krbtgt is correct. After kinit, that's the type of ticket you have. I've been reading https://www.freebsd.org/doc/handbook/kerberos5.html and following their examples.

Perhaps we are doing this entirely the wrong way. I conclude that based on all attempts give 'Server (krbtgt/poke.int.example.org@INT.EXAMPLE.ORG) unknown'. I have no idea if we're missing some setup, configuration, etc. Ideas please?

FYI, if I put https://poke.int.example.org/path/to/file into my browser, I get the expected file.

import os
import requests
import logging
import logging.handlers
from subprocess import Popen, PIPE
from requests_kerberos import HTTPKerberosAuth, OPTIONAL

def init_logging():
    formatter = logging.Formatter("%(asctime)s %(filename)s - %(levelname)s - %(message)s")
    fh = logging.handlers.WatchedFileHandler(os.path.join("hash_poker.log"))
    fh.setFormatter(formatter)
    logger = logging.getLogger('requests_kerberos')
    logger.addHandler(fh)
    ch = logging.StreamHandler()
    ch.setFormatter(formatter)
    logger.addHandler(ch)
    logger.setLevel(logging.INFO)

init_logging()

r = requests.get("https://poke.int.example.org/path/to/file", auth=HTTPKerberosAuth(), verify=False)

/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:730: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html (This warning will only appear once by default.)
  InsecureRequestWarning)
2015-04-02 11:17:58,677 kerberos_.py - ERROR - generate_request_header(): authGSSClientStep() failed:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/requests_kerberos/kerberos_.py", line 113, in generate_request_header
    _negotiate_value(response))
GSSError: ((' Miscellaneous failure (see text)', 851968), ('Server (krbtgt/poke.int.example.org@INT.EXAMPLE.ORG) unknown', -1765328377))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 60, in get
    return request('get', url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 49, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 457, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 575, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/hooks.py", line 41, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests_kerberos/kerberos_.py", line 252, in handle_response
    _r = self.handle_401(response, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests_kerberos/kerberos_.py", line 164, in handle_401
    _r.raise_for_status()
  File "/usr/local/lib/python2.7/site-packages/requests/models.py", line 825, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Authorization Required

We have success.

import kerberos
import requests
from requests_kerberos import HTTPKerberosAuth, OPTIONAL

__, krb_context = kerberos.authGSSClientInit("HTTP/poke.int.example.org@INT.EXAMPLE.ORG")
kerberos.authGSSClientStep(krb_context, "")

negotiate_details = kerberos.authGSSClientResponse(krb_context)

headers = {"Authorization": "Negotiate " + negotiate_details}

url = "https://poke.int.example.org/path/to/file"
r = requests.get(url, headers=headers, verify=False)
/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:730: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html (This warning will only appear once by default.)
  InsecureRequestWarning)
>>> r.status_code
200

This seems quite... interesting. We mostly do exactly that with

requests-kerberos/requests_kerberos/kerberos_.py

Line 100 in b3a19a0

result, self.context[host] = kerberos.authGSSClientInit(

. Can you add some debugging/print statements to that section of the plugin to determine what's causing the problem? What are we doing that is different from what you're doing?

Ian: We are using a patched version of py-kerberos which uses Heimdal Kerberos supplied in FreeBSD base. Might that explain it? ref https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=199123

I can do some of this testing tomorrow it it will help. Glad to do that.

I'm just curious what's so different between what you did by hand and what we're doing in the plugin. That's all. But perhaps the patched version is the problem. shrug

I will see what I can come up with tomorrow morning.

Ian: can you guide me a bit with the auth process for requests_kerberos?

Is this what you are doing?

r = requests.get(url, auth=HTTPKerberosAuth("HTTP/poke.INTexample.org@INT.EXAMPLE.ORG), verify=False)

So I'm a bit confused. Looking at this it looks like we would construct something like:

HTTP@poke.int.example.org

Assuming we received a response from poke.int.example.org. That said, if you're authenticating against int.example.org (or requesting something on that domain), it looks like you would use

r = requests.get(url, auth=HTTPKerberosAuth(service='HTTP/poke.int.example.org'), verify=False)

The last time I used this was a long time ago for a one-off script that I no longer have access to. I don't remember running into this though. =/

That's what was blocking me before I tried kerberos.authGSSClientInit.

This is what we get:

[dvl@dev ~]$ python
Python 2.7.8 (default, Sep 25 2014, 19:00:23) 
[GCC 4.2.1 20070831 patched [FreeBSD]] on freebsd9
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> import requests
>>> from requests_kerberos import HTTPKerberosAuth, OPTIONAL
>>> url = 'https://poke.int.example.org/v0/bulk/sha256/76e2c4f17a3c2a94848b4491b86cf64611b4f4c9c5fb540ebcd57d0c96622d9e'
>>> r = requests.get(url, auth=HTTPKerberosAuth(service='HTTP/poke.int.example.org'), verify=False)
/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:730: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html (This warning will only appear once by default.)
  InsecureRequestWarning)
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 60, in get
    return request('get', url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 49, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 457, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 575, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/hooks.py", line 41, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests_kerberos/kerberos_.py", line 252, in handle_response
    _r = self.handle_401(response, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests_kerberos/kerberos_.py", line 164, in handle_401
    _r.raise_for_status()
  File "/usr/local/lib/python2.7/site-packages/requests/models.py", line 825, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Authorization Required
>>>

Yes, this method, pasted a few days ago, succeeds:

__, krb_context = kerberos.authGSSClientInit("HTTP/poke.int.example.org@INT.EXAMPLE.ORG")
kerberos.authGSSClientStep(krb_context, "")

It's as if the first method doesn't know about int.example.org...

There's a lot going on here. Sorry I didn't notice it for all these months.

There's a few things to keep in mind which make working with HTTP & Kerberos confusing.

First, I don't believe there is an actual standard API for working with kerberos. They're all implementation specific. MIT & Heimdal are similar, but not identical. which is why people tend to write against GSSAPI instead of directly against kerberos.

GSSAPI is a generic API that works with pluggable authentication mechanisms. One of the (and possibly the most common) mechanisms is Kerberos. This is why most of the methods we use in the (confusingly named) kerberos library include gssapi in their name.

One problem with using GSSAPI as an abstraction over Kerberos is that while Kerberos has a concept of a REALM (the @INT.EXAMPLE.ORG in your code), GSSAPI does not. That's why you don't see realm names in anything in this library. There's no way to infer it -- your kerberos libraries (MIT/Heimdal) will assume the service is in the same realm as your tgt (unless the client library is statically confirmed with a realm mapping for the hostname in question), and send the request to your realms kdc for (by convention) service/hostname@YOURREALM. The kdc will then check to see if the service is present in its database, and if it is, respond with a service ticket, and if not, it may issue a referral to another realm (if cross-realm trust has been established), or fail.

Another thing that causes confusion is the similarity between a gss name and a kerberos principal. A gss nt hostbased service takes the form of service@hostname, in this case, the service is HTTP (and no, I can't explain why I made the service configurable in this library. It's misleading and can cause confusion, as it did here). So the gss nt hostbased service name you want is going to be HTTP@poke.int.example.org, which is quite similar to the principal HTTP/poke.int.example.org@INT.EXAMPLE.ORG.

If you were using MIT kerberos, I"d suggest you try running the code with the environment variable KRB5_TRACE set to /dev/stderr -- but I don't know how to get similar debugging output from heimdal.

If the library can't come up with the correct service principle, the only way to force it (as of v0.11.0) without having to call the underlying kerberos libraries itself is:

given that you want the service principle to be HTTP/poke.int.example.org@INT.EXAMPLE.ORG

auth=HTTPKerberosAuth(service='HTTP/poke.int.example.org',hostname_override='INT.EXAMPLE.ORG')
requests.get(url, auth=auth)

This is kind of a hack. Internally the library just combines service and hostname_override with a @.
It would be nice if one could specify the full service principle explicitly. Putting it in the principle parameter doesn't work.

Closing due to the age of the ticket, by all indications the issue is with trying to find the target host using the SPN required. There are many good suggestions in this thread already to try and help figure out what it is.