Issue with Oathkeeper authenticator oauth_introspection

Question

Issue with Oathkeeper authenticator oauth_introspection

dblane-digicatapult opened this issue 2 years ago · 1 comments

dblane-digicatapult commented 2 years ago

Preflight checklist

I could not find a solution in the existing issues, docs, nor discussions.
I agree to follow this project's Code of Conduct.
I have read and am following this repository's Contribution Guidelines.
This issue affects my Ory Network project.
I have joined the Ory Community Slack.
I am signed up to the Ory Security Patch Newsletter.

Describe the bug

Oathkeeper authenticator oauth2_introspection config item introspection_url is being parsed and appears in the loaded config however requests that match are instead going to an incorrect introspection_url of http://localhost:4445/admin/oauth2/introspect.

Reproducing the bug

I am using nginx-ingress-controller and forwarding auth requests to the Ory Decision API using the global-auth-url configuration method.

In Oathkeeper I have two rules, one for authenticated paths and one for unauthenticated paths:

...
  name: oathkeeper-authenticated-paths
  namespace: ory
spec:
  match:
    url: <.*>/<alice|bob|charlie>/<dscp-api|dscp-identity-service>/<v1|v3>/<(?!swagger|api-docs).*>
    methods:
      - GET
      - POST
  authenticators:
    - handler: oauth2_introspection
      config:
        introspection_url: http://hydra-admin:4445/oauth2/introspect
        scope_strategy: wildcard
        cache:
          enabled: false
  authorizer:
    handler: allow
  mutators:
    - handler: noop
---
apiVersion: oathkeeper.ory.sh/v1alpha1
kind: Rule
metadata:
  name: oathkeeper-unauthenticated-paths
  namespace: ory
spec:
  match:
    url: <.*>/<(swagger|api-docs).*>
    methods:
      - GET
      - POST
  authenticators:
    - handler: noop
  authorizer:
    handler: allow

These both appear in the configuration oathkeeper-rules.json configmap are are successfully loaded by Oathkeeper.

I can see that this has been loaded at startup by Oathkeeper.

Curling a valid URL for an authenticated path with a valid bearer token from Hydra returns a 500 from nginx-ingress-controller. The request for authentication reaches Oathkeeper which then tries to contact the localhost introspection url mentioned above, despite this not being set in any config I have created.

curl -X 'GET'   'http://localhost:3080/alice/dscp-identity-service/v1/self'   -H 'accept: application/json' -H 'Authorization: bearer <token omitted>'
<html>
<head><title>500 Internal Server Error</title></head>
<body>
<center><h1>500 Internal Server Error</h1></center>
<hr><center>nginx</center>
</body>
</html>
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->

Additionally oathkeeper doesn't seem to output logs as JSON despite this being specified in config.

Relevant log output

Thank you for using ORY Oathkeeper v0.40.1!

Take security seriously and subscribe to the ORY Security Newsletter. Stay on top of new patches and security insights.                                                                                                

>> Subscribe now: http://eepurl.com/di390P <<
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.bearer_token.config.preserve_query = true audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.proxy.cors.exposed_headers = [Content-Type] audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.anonymous.config.subject = anonymous audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.bearer_token.config.extra_from = extra audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.prometheus.metrics_path = /metrics audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: errors.handlers.redirect.enabled = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.oauth2_client_credentials.enabled = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: mutators.hydrator.config.api.retry.give_up_after = 1s audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: mutators.hydrator.config.cache.enabled = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.anonymous.enabled = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.cookie_session.config.subject_from = subject audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.bearer_token.config.subject_from = sub audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.api.timeout.read = 5s audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.cookie_session.config.extra_from = extra audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: errors.handlers.json.config.verbose = true audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.prometheus.collapse_request_paths = true audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.prometheus.hide_request_paths = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.api.cors.allowed_headers = [Authorization Content-Type] audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.cookie_session.config.preserve_query = true audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.proxy.cors.allow_credentials = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.oauth2_client_credentials.config.cache.max_tokens = 1000 audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.api.cors.allow_credentials = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.bearer_token.enabled = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.prometheus.host =  audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.jwt.config.scope_strategy = none audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: errors.fallback = [json] audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.proxy.timeout.idle = 120s audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authorizers.remote.config.retry.max_delay = 100ms audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.api.cors.debug = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: log.format = json audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.cookie_session.enabled = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.api.timeout.write = 120s audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: errors.handlers.json.enabled = true audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: mutators.hydrator.config.api.retry.max_delay = 100ms audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.proxy.timeout.write = 120s audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authorizers.allow.enabled = true audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.proxy.port = 4455 audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: mutators.header.enabled = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: mutators.id_token.config.ttl = 15m audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: access_rules.matching_strategy = regexp audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: errors.handlers.redirect.config.return_to_query_param =  audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: mutators.id_token.enabled = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.oauth2_introspection.config.retry.give_up_after = 1s audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.proxy.cors.enabled = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.proxy.cors.allowed_methods = [GET POST PUT PATCH DELETE] audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: errors.handlers.redirect.config.code = 302 audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.oauth2_introspection.config.retry.max_delay = 100ms audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: access_rules.repositories = [file:///etc/rules/access-rules.json] audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.oauth2_client_credentials.config.retry.give_up_after = 1s audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.api.cors.enabled = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.oauth2_introspection.enabled = true audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.jwt.config.jwks_ttl = 30s audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.proxy.cors.allowed_headers = [Authorization Content-Type] audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authorizers.remote.enabled = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: mutators.hydrator.enabled = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.proxy.timeout.read = 5s audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: mutators.cookie.enabled = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.unauthorized.enabled = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: errors.handlers.www_authenticate.config.realm = Please authenticate. audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.api.cors.allowed_methods = [GET POST PUT PATCH DELETE] audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authorizers.remote_json.config.forward_response_headers_to_upstream = [] audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authorizers.remote_json.config.retry.max_delay = 100ms audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.noop.enabled = true audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authorizers.remote.config.retry.give_up_after = 1s audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.oauth2_introspection.config.introspection_url = http://hydra-admin:4445/oauth2/introspect audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: mutators.hydrator.config.cache.ttl = 1m audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.jwt.enabled = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: tracing.providers.jaeger.sampling.trace_id_ratio = 1 audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.jwt.config.jwks_max_wait = 1s audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.api.host =  audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.oauth2_introspection.config.pre_authorization.enabled = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.oauth2_client_credentials.config.retry.max_delay = 100ms audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.proxy.cors.debug = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.proxy.host =  audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authorizers.remote.config.forward_response_headers_to_upstream = [] audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.oauth2_client_credentials.config.cache.enabled = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authorizers.keto_engine_acp_ory.enabled = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.api.cors.max_age = 0 audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.prometheus.metric_name_prefix = ory_oathkeeper_ audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.proxy.cors.allowed_origins = [*] audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.oauth2_introspection.config.scope_strategy = wildcard audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.api.cors.exposed_headers = [Content-Type] audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authorizers.deny.enabled = true audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: log.level = debug audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.api.timeout.idle = 120s audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: errors.handlers.www_authenticate.enabled = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.proxy.cors.max_age = 0 audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.api.cors.allowed_origins = [*] audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.api.port = 4456 audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: serve.prometheus.port = 9000 audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.oauth2_introspection.config.cache.max_cost = 1000 audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: mutators.noop.enabled = true audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authenticators.oauth2_introspection.config.cache.enabled = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authorizers.remote_json.config.retry.give_up_after = 1s audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Loaded config: authorizers.remote_json.enabled = false audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=No tracer configured - skipping tracing setup audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=No tracer configured - skipping tracing setup audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Software quality assurance features are enabled. Learn more at: https://www.ory.sh/docs/ecosystem/sqa audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Listening on http://:9000 audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=TLS has not been configured for proxy, skipping audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Listening on http://:4455 audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=TLS has not been configured for api, skipping audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:25Z level=info msg=Listening on http://:4456 audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:50Z level=info msg=started handling request http_request=map[headers:map[accept:application/json authorization:Value is sensitive and has been redacted. To see the value set config key "log.leak_sensitive_values = true" or environment variable "LOG_LEAK_SENSITIVE_VALUES=true". connection:close user-agent:Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) x-auth-request-redirect:/alice/dscp-identity-service/v1/self x-forwarded-for:172.18.0.1 x-original-method:GET x-original-url:http://localhost:3080/alice/dscp-identity-service/v1/self x-real-ip:172.18.0.1 x-request-id:43b5abb40839248ad26bc40a959e4ad7 x-sent-from:nginx-ingress-controller] host:oathkeeper-api.ory.svc.cluster.local method:GET path:/decisions/alice/dscp-identity-service/v1/self query:<nil> remote:172.18.0.2:47328 scheme:http]
time=2023-03-14T09:23:50Z level=warning msg=The authentication handler encountered an error audience=application authentication_handler=oauth2_introspection error=map[message:Post "http://localhost:4445/admin/oauth2/introspect": dial tcp [::1]:4445: connect: connection refused] granted=false http_host=oathkeeper-api.ory.svc.cluster.local http_method=GET http_url=http://oathkeeper-api.ory.svc.cluster.local/alice/dscp-identity-service/v1/self http_user_agent=Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) reason_id=authentication_handler_error rule_id=oathkeeper-authenticated-paths.ory service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:50Z level=info msg=Access request denied audience=application error=map[message:Post "http://localhost:4445/admin/oauth2/introspect": dial tcp [::1]:4445: connect: connection refused] granted=false http_host=oathkeeper-api.ory.svc.cluster.local http_method=GET http_url=http://oathkeeper-api.ory.svc.cluster.local/alice/dscp-identity-service/v1/self http_user_agent=Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:50Z level=error msg=An error occurred while handling a request audience=application error=map[message:Post "http://localhost:4445/admin/oauth2/introspect": dial tcp [::1]:4445: connect: connection refused] http_request=map[headers:map[accept:application/json authorization:Value is sensitive and has been redacted. To see the value set config key "log.leak_sensitive_values = true" or environment variable "LOG_LEAK_SENSITIVE_VALUES=true". connection:close user-agent:Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) x-auth-request-redirect:/alice/dscp-identity-service/v1/self x-forwarded-for:172.18.0.1 x-original-method:GET x-original-url:http://localhost:3080/alice/dscp-identity-service/v1/self x-real-ip:172.18.0.1 x-request-id:43b5abb40839248ad26bc40a959e4ad7 x-sent-from:nginx-ingress-controller] host:oathkeeper-api.ory.svc.cluster.local method:GET path:/alice/dscp-identity-service/v1/self query:<nil> remote:172.18.0.2:47328 scheme:http] http_response=map[status_code:500] service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-03-14T09:23:50Z level=info msg=completed handling request http_request=map[headers:map[accept:application/json authorization:Value is sensitive and has been redacted. To see the value set config key "log.leak_sensitive_values = true" or environment variable "LOG_LEAK_SENSITIVE_VALUES=true". connection:close user-agent:Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) x-auth-request-redirect:/alice/dscp-identity-service/v1/self x-forwarded-for:172.18.0.1 x-original-method:GET x-original-url:http://localhost:3080/alice/dscp-identity-service/v1/self x-real-ip:172.18.0.1 x-request-id:43b5abb40839248ad26bc40a959e4ad7 x-sent-from:nginx-ingress-controller] host:oathkeeper-api.ory.svc.cluster.local method:GET path:/alice/dscp-identity-service/v1/self query:<nil> remote:172.18.0.2:47328 scheme:http] http_response=map[headers:map[content-type:application/json] size:218 status:500 text_status:Internal Server Error took:31.426292ms]

Relevant configuration

log:
  level: debug
  format: json
serve:
  api:
    port: 4456
  proxy:
    port: 4455
errors:
  fallback:
    - json
  handlers:
    json:
      enabled: true
      config:
        verbose: true
mutators:
  noop:
    enabled: true
authorizers:
  allow:
    enabled: true
  deny:
    enabled: true
authenticators:
  oauth2_introspection:
    enabled: true
    config:
      introspection_url: http://hydra-admin:4445/oauth2/introspect
      scope_strategy: wildcard
  noop:
    enabled: true

Version

v0.40.1

On which operating system are you observing this issue?

Linux

In which environment are you deploying?

Kubernetes with Helm

Additional Context

No response

Answer 1 · 2023-03-21T15:10:19.000Z

The issue is resolved (kind of).

in Hydra the self.admin url was not set so it was causing a redirect to localhost.

It would have been much easier to debug this if Oathkeeper showed us in the logs that it was following a redirect, which it did not despite logging being set to debug level.