X-Forwarded headers missing from oauth2-client-credentials authenticator request on v.0.40.3, breaking hydra TLS termination

[mathportillo]

Preflight checklist

• I could not find a solution in the existing issues, docs, nor discussions.
• I agree to follow this project's Code of Conduct.
• I have read and am following this repository's Contribution Guidelines.
• This issue affects my Ory Network project.
• I have joined the Ory Community Slack.
• I am signed up to the Ory Security Patch Newsletter.

Describe the bug

I was using oathkeeper on docker on version oryd/oathkeeper:v0.40.2.
Also using oryd/hydra:v1.11.10. Hydra was setup to tls allow termination and working just fine.

After updating oathkeeper to image oryd/oathkeeper:v0.40.2, hydra started denying requests on /oauth2/token. The reason was that oathkeeper was not sending x-forwarded-proto and x-forwarded-for headers.

The only thing changed was oathkeeper version from v0.40.2 to v0.40.3. Nothing else changed, no new rules, no new config, no new version for hydra.

I rolled back to v0.40.2 and everything works fine again.

Reproducing the bug

1. Set up oathkeeper using docker v0.40.3 to use authenticator oauth2_client_credentials as shown in relevant configuration
2. Set up access rule that using client_credentials
3. Set up hydra normally with some client registered
4. Make request to oathkeeper path that matches access rule, using the right auth headers

Relevant log output

// HYDRA receiving from oathkeeper v0.40.3

time=2023-05-04T12:34:30Z level=error msg=An error occurred while handling a request audience=application error=map[message:can not serve request over insecure http] http_request=map[headers:map[accept-encoding:gzip authorization:Value is sensitive and has been redacted. To see the value set config key "log.leak_sensitive_values = true" or environment variable "LOG_LEAK_SENSITIVE_VALUES=true". content-length:29 content-type:application/x-www-form-urlencoded user-agent:Go-http-client/2.0 via:1.1 google x-cloud-trace-context:3d22f5f77ba3c63ce32872ce56b680be/12868810477363431831] host:[REDACTED] method:POST path:/oauth2/token query:<nil> remote:10.4.3.142:55716 scheme:http] http_response=map[status_code:502] service_name=Ory Hydra service_version=v1.11.10"


// HYDRA receiving from oathkeeper v0.40.2

time=2023-05-04T14:17:11Z level=info msg=completed handling request http_request=map[headers:map[accept-encoding:gzip authorization:Value is sensitive and has been redacted. To see the value set config key "log.leak_sensitive_values = true" or environment variable "LOG_LEAK_SENSITIVE_VALUES=true". content-length:29 content-type:application/x-www-form-urlencoded user-agent:Go-http-client/2.0 via:1.1 google x-cloud-trace-context:10f9013ceb74747e4a60db950ca2a678/17866090265765011028 x-forwarded-for:[REDACTED], [REDACTED], [REDACTED] x-forwarded-proto:https] host:[REDACTED] method:POST path:/oauth2/token query:<nil> remote:10.4.3.146:34584 scheme:http] http_response=map[headers:map[cache-control:no-store content-type:application/json;charset=UTF-8 pragma:no-cache] size:157 status:200 text_status:OK took:341.772251ms]

Relevant configuration

authenticators:
  oauth2_client_credentials:
    enabled: true
    config:
      token_url: ${HYDRA_URL}/oauth2/token
      retry:
        give_up_after: '60s'
        max_delay: '5s'

Version

v0.40.3

On which operating system are you observing this issue?

Linux

In which environment are you deploying?

Kubernetes

Additional Context

No response