Rack::Protection::IPSpoofing error while sending a request to pact broker
damlagul opened this issue · 14 comments
We are using pact broker with python and deployed it to Azure Web App using the docker image: pactfoundation/pact-broker:2.102.1.0
When we want to access the endpoint of this app through our locals, we are able to get 200 - however, when we tried this on Azure DevOps, we are getting 403 Forbidden errors in the pipelines. After investigating the issue using the app logs (which you can find the logs below), we saw that Rack blocks the IP address as it gets the requests as an IPSpoofing thread.
We tried to whitelist the IP address of the agent and also used PACT_BROKER_WEBHOOK_HOST_WHITELIST in the environment variables to allow all IPs by setting its value to [/.*/] but again we are getting the same response from the logs.
Do you have any suggestions to resolve this issue?
2022-08-31T11:30:57.774778105Z 2022-08-31 11:30:57.773431 I [6:puma srv tp 001] PactBroker::App -- Configuring basic auth
2022-08-31T11:30:57.774954505Z 2022-08-31 11:30:57.773526 I [6:puma srv tp 001] PactBroker::App -- Public read access is enabled
2022-08-31T11:30:57.777436712Z 2022-08-31 11:30:57.777008 I [6:puma srv tp 001] PactBroker::App -- Mounting HAL browser
2022-08-31T11:30:57.777836613Z 2022-08-31 11:30:57.777062 I [6:puma srv tp 001] PactBroker::App -- Mounting UI
2022-08-31T11:30:58.079278059Z 2022-08-31 11:30:58.078723 I [6:puma srv tp 001] PactBroker::App -- Mounting PactBroker::API
2022-08-31T11:35:27.870304912Z W, [2022-08-31T11:35:27.870043 #6] WARN -- : attack prevented by Rack::Protection::IPSpoofing
Looking at the source code, I would say that your X-Forwarded-For headers are misconfigured.
https://github.com/sinatra/sinatra/blob/master/rack-protection/lib/rack/protection/ip_spoofing.rb
Headers
Accept: */*
Host: <app_endpoint>
Max-Forwards: 10
User-Agent: curl/7.68.0
X-ARR-LOG-ID: xxx
CLIENT-IP: <IPV6:PORT>
DISGUISED-HOST: <app_endpoint>
X-SITE-DEPLOYMENT-ID: xxx
WAS-DEFAULT-HOSTNAME: <app_endpoint>
X-Forwarded-Proto: https
X-AppService-Proto: https
X-ARR-SSL: xxxx
X-Forwarded-TlsVersion: 1.2
X-Forwarded-For: <IPV6:PORT>
X-Original-URL: /
X-WAWS-Unencoded-URL: /
X-forwarded-For URL has the same value as for the CLIENT-IP. In Azure. when we disabled Microsoft.Web endpoint in the subnet config, it returns an ipv4 and we are able to get 200. However, we need to have this endpoint and this will generate an ipv6. I couldn't figure out why it accepts the ipv4 and not the ipv6 for Rack.
You could try enabling http debug logging to see if it gives you more insight as to the values rack is seeing. https://docs.pact.io/pact_broker/configuration/settings#http_debug_logging_enabled
Any progress on this? I'm sure other people will have the same issue, so if you have found a solution, please do share it.
Unfortunately, no. Using the HTTP debug logging, I found out what is being sent to Rack:
When Microsoft.Web is:
enabled:
HTTP_X_FORWARDED_FOR= IPV4
HTTP_CLIENT_IP= [IPV6]:PORT
disabled:
HTTP_X_FORWARDED_FOR=[IPV4]:PORT
HTTP_CLIENT_IP= [IPV4]:PORT
When it is disabled, HTTP_X_FORWARDED_FOR and HTTP_CLIENT_IP have the same values; therefore I am getting 200 from the app.
Is there a way to disable this functionality -that does the check for the IPSpoofing thread- on the pact broker's side? There is an attribute in the source code: use_rack_protection But I am not sure how to set it to False to avoid from this check.
Otherwise, the only solution we found is to create a new agent and disable Microsoft.Web endpoint for its subnet so that it can use the public IP address while sending requests to the app.
You can turn off rack protection by setting the environment variable PACT_BROKER_USE_RACK_PROTECTION=false. It's not documented because I don't want people to turn it off generally! You'll need to rely on something else to protect the app in that case.
I think it might be worth raising with the rack protection gem.
I agree and I guess the best option here is to change the way Rack library uses to check the IPSpoofing thread. I created an issue on their side.
@bethesque as mentioned here: sinatra/sinatra#1812 (comment), can pact broker make the list of protections configurable? Or possibly consider adding ip spoofing to the exclusions?
We can make the exclusions configurable. I'm actually not sure why the ones that are currently excluded are excluded. It's been too long since I set it up.
Hi @bethesque is there any update for this new configuration?
I've just put out 2.103.0.0.
If you set PACT_BROKER_RACK_PROTECTION_EXCEPT="ip_spoofing" it should do the trick.
@bethesque after setting this variable, we are getting the error below:
2022-09-30T13:59:18.449433861Z 2022-09-30 13:59:18 +0000 Rack app ("GET /" - (127.0.0.1:14000)): #<RuntimeError: you need to set up a session middleware *before* Rack::Protection::SessionHijacking>2022-09-30T13:59:18.443740703Z 2022-09-30 13:59:18 +0000 Rack app ("GET /" - (127.0.0.1:13999)): #<RuntimeError: you need to set up a session middleware *before*>
The default value of the except option is [:path_traversal, :remote_token, :session_hijacking, :http_origin]. You'll need to include all of those, as you're overwriting it.
PACT_BROKER_RACK_PROTECTION_EXCEPT="ip_spoofing path_traversal remote_token session_hijacking http_origin"