Timeout Issues stemming from Puma persistent_timeout default setting
Opened this issue · 3 comments
Deleted user commented
Pre issue-raising checklist
I have already (please mark the applicable with an x):
- Confirmed this is the right place to raise the issue - only issues related to the Dockerization of the Pact Broker should be raised here. Issues related to the Pact Broker application itself should be raised in the Pact Broker project.
- Upgraded to the latest Pact Broker Docker image OR
- Checked the CHANGELOG to see if the issue I am about to raise has been fixed
- Read the Troubleshooting page
Software versions
- pact-broker gem version: eg 2.57.0
- pact-broker docker version: 2.57.0.0
- OS: N/A
- pact broker client details: pact-js v9.11.0
Expected behaviour
Running Pact verification tests should successfully publish results to the Pact Broker.
Actual behaviour
We currently run this Docker image on AWS behind an ALB, which has an idle timeout. This Docker image makes use of Puma, which has a default keep alive of 20 seconds. We have seen issues where if verification tests take longer than 20 seconds, we see errors coming from the load balancer that stem from the fact that the connection between the load balancer and the Pact Broker has been closed, leading to the ALB to return a 504 status code. This leads to errors in our CI server since the Pact binary is unable to publish verification results to the Pact Broker.
Steps to reproduce
Unfortunately the source code is for a closed-source application. Issue seems to stem from running verification tests lasting longer than 20 seconds. Should be able to reproduce with a simple delay of longer than 20 seconds.
Relevent log files
Logs from verification test run:
/usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/json-2.3.0/lib/json/common.rb:156:in `parse': 757: unexpected token at '<html>
(JSON::ParserError)
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>
'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/json-2.3.0/lib/json/common.rb:156:in `parse'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/hal/http_client.rb:70:in `body'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/provider/verification_results/publish.rb:103:in `publish_verification_results'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/provider/verification_results/publish.rb:35:in `call'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/provider/verification_results/publish.rb:21:in `call'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/provider/verification_results/publish_all.rb:22:in `block in call'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/provider/verification_results/publish_all.rb:19:in `collect'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/provider/verification_results/publish_all.rb:19:in `call'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/provider/verification_results/publish_all.rb:10:in `call'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/provider/rspec/pact_broker_formatter.rb:28:in `close'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/rspec-core-3.9.1/lib/rspec/core/reporter.rb:209:in `block in notify'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/rspec-core-3.9.1/lib/rspec/core/reporter.rb:208:in `each'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/rspec-core-3.9.1/lib/rspec/core/reporter.rb:208:in `notify'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/rspec-core-3.9.1/lib/rspec/core/reporter.rb:243:in `close'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/rspec-core-3.9.1/lib/rspec/core/reporter.rb:196:in `close_after'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/rspec-core-3.9.1/lib/rspec/core/reporter.rb:174:in `finish'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/rspec-core-3.9.1/lib/rspec/core/reporter.rb:76:in `report'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/rspec-core-3.9.1/lib/rspec/core/runner.rb:115:in `run_specs'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/rspec-core-3.9.1/lib/rspec/core/runner.rb:89:in `run'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/rspec-core-3.9.1/lib/rspec/core/runner.rb:71:in `run'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/provider/pact_spec_runner.rb:88:in `run_specs'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/provider/pact_spec_runner.rb:34:in `run'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/cli/run_pact_verification.rb:69:in `run_with_pact_uri_object'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/cli/run_pact_verification.rb:50:in `run_specs'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/cli/run_pact_verification.rb:21:in `call'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/cli/run_pact_verification.rb:13:in `call'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.30.1/lib/pact/provider_verifier/app.rb:169:in `verify_pact'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.30.1/lib/pact/provider_verifier/app.rb:43:in `block in call'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.30.1/lib/pact/provider_verifier/app.rb:42:in `collect'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.30.1/lib/pact/provider_verifier/app.rb:42:in `call'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.30.1/lib/pact/provider_verifier/app.rb:34:in `call'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.30.1/lib/pact/provider_verifier/cli/verify.rb:56:in `verify'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.30.1/lib/pact/provider_verifier/cli/custom_thor.rb:17:in `start'
from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/app/pact-provider-verifier.rb:33:in `<main>'
]
at internal/process/task_queues.js:97:5
Would it be possible to expose the relevant Puma configuration for PERSISTENT_TIMEOUT via some configuration within this Dockerfile?
bethesque commented
I haven't tested this myself, but I've made this change to the edge branch: 1e51b9a
Grab pactfoundation/pact-broker:edge and see if that fixes your issue. If it does, I'll put it on master.
Deleted user commented
@bethesque - That seems to have done the trick! Thank you so much for the quick turnaround time. It is most appreciated.
bethesque commented
This has been released in tag 2.58.0.0