GCP Secret Manager empty reply/broken pipe on Ubuntu 22 VM in PHP
BrianPez opened this issue · 1 comments
(https://stackoverflow.com/q/77787335)
I have an Ubuntu 22 VM running in GCP. I have about 850 legacy IIoT devices talking to the server via Apache to a php script. The keep alive is set to 10 1/2 minutes. The IIoT devices are talking at least every ten minutes. At the start of the php script, a call is made to GCP's secret manager (via Google's API). Every 10 minutes, (basically) every request to secret manager fails. The exception shows one of two failure messages.
cURL error 52: Empty reply from server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for XXX
cURL error 55: Send failure: Broken pipe (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for XXX
Things I have tried: I changed the keep-alive to 10 hours since the timing between the two was suspicious. It had no effect.
I read about an Ubuntu 22 issue with openssl 3.0.X, but I was under the impression that was a server side (GCP) error. I tried different libcurl versions (7.81.0-1ubuntu1.7, 7.81.0-1ubuntu1.14, 7.81.0-1ubuntu1.15, 8.5.0) with no effect.
I split the devices between two VMs (400 on one and 450 on the other). And the problem went away. Checking my GCP quota limits and I am not at even 1% usage. I do see higher request counts when the errors are occurring, but no where close to the 90k per minute GCP limit (https://cloud.google.com/secret-manager/quotas#request-rate-quotas). Picking a random 30 minute window:
12:00 782
12:01 1623
12:02 762
12:03 148
12:04 140
12:05 150
12:06 144
12:07 161
12:08 161
12:09 603
12:10 748
12:11 1648
12:12 762
12:13 216
12:14 133
12:15 107
12:16 112
12:17 89
12:18 71
12:19 624
12:20 784
12:21 1628
12:22 720
12:23 160
12:24 103
12:25 125
12:26 120
12:27 105
12:28 61
12:29 536
Here are the requests for this morning on the 450 server. There are higher request and there are no failures:
08:00 954
08:01 52
08:02 63
08:03 47
08:04 57
08:05 35
08:06 61
08:07 77
08:08 53
08:09 326
08:10 963
08:11 71
08:12 92
08:13 81
08:14 50
08:15 65
08:16 96
08:17 116
08:18 132
08:19 2242
08:20 2089
08:21 331
08:22 91
cURL error 52: Empty reply from server
The cURL error 52: Empty reply from server means that the server did not send a response back to the client. This could be due to a network issue or a problem with the server
cURL error 55: Send failure: Broken pipe
The "curl send error broken pipe" error message indicates that there was a problem sending data over a network connection.
Both of these errors are server-side or network related, and not related to the PHP client library.