TLS / SSL issues when connecting to AU0 region

Question

TLS / SSL issues when connecting to AU0 region

ceguimaraes opened this issue a year ago · 1 comments

Hi guys,

 I am having issues connecting to the AU0 region. We are shifting from US1 to AU0. US1 works fine, but when we change it to AU0 realm we get the the following error:

AppVersion: 0.79.0
Deploying using helm chart version 0.79.0
K8S 1.24

go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).onTemporaryFailure
go.opentelemetry.io/collector/exporter@v0.79.0/exporterhelper/queued_retry.go:165
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
go.opentelemetry.io/collector/exporter@v0.79.0/exporterhelper/queued_retry.go:407
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
go.opentelemetry.io/collector/exporter@v0.79.0/exporterhelper/metrics.go:125
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
go.opentelemetry.io/collector/exporter@v0.79.0/exporterhelper/queued_retry.go:195
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
go.opentelemetry.io/collector/exporter@v0.79.0/exporterhelper/internal/bounded_memory_queue.go:47
2023-06-23T05:59:44.737Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post "https://ingest.au0.signalfx.com/v2/datapoint\": read tcp 244.17.10.232:51536->3.104.137.75:443: read: connection reset by peer", "interval": "5.378545282s"}
2023-06-23T05:59:46.055Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "signalfx", "error": "Post "https://ingest.au0.signalfx.com/v2/event\": read tcp 244.17.10.232:37026->54.206.105.21:443: read: connection reset by peer", "interval": "35.4606517s"}
2023-06-23T05:59:54.812Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post "https://ingest.au0.signalfx.com/v2/datapoint\": read tcp 244.17.10.232:51922->13.210.200.67:443: read: connection reset by peer", "interval": "16.541523521s"}
2023-06-23T05:59:58.794Z error exporterhelper/queued_retry.go:165 Exporting failed. No more retries left. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "max elapsed time expired Post "https://ingest.au0.signalfx.com/v2/datapoint\": read tcp -------------------->13.210.200.67:443: read: connection reset by peer", "dropped_items": 28}

More info:

US1
$ curl -v -L https://ingest.us1.signalfx.com

Rebuilt URL to: https://ingest.us1.signalfx.com/
Trying 35.155.106.192...
TCP_NODELAY set
Connected to ingest.us1.signalfx.com (35.155.106.192) port 443 (#0)
ALPN, offering h2
ALPN, offering http/1.1
successfully set certificate verify locations:
CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
TLSv1.3 (OUT), TLS handshake, Client hello (1):
TLSv1.3 (IN), TLS handshake, Server hello (2):
TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
TLSv1.3 (IN), TLS handshake, Unknown (8):
TLSv1.3 (IN), TLS handshake, Certificate (11):
TLSv1.3 (IN), TLS handshake, CERT verify (15):
TLSv1.3 (IN), TLS handshake, Finished (20):
TLSv1.3 (OUT), TLS change cipher, Client hello (1):
TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
TLSv1.3 (OUT), TLS handshake, Finished (20):
SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
ALPN, server accepted to use h2
Server certificate:
subject: CN=*.us1.signalfx.com
start date: Aug 30 18:02:44 2022 GMT
expire date: Oct 1 18:02:44 2023 GMT
subjectAltName: host "ingest.us1.signalfx.com" matched cert's "*.us1.signalfx.com"
issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=http://certs.godaddy.com/repository/; CN=Go Daddy Secure Certificate Authority - G2
SSL certificate verify ok.
Using HTTP2, server supports multi-use
Connection state changed (HTTP/2 confirmed)
Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
TLSv1.3 (OUT), TLS Unknown, Unknown (23):
TLSv1.3 (OUT), TLS Unknown, Unknown (23):
TLSv1.3 (OUT), TLS Unknown, Unknown (23):
Using Stream ID: 1 (easy handle 0x55cf0c79ed80)
TLSv1.3 (OUT), TLS Unknown, Unknown (23):

GET / HTTP/2
Host: ingest.us1.signalfx.com
User-Agent: curl/7.58.0
Accept: /

TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
TLSv1.3 (IN), TLS Unknown, Unknown (23):
Connection state changed (MAX_CONCURRENT_STREAMS updated)!
TLSv1.3 (OUT), TLS Unknown, Unknown (23):
TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/2 404
< date: Fri, 23 Jun 2023 06:11:21 GMT
< server: istio-envoy
<
Connection #0 to host ingest.us1.signalfx.com left intact

AU0
$ curl -v -L https://ingest.au0.signalfx.com

Rebuilt URL to: https://ingest.au0.signalfx.com/
Trying 13.210.200.67...
TCP_NODELAY set
Connected to ingest.au0.signalfx.com (13.210.200.67) port 443 (#0)
ALPN, offering h2
ALPN, offering http/1.1
successfully set certificate verify locations:
CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
TLSv1.3 (OUT), TLS handshake, Client hello (1):
OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to ingest.au0.signalfx.com:443
stopped the pause stream!
Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to ingest.au0.signalfx.com:443

We know we are reaching AU0 server because when we set a wrong token we can see a different error:
net/http: invalid header field value for "X-Sf-Token"

As far as I know, AU realm is quite new. I think there are certificates missing in the image, but I am not an expert.

Thank you.

Answer 1 · 2023-06-29T01:22:58.000Z

Please open a support case.