dragonflyoss/Dragonfly2

dragonfly manager config tls error: 500 Internal Server Error

Closed this issue · 8 comments

mansoncui commented

Bug report:

helm install dragonfly

Error Info :
2024-04-01T09:15:03Z [INFO] [/pkg/p2p/preheat/job.go:93]: Preheating image 'dragonfly/bitnami-shell:11-debian-11-r72@sha256:5098aa7e6360c5ee5e38152e49e03faaee8a57d5a8dcd94ea00c5490dbe87858' to the target preheat provider: dragonfly Dragonfly:http://10.215.1.168:8080
2024-04-01T09:15:03Z [INFO] [/pkg/p2p/preheat/job.go:120]: Get preheat provider driver: dragonfly
2024-04-01T09:15:03Z [INFO] [/pkg/p2p/preheat/job.go:134]: Check health of preheat provider instance: Healthy
2024-04-01T09:15:03Z [ERROR] [/pkg/p2p/preheat/job.go:75]: POST 'http://10.215.1.168:8080/preheats' error: 500 Internal Server Error {"message":"Get "https://harbor-xx.xx.xxx.xx/v2/\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}

Tls has been enabled in the manager service ，The configuration is as follows：
manager:
replicas: 1
metrics:
enable: true
config:
verbose: true
pprofPort: 18066
job:
syncPeers:
timeout: 10m
preheat:
registryTimeout: 1m
tls:
# caCert is the CA certificate for preheat tls handshake, it can be path or PEM format string.
caCert: '-----BEGIN CERTIFICATE-----
xxxxxxxx
-----END CERTIFICATE-----'

It is possible to request harbor 443 through local curl, as shown below:
dragonfly

Environment:

  • Dragonfly version: 2.1.30
  • OS: ubuntu 18.04
  • Kernel : 5.4.193-0504193-generic
  • Others: chat version 1.1.26
gaius-qi commented

Please show your configuration file.

gaius-qi commented

https://d7y.io/docs/next/reference/configuration/manager

# Job configuration.
job:
  # Sync peers configuration.
  syncPeers:
    # Interval is the interval for syncing all peers information from the scheduler and
    # display peers information in the manager console.
    interval: 24h
    # Timeout is the timeout for syncing peers information from the single scheduler.
    timeout: 10m
  # Preheat configuration.
  preheat:
    # registryTimeout is the timeout for requesting registry to get token and manifest.
    registryTimeout: 1m
    tls:
      # caCert is the CA certificate for preheat tls handshake, it can be path or PEM format string.
      caCert: ''
mansoncui commented

client-cert.pem file config：
image
or
image

error info：
2024-04-02T11:14:09Z [INFO] [/pkg/p2p/preheat/job.go:93]: Preheating image 'dragonfly/bitnami-shell:11-debian-11-r72@sha256:5098aa7e6360c5ee5e38152e49e03faaee8a57d5a8dcd94ea00c5490dbe87858' to the target preheat provider: dragonfly Dragonfly:http://10.215.40.234:8080
2024-04-02T11:14:09Z [INFO] [/pkg/p2p/preheat/job.go:120]: Get preheat provider driver: dragonfly
2024-04-02T11:14:09Z [INFO] [/pkg/p2p/preheat/job.go:134]: Check health of preheat provider instance: Healthy
2024-04-02T11:14:09Z [ERROR] [/pkg/p2p/preheat/job.go:75]: POST 'http://10.215.40.234:8080/preheats' error: 500 Internal Server Error {"message":"Get "https://harbor-wf.internal.zenmen.com/v2/\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}

gaius-qi commented

client-cert.pem file config：

image

or

image

error info：

2024-04-02T11:14:09Z [INFO] [/pkg/p2p/preheat/job.go:93]: Preheating image 'dragonfly/bitnami-shell:11-debian-11-r72@sha256:5098aa7e6360c5ee5e38152e49e03faaee8a57d5a8dcd94ea00c5490dbe87858' to the target preheat provider: dragonfly Dragonfly:http://10.215.40.234:8080

2024-04-02T11:14:09Z [INFO] [/pkg/p2p/preheat/job.go:120]: Get preheat provider driver: dragonfly

2024-04-02T11:14:09Z [INFO] [/pkg/p2p/preheat/job.go:134]: Check health of preheat provider instance: Healthy

2024-04-02T11:14:09Z [ERROR] [/pkg/p2p/preheat/job.go:75]: POST 'http://10.215.40.234:8080/preheats' error: 500 Internal Server Error {"message":"Get "https://harbor-wf.internal.zenmen.com/v2/\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}

You can add a log to confirm whether the ca cert is loaded correctly.

mansoncui commented

client-cert.pem 文件配置：
图像
或
图像
错误信息：
2024-04-02T11：14：09Z [INFO] [/pkg/p2p/preheat/job.go：93]：预热映像“dragonfly/bitnami-shell：11-debian-11-r72@sha256：5098aa7e6360c5ee5e38152e49e03faaee8a57d5a8dcd94ea00c5490dbe87858”到目标预热提供者：dragonfly 蜻蜓：http://10.215.40.234:8080
2024-04-02T11：14：09Z [信息] [/pkg/p2p/preheat/job.go：120]：获取预热提供程序驱动程序：蜻蜓
2024-04-02T11：14：09Z [信息] [/pkg/p2p/preheat/job.go：134]：检查预热提供程序实例的运行状况：健康
2024-04-02T11：14：09Z [错误] [/pkg/p2p/preheat/job.go：75]：POST “http://10.215.40.234:8080/preheats”错误：500 内部服务器错误 {“message”：“获取”[https://harbor-wf.internal.zenmen.com/v2/](https://harbor-wf.internal.zenmen.com/v2/%5C)“：tls：无法验证证书：x509：证书由未知机构签名”}

您可以添加日志来确认 ca 证书是否正确加载。

tail -f /var/log/dragonfly/manager/gin.log error info:
{"level":"error","ts":"2024-04-03 03:43:17.687","caller":"gin@v1.9.1/context.go:174","msg":"Get "https://harbor-wf.internal.zenmen.com/v2/\": tls: failed to verify certificate: x509: certificate signed by unknown authority","status":500,"method":"POST","path":"/preheats","query":"","ip":"10.2.72.38","user-agent":"Go-http-client/1.1","latency":0.01649769,"time":"2024-04-03T03:43:17Z","stacktrace":"github.com/gin-gonic/gin.(*Context).Next\n\t/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174\ngithub.com/gin-gonic/gin.CustomRecoveryWithWriter.func1\n\t/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/recovery.go:102\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174\ngithub.com/mcuadros/go-gin-prometheus.(*Prometheus).Use.(*Prometheus).HandlerFunc.func1\n\t/go/pkg/mod/github.com/mcuadros/go-gin-prometheus@v0.1.0/middleware.go:364\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174\ngithub.com/gin-gonic/gin.(*Engine).handleHTTPRequest\n\t/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:620\ngithub.com/gin-gonic/gin.(*Engine).ServeHTTP\n\t/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:576\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}

mansoncui commented

client-cert.pem 文件配置：
图像
或
图像
错误信息：
2024-04-02T11：14：09Z [INFO] [/pkg/p2p/preheat/job.go：93]：预热映像“dragonfly/bitnami-shell：11-debian-11-r72@sha256：5098aa7e6360c5ee5e38152e49e03faaee8a57d5a8dcd94ea00c5490dbe87858”到目标预热提供者：dragonfly 蜻蜓：http://10.215.40.234:8080
2024-04-02T11：14：09Z [信息] [/pkg/p2p/preheat/job.go：120]：获取预热提供程序驱动程序：蜻蜓
2024-04-02T11：14：09Z [信息] [/pkg/p2p/preheat/job.go：134]：检查预热提供程序实例的运行状况：健康
2024-04-02T11：14：09Z [错误] [/pkg/p2p/preheat/job.go：75]：POST “http://10.215.40.234:8080/preheats”错误：500 内部服务器错误 {“message”：“获取”https://harbor-wf.internal.zenmen.com/v2/“：tls：无法验证证书：x509：证书由未知机构签名”}

您可以添加日志来确认 ca 证书是否正确加载。

manager tail -f /var/log/dragonfly/manager/gin.log error info: {"level":"error","ts":"2024-04-03 03:43:17.687","caller":"gin@v1.9.1/context.go:174","msg":"Get "[https://harbor-wf.internal.zenmen.com/v2/](https://harbor-wf.internal.zenmen.com/v2/%5C)": tls: failed to verify certificate: x509: certificate signed by unknown authority","status":500,"method":"POST","path":"/preheats","query":"","ip":"10.2.72.38","user-agent":"Go-http-client/1.1","latency":0.01649769,"time":"2024-04-03T03:43:17Z","stacktrace":"github.com/gin-gonic/gin.(*Context).Next\n\t/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174\ngithub.com/gin-gonic/gin.CustomRecoveryWithWriter.func1\n\t/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/recovery.go:102\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174\ngithub.com/mcuadros/go-gin-prometheus.(*Prometheus).Use.(*Prometheus).HandlerFunc.func1\n\t/go/pkg/mod/github.com/mcuadros/go-gin-prometheus@v0.1.0/middleware.go:364\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174\ngithub.com/gin-gonic/gin.(*Engine).handleHTTPRequest\n\t/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:620\ngithub.com/gin-gonic/gin.(*Engine).ServeHTTP\n\t/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:576\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}

manager: tail -f /var/log/dragonfly/manager/core.log
{"level":"debug","ts":"2024-04-03 11:07:30.670","caller":"rpcserver/manager_server_v1.go:540","msg":"list schedulers, version v2.1.0, commit 31f0155","hostname":"ops-k8s-wx3node96.wxxdc","ip":"10.2.40.165"}
{"level":"warn","ts":"2024-04-03 11:07:30.670","caller":"rpcserver/manager_server_v1.go:548","msg":"manager:peers:ops-k8s-wx3node96.wxxdc-10.2.40.165:schedulers cache miss because of cache: key is missing","hostname":"ops-k8s-wx3node96.wxxdc","ip":"10.2.40.165","stacktrace":"d7y.io/dragonfly/v2/manager/rpcserver.(*managerServerV1).ListSchedulers\n\t/go/src/d7y.io/dragonfly/v2/manager/rpcserver/manager_server_v1.go:548\nd7y.io/api/v2/pkg/apis/manager/v1._Manager_ListSchedulers_Handler.func1\n\t/go/pkg/mod/d7y.io/api/v2@v2.0.79/pkg/apis/manager/v1/manager_grpc.pb.go:362\ngithub.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/recovery/interceptors.go:33\nd7y.io/dragonfly/v2/pkg/rpc/manager/server.New.ChainUnaryServer.func5.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48\nd7y.io/dragonfly/v2/pkg/rpc/manager/server.New.UnaryServerInterceptor.func3\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/validator/validator.go:47\nd7y.io/dragonfly/v2/pkg/rpc/manager/server.New.ChainUnaryServer.func5.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48\ngithub.com/grpc-ecosystem/go-grpc-middleware/logging/zap.UnaryServerInterceptor.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/logging/zap/server_interceptors.go:31\nd7y.io/dragonfly/v2/pkg/rpc/manager/server.New.ChainUnaryServer.func5.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48\ngithub.com/grpc-ecosystem/go-grpc-prometheus.init.(*ServerMetrics).UnaryServerInterceptor.func3\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:107\nd7y.io/dragonfly/v2/pkg/rpc/manager/server.New.ChainUnaryServer.func5.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48\nd7y.io/dragonfly/v2/pkg/rpc/manager/server.New.UnaryServerInterceptor.func2\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/ratelimit/ratelimit.go:24\nd7y.io/dragonfly/v2/pkg/rpc/manager/server.New.ChainUnaryServer.func5\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:53\nd7y.io/api/v2/pkg/apis/manager/v1._Manager_ListSchedulers_Handler\n\t/go/pkg/mod/d7y.io/api/v2@v2.0.79/pkg/apis/manager/v1/manager_grpc.pb.go:364\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.1/server.go:1372\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.1/server.go:1783\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.1/server.go:1016"}
{"level":"debug","ts":"2024-04-03 11:07:30.671","caller":"recovery/interceptors.go:33","msg":"trace","elapsed":0.000547491,"rows":1,"sql":"SELECT * FROM scheduler WHERE scheduler.scheduler_cluster_id = 1 AND state = 'active' AND scheduler.is_del = 0"}
{"level":"debug","ts":"2024-04-03 11:07:30.672","caller":"recovery/interceptors.go:33","msg":"trace","elapsed":0.00033995,"rows":1,"sql":"SELECT * FROM seed_peer_cluster_scheduler_cluster WHERE seed_peer_cluster_scheduler_cluster.scheduler_cluster_id = 1"}
{"level":"debug","ts":"2024-04-03 11:07:30.672","caller":"recovery/interceptors.go:33","msg":"trace","elapsed":0.000248708,"rows":1,"sql":"SELECT * FROM seed_peer WHERE seed_peer.seed_peer_cluster_id = 1 AND state = 'active' AND seed_peer.is_del = 0"}
{"level":"debug","ts":"2024-04-03 11:07:30.672","caller":"recovery/interceptors.go:33","msg":"trace","elapsed":0.000617042,"rows":1,"sql":"SELECT * FROM seed_peer_cluster WHERE seed_peer_cluster.id = 1 AND seed_peer_cluster.is_del = 0"}
{"level":"debug","ts":"2024-04-03 11:07:30.673","caller":"recovery/interceptors.go:33","msg":"trace","elapsed":0.002348206,"rows":1,"sql":"SELECT * FROM scheduler_cluster WHERE scheduler_cluster.is_del = 0"}
{"level":"debug","ts":"2024-04-03 11:07:30.673","caller":"rpcserver/manager_server_v1.go:577","msg":"list scheduler clusters [cluster-1] with hostInfo map[string]string{"idc":"", "location":""}","hostname":"ops-k8s-wx3node96.wxxdc","ip":"10.2.40.165"}
{"level":"debug","ts":"2024-04-03 11:07:30.673","caller":"rpcserver/manager_server_v1.go:592","msg":"find matching scheduler cluster [cluster-1]","hostname":"ops-k8s-wx3node96.wxxdc","ip":"10.2.40.165"}

mansoncui commented

您可以添加日志来确认 ca 证书是否正确加载

Configure manager log? I have seen error messages in manager core and gin.