The dfget daemon program channel is created repeatedly
The-half opened this issue · 1 comments
The-half commented
Bug report:
在本地执行dfget daemon代理程序，用来代理gitlab 的lfs 仓库，执行命令为:"dfget daemon --config=dfget-peer.yaml --verbose" 但执行后一直报出上述问题，channnel 每隔十秒重试连接；补充， dragonfly 服务端均已使用二进制部署完成
- dfget daemon 的日志如下：
2024-09-02T15:45:10.316+0800 INFO grpclog/grpclog.go:37 [core][Channel #51]Channel created
2024-09-02T15:45:10.316+0800 INFO grpclog/grpclog.go:37 [core][Channel #51]parsed dial target is: resolver.Target{URL:url.URL{Scheme:"passthrough", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"/10.10.101.123:8002", RawPath:"", OmitHost:false, ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}}
2024-09-02T15:45:10.316+0800 INFO grpclog/grpclog.go:37 [core][Channel #51]Channel authority set to "10.10.101.123:8002"
2024-09-02T15:45:10.317+0800 INFO grpclog/grpclog.go:37 [core][Channel #51]Resolver state updated: {
"Addresses": [
{
"Addr": "10.10.101.123:8002",
"ServerName": "",
"Attributes": null,
"BalancerAttributes": null,
"Metadata": null
}
],
"Endpoints": [
{
"Addresses": [
{
"Addr": "10.10.101.123:8002",
"ServerName": "",
"Attributes": null,
"BalancerAttributes": null,
"Metadata": null
}
],
"Attributes": null
}
],
"ServiceConfig": null,
"Attributes": null
} (resolver returned new addresses)
2024-09-02T15:45:10.317+0800 INFO grpclog/grpclog.go:37 [core][Channel #51]Channel switches to new LB policy "pick_first"
2024-09-02T15:45:10.317+0800 INFO grpclog/grpclog.go:37 [core][Channel #51 SubChannel #52]Subchannel created
2024-09-02T15:45:10.317+0800 INFO grpclog/grpclog.go:37 [core][Channel #51]Channel Connectivity change to CONNECTING
2024-09-02T15:45:10.317+0800 INFO grpclog/grpclog.go:37 [core][Channel #51]Channel exiting idle mode
2024-09-02T15:45:10.317+0800 INFO grpclog/grpclog.go:37 [core][Channel #51 SubChannel #52]Subchannel Connectivity change to CONNECTING
2024-09-02T15:45:10.317+0800 INFO grpclog/grpclog.go:37 [core][Channel #51 SubChannel #52]Subchannel picks a new address "10.10.101.123:8002" to connect
2024-09-02T15:45:10.319+0800 INFO grpclog/grpclog.go:37 [core][Channel #51 SubChannel #52]Subchannel Connectivity change to READY
2024-09-02T15:45:10.319+0800 INFO grpclog/grpclog.go:37 [core][Channel #51]Channel Connectivity change to READY
2024-09-02T15:45:10.321+0800 DEBUG zap/client_interceptors.go:52 finished client unary call {"system": "grpc", "span.kind": "client", "grpc.service": "grpc.health.v1.Health", "grpc.method": "Check", "grpc.code": "OK", "grpc.time_ms": 4.392}
2024-09-02T15:45:10.321+0800 INFO grpclog/grpclog.go:37 [core][Channel #51]Channel Connectivity change to SHUTDOWN
2024-09-02T15:45:10.322+0800 INFO grpclog/grpclog.go:37 [core][Channel #51]Closing the name resolver
2024-09-02T15:45:10.322+0800 INFO grpclog/grpclog.go:37 [core][Channel #51]ccBalancerWrapper: closing
2024-09-02T15:45:10.322+0800 INFO grpclog/grpclog.go:37 [core][Channel #51 SubChannel #52]Subchannel Connectivity change to SHUTDOWN
2024-09-02T15:45:10.322+0800 INFO grpclog/grpclog.go:37 [core][Channel #51 SubChannel #52]Subchannel deleted
2024-09-02T15:45:10.322+0800 INFO grpclog/grpclog.go:37 [core][Channel #51]Channel deleted
- dfget-peer.yaml的配置文件如下:
aliveTime: 0s
gcInterval: 1m0s
workHome: ''
workHomeMode: 0755
logDir: ''
cacheDir: ''
cacheDirMode: 0755
pluginDir: ''
dataDir: ''
dataDirMode: 0755
keepStorage: false
console: true
verbose: true
pprof-port: -1
jaeger: ''
scheduler:
manager:
enable: true
netAddrs:
- type: tcp
addr: 10.10.101.123:65003
refreshInterval: 10m
seedPeer:
enable: true
type: super
clusterID: 1
keepAlive:
internal: 5s
scheduleTimeout: 30s
disableAutoBackSource: false
host:
location: ''
idc: ''
download:
calculateDigest: false
totalRateLimit: 1024Mi
perPeerRateLimit: 512Mi
pieceDownloadTimeout: 30s
prefetch: false
transportOption:
dialTimeout: 2s
keepAlive: 30s
maxIdleConns: 100
idleConnTimeout: 90s
responseHeaderTimeout: 2s
tlsHandshakeTimeout: 1s
expectContinueTimeout: 2s
resourceClients:
https:
dialTimeout: 30s
keepAlive: 30s
maxIdleConns: 100
idleConnTimeout: 90s
responseHeaderTimeout: 30s
tlsHandshakeTimeout: 30s
expectContinueTimeout: 10s
insecureSkipVerify: true
http:
dialTimeout: 30s
keepAlive: 30s
maxIdleConns: 100
idleConnTimeout: 90s
responseHeaderTimeout: 30s
tlsHandshakeTimeout: 30s
expectContinueTimeout: 10s
oras:
dialTimeout: 30s
keepAlive: 30s
maxIdleConns: 100
idleConnTimeout: 90s
responseHeaderTimeout: 30s
tlsHandshakeTimeout: 30s
expectContinueTimeout: 10s
insecureSkipVerify: true
concurrent:
thresholdSize: 10M
thresholdSpeed: 2M
goroutineCount: 4
initBackoff: 0.5
maxBackoff: 3
maxAttempts: 3
downloadGRPC:
security:
insecure: true
cacert: ''
cert: ''
key: ''
tlsVerify: true
tlsConfig: null
unixListen:
socket: ''
peerGRPC:
security:
insecure: true
cacert: ''
cert: ''
key: ''
tlsVerify: true
tcpListen:
port: 65000
upload:
rateLimit: 1024Mi
security:
insecure: true
cacert: ''
cert: ''
key: ''
tlsVerify: false
tcpListen:
port: 65002
objectStorage:
enable: false
filter: 'Expires&Signature&ns'
maxReplicas: 3
security:
insecure: true
tlsVerify: true
tcpListen:
port: 65004
storage:
taskExpireTime: 6h
strategy: io.d7y.storage.v2.simple
diskGCThreshold: 50Gi
diskGCThresholdPercent: 80
multiplex: true
health:
security:
insecure: true
cacert: ''
cert: ''
key: ''
tlsVerify: false
tcpListen:
path: /server/ping
port: 40901
proxy:
defaultFilter: 'Expires&Signature&ns'
defaultTag: ''
security:
insecure: true
cacert: ''
cert: ''
key: ''
tlsVerify: false
tcpListen:
namespace: ''
port: 65001
registryMirror:
dynamic: false
url: https://index.docker.io
insecure: true
certs: []
direct: false
useProxies: false
proxies:
- regx: blobs/sha256.*
- regx: some-registry/
useHTTPS: true
- regx: no-proxy-reg
direct: true
- regx: some-registry
redirect: another-registry
- regx: ^http://some-registry/(.*)
redirect: http://another-registry/$1
- regex: '.*gitlab.example.com:32415'
use_https: true
insecure: true
redirect: 'http://localhost:65002'
hijackHTTPS:
cert: ''
key: ''
hosts:
- regx: gitlab.example.com:32415
use_https: true
insecure: true
certs: []
maxConcurrency: 0
whiteList:
- host: ''
regx:
ports:
basicAuth:
username: 'admin'
password: 'password'
security:
autoIssueCert: false
caCert: ''
tlsVerify: false
tlsPolicy: 'prefer'
certSpec:
dnsNames:
ipAddresses:
validityPeriod: 4320h
network:
enableIPv6: false
announcer:
schedulerInterval: 30s
networkTopology:
enable: true
probe:
interval: 20m- 其中dragonfly 的manager 程序的 /var/log/dragonfly/manger/core.log 的部分日志字段为
{"level":"warn","ts":"2024-09-02 11:50:44.193","caller":"rpcserver/manager_server_v2.go:682","msg":"manager:applications cache miss because of cache: key is missing","hostname":"k8s-master","ip":"10.10.101.123","stacktrace":"d7y.io/dragonfly/v2/manager/rpcserver.(*managerServerV2).ListApplications\n\t/home/runner/work/Dragonfly2/Dragonfly2/manager/rpcserver/manager_server_v2.go:682\nd7y.io/api/v2/pkg/apis/manager/v2._Manager_ListApplications_Handler.func1\n\t/home/runner/go/pkg/mod/d7y.io/api/v2@v2.0.148/pkg/apis/manager/v2/manager_grpc.pb.go:366\ngithub.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/recovery/interceptors.go:33\nd7y.io/dragonfly/v2/pkg/rpc/manager/server.New.ChainUnaryServer.func7.1\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48\nd7y.io/dragonfly/v2/pkg/rpc/manager/server.New.UnaryServerInterceptor.func5\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/validator/validator.go:47\nd7y.io/dragonfly/v2/pkg/rpc/manager/server.New.ChainUnaryServer.func7.1\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48\ngithub.com/grpc-ecosystem/go-grpc-middleware/logging/zap.UnaryServerInterceptor.func1\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/logging/zap/server_interceptors.go:31\nd7y.io/dragonfly/v2/pkg/rpc/manager/server.New.ChainUnaryServer.func7.1\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48\ngithub.com/grpc-ecosystem/go-grpc-prometheus.init.(*ServerMetrics).UnaryServerInterceptor.func3\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:107\nd7y.io/dragonfly/v2/pkg/rpc/manager/server.New.ChainUnaryServer.func7.1\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48\nd7y.io/dragonfly/v2/pkg/rpc/manager/server.New.UnaryServerInterceptor.func4\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/ratelimit/ratelimit.go:24\nd7y.io/dragonfly/v2/pkg/rpc/manager/server.New.ChainUnaryServer.func7\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:53\nd7y.io/api/v2/pkg/apis/manager/v2._Manager_ListApplications_Handler\n\t/home/runner/go/pkg/mod/d7y.io/api/v2@v2.0.148/pkg/apis/manager/v2/manager_grpc.pb.go:368\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1379\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1790\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1029"}
会不会 是dragonfly 的服务端的listApplications 访问不到，导致的客户端grpc 连接重试
Environment:
- Dragonfly version: v2.1.55
- OS: centos 7.9
- Kernel (e.g.
uname -a): 3.10.0-693.el7.x86_64 - Others: 使用 二进制部署的dragonfly 服务端，服务端均正常，可以正常访问网页
gaius-qi commented