Why does the seed-peer in Dragonfly's multi-cluster environment cyclically connect to the scheduler of other clusters?

Question

Why does the seed-peer in Dragonfly's multi-cluster environment cyclically connect to the scheduler of other clusters?

Closed this issue 5 months ago · 9 comments

nicefuture2016 commented 6 months ago

I have deployed 13 Dragonfly clusters with a centralized architecture where one IDC serves as the central cluster deploying only the manage component. The other IDCs deploy seed-peer and scheduler components. The issue I've identified is that with each cluster deployment, its seed-peer continuously connects to the scheduler of the previously deployed cluster in a loop. This results in slower startup times for seed-peer in clusters deployed later, to the extent that the program may fail to start.

Please review the attachment, which includes deployment files and my values file.
This issue has been bothering me for several weeks.
thanks

issue.zip

Answer 1 · 2024-01-12T05:36:25.000Z

@nicefuture2016 Please upload the complete log. Currently there are only grpc logs and the problem cannot be located.

Answer 2 · 2024-01-12T06:20:04.000Z

@nicefuture2016 Please upload the complete log. Currently there are only grpc logs and the problem cannot be located.
cluster02.tar.gz
cluster01.tar.gz

cluster03.tar.gz

Answer 3 · 2024-01-12T06:24:39.000Z

@nicefuture2016 Please turn on debug level logs.

Answer 4 · 2024-01-12T06:48:14.000Z

@nicefuture2016 Please turn on debug level logs.

can you tell me how can I turn on debug level logs? thanks

Answer 5 · 2024-01-12T07:19:10.000Z

@nicefuture2016 Please turn on debug level logs.

cluster02.tar.gz
cluster01.tar.gz
cluster03.tar.gz

Answer 6 · 2024-01-17T06:49:35.000Z

The manager will evaluate based on attributes such as idc, location, etc, and then return the sorted clusters. The seed peer will check whether the scheduler address is accessible in the cluster. If it is not accessible, it will health check the scheduler address of the next cluster until it finds a cluster in which the scheduler address is accessible. The seed peer will use the scheduler of this cluster. It looks like the scheduler addresses of all your clusters are inaccessible.

{"level":"warn","ts":"2024-01-12 07:03:15.795","caller":"config/dynconfig_manager.go:110","msg":"scheduler ip address 172.20.140.124:8002 is unreachable: rpc error: code = DeadlineExceeded desc = context deadline exceeded","stacktrace":"d7y.io/dragonfly/v2/client/config.(*dynconfigManager).GetResolveSchedulerAddrs\n\t/go/src/d7y.io/dragonfly/v2/client/config/dynconfig_manager.go:110\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).ResolveNow\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:73\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).Build\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:64\ngoogle.golang.org/grpc.newCCResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/resolver_conn_wrapper.go:89\ngoogle.golang.org/grpc.(*ClientConn).initResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:2015\ngoogle.golang.org/grpc.(*ClientConn).exitIdleMode\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:393\ngoogle.golang.org/grpc.DialContext\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:263\nd7y.io/dragonfly/v2/pkg/rpc/scheduler/client.GetV1\n\t/go/src/d7y.io/dragonfly/v2/pkg/rpc/scheduler/client/client_v1.go:50\nd7y.io/dragonfly/v2/client/daemon.New\n\t/go/src/d7y.io/dragonfly/v2/client/daemon/daemon.go:226\nd7y.io/dragonfly/v2/cmd/dfget/cmd.runDaemon\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:194\nd7y.io/dragonfly/v2/cmd/dfget/cmd.glob..func1\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:80\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\nd7y.io/dragonfly/v2/cmd/dfget/cmd.Execute\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/root.go:120\nmain.main\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/main.go:24\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
{"level":"warn","ts":"2024-01-12 07:03:15.797","caller":"config/dynconfig_manager.go:115","msg":"scheduler host address dragonfly-scheduler-0.scheduler.dragonfly-system.svc.cluster.local.:8002 is unreachable: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp: lookup dragonfly-scheduler-0.scheduler.dragonfly-system.svc.cluster.local. on 10.68.0.2:53: no such host\"","stacktrace":"d7y.io/dragonfly/v2/client/config.(*dynconfigManager).GetResolveSchedulerAddrs\n\t/go/src/d7y.io/dragonfly/v2/client/config/dynconfig_manager.go:115\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).ResolveNow\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:73\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).Build\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:64\ngoogle.golang.org/grpc.newCCResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/resolver_conn_wrapper.go:89\ngoogle.golang.org/grpc.(*ClientConn).initResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:2015\ngoogle.golang.org/grpc.(*ClientConn).exitIdleMode\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:393\ngoogle.golang.org/grpc.DialContext\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:263\nd7y.io/dragonfly/v2/pkg/rpc/scheduler/client.GetV1\n\t/go/src/d7y.io/dragonfly/v2/pkg/rpc/scheduler/client/client_v1.go:50\nd7y.io/dragonfly/v2/client/daemon.New\n\t/go/src/d7y.io/dragonfly/v2/client/daemon/daemon.go:226\nd7y.io/dragonfly/v2/cmd/dfget/cmd.runDaemon\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:194\nd7y.io/dragonfly/v2/cmd/dfget/cmd.glob..func1\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:80\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\nd7y.io/dragonfly/v2/cmd/dfget/cmd.Execute\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/root.go:120\nmain.main\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/main.go:24\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
{"level":"warn","ts":"2024-01-12 07:03:15.797","caller":"config/dynconfig_manager.go:125","msg":"scheduler 172.20.140.124 dragonfly-scheduler-0.scheduler.dragonfly-system.svc.cluster.local. 8002 has not reachable addresses","stacktrace":"d7y.io/dragonfly/v2/client/config.(*dynconfigManager).GetResolveSchedulerAddrs\n\t/go/src/d7y.io/dragonfly/v2/client/config/dynconfig_manager.go:125\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).ResolveNow\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:73\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).Build\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:64\ngoogle.golang.org/grpc.newCCResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/resolver_conn_wrapper.go:89\ngoogle.golang.org/grpc.(*ClientConn).initResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:2015\ngoogle.golang.org/grpc.(*ClientConn).exitIdleMode\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:393\ngoogle.golang.org/grpc.DialContext\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:263\nd7y.io/dragonfly/v2/pkg/rpc/scheduler/client.GetV1\n\t/go/src/d7y.io/dragonfly/v2/pkg/rpc/scheduler/client/client_v1.go:50\nd7y.io/dragonfly/v2/client/daemon.New\n\t/go/src/d7y.io/dragonfly/v2/client/daemon/daemon.go:226\nd7y.io/dragonfly/v2/cmd/dfget/cmd.runDaemon\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:194\nd7y.io/dragonfly/v2/cmd/dfget/cmd.glob..func1\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:80\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\nd7y.io/dragonfly/v2/cmd/dfget/cmd.Execute\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/root.go:120\nmain.main\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/main.go:24\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
{"level":"warn","ts":"2024-01-12 07:03:17.797","caller":"config/dynconfig_manager.go:110","msg":"scheduler ip address 172.20.248.209:8002 is unreachable: rpc error: code = DeadlineExceeded desc = context deadline exceeded","stacktrace":"d7y.io/dragonfly/v2/client/config.(*dynconfigManager).GetResolveSchedulerAddrs\n\t/go/src/d7y.io/dragonfly/v2/client/config/dynconfig_manager.go:110\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).ResolveNow\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:73\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).Build\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:64\ngoogle.golang.org/grpc.newCCResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/resolver_conn_wrapper.go:89\ngoogle.golang.org/grpc.(*ClientConn).initResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:2015\ngoogle.golang.org/grpc.(*ClientConn).exitIdleMode\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:393\ngoogle.golang.org/grpc.DialContext\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:263\nd7y.io/dragonfly/v2/pkg/rpc/scheduler/client.GetV1\n\t/go/src/d7y.io/dragonfly/v2/pkg/rpc/scheduler/client/client_v1.go:50\nd7y.io/dragonfly/v2/client/daemon.New\n\t/go/src/d7y.io/dragonfly/v2/client/daemon/daemon.go:226\nd7y.io/dragonfly/v2/cmd/dfget/cmd.runDaemon\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:194\nd7y.io/dragonfly/v2/cmd/dfget/cmd.glob..func1\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:80\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\nd7y.io/dragonfly/v2/cmd/dfget/cmd.Execute\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/root.go:120\nmain.main\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/main.go:24\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
{"level":"warn","ts":"2024-01-12 07:03:17.799","caller":"config/dynconfig_manager.go:115","msg":"scheduler host address dragonfly-scheduler-1.scheduler.dragonfly-system.svc.cluster.local.:8002 is unreachable: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp: lookup dragonfly-scheduler-1.scheduler.dragonfly-system.svc.cluster.local. on 10.68.0.2:53: no such host\"","stacktrace":"d7y.io/dragonfly/v2/client/config.(*dynconfigManager).GetResolveSchedulerAddrs\n\t/go/src/d7y.io/dragonfly/v2/client/config/dynconfig_manager.go:115\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).ResolveNow\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:73\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).Build\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:64\ngoogle.golang.org/grpc.newCCResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/resolver_conn_wrapper.go:89\ngoogle.golang.org/grpc.(*ClientConn).initResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:2015\ngoogle.golang.org/grpc.(*ClientConn).exitIdleMode\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:393\ngoogle.golang.org/grpc.DialContext\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:263\nd7y.io/dragonfly/v2/pkg/rpc/scheduler/client.GetV1\n\t/go/src/d7y.io/dragonfly/v2/pkg/rpc/scheduler/client/client_v1.go:50\nd7y.io/dragonfly/v2/client/daemon.New\n\t/go/src/d7y.io/dragonfly/v2/client/daemon/daemon.go:226\nd7y.io/dragonfly/v2/cmd/dfget/cmd.runDaemon\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:194\nd7y.io/dragonfly/v2/cmd/dfget/cmd.glob..func1\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:80\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\nd7y.io/dragonfly/v2/cmd/dfget/cmd.Execute\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/root.go:120\nmain.main\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/main.go:24\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
{"level":"warn","ts":"2024-01-12 07:03:17.799","caller":"config/dynconfig_manager.go:125","msg":"scheduler 172.20.248.209 dragonfly-scheduler-1.scheduler.dragonfly-system.svc.cluster.local. 8002 has not reachable addresses","stacktrace":"d7y.io/dragonfly/v2/client/config.(*dynconfigManager).GetResolveSchedulerAddrs\n\t/go/src/d7y.io/dragonfly/v2/client/config/dynconfig_manager.go:125\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).ResolveNow\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:73\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).Build\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:64\ngoogle.golang.org/grpc.newCCResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/resolver_conn_wrapper.go:89\ngoogle.golang.org/grpc.(*ClientConn).initResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:2015\ngoogle.golang.org/grpc.(*ClientConn).exitIdleMode\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:393\ngoogle.golang.org/grpc.DialContext\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:263\nd7y.io/dragonfly/v2/pkg/rpc/scheduler/client.GetV1\n\t/go/src/d7y.io/dragonfly/v2/pkg/rpc/scheduler/client/client_v1.go:50\nd7y.io/dragonfly/v2/client/daemon.New\n\t/go/src/d7y.io/dragonfly/v2/client/daemon/daemon.go:226\nd7y.io/dragonfly/v2/cmd/dfget/cmd.runDaemon\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:194\nd7y.io/dragonfly/v2/cmd/dfget/cmd.glob..func1\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:80\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\nd7y.io/dragonfly/v2/cmd/dfget/cmd.Execute\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/root.go:120\nmain.main\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/main.go:24\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
{"level":"warn","ts":"2024-01-12 07:03:19.800","caller":"config/dynconfig_manager.go:110","msg":"scheduler ip address 172.20.186.234:8002 is unreachable: rpc error: code = DeadlineExceeded desc = context deadline exceeded","stacktrace":"d7y.io/dragonfly/v2/client/config.(*dynconfigManager).GetResolveSchedulerAddrs\n\t/go/src/d7y.io/dragonfly/v2/client/config/dynconfig_manager.go:110\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).ResolveNow\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:73\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).Build\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:64\ngoogle.golang.org/grpc.newCCResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/resolver_conn_wrapper.go:89\ngoogle.golang.org/grpc.(*ClientConn).initResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:2015\ngoogle.golang.org/grpc.(*ClientConn).exitIdleMode\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:393\ngoogle.golang.org/grpc.DialContext\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:263\nd7y.io/dragonfly/v2/pkg/rpc/scheduler/client.GetV1\n\t/go/src/d7y.io/dragonfly/v2/pkg/rpc/scheduler/client/client_v1.go:50\nd7y.io/dragonfly/v2/client/daemon.New\n\t/go/src/d7y.io/dragonfly/v2/client/daemon/daemon.go:226\nd7y.io/dragonfly/v2/cmd/dfget/cmd.runDaemon\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:194\nd7y.io/dragonfly/v2/cmd/dfget/cmd.glob..func1\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:80\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\nd7y.io/dragonfly/v2/cmd/dfget/cmd.Execute\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/root.go:120\nmain.main\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/main.go:24\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
{"level":"warn","ts":"2024-01-12 07:03:19.801","caller":"config/dynconfig_manager.go:115","msg":"scheduler host address dragonfly-scheduler-2.scheduler.dragonfly-system.svc.cluster.local.:8002 is unreachable: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp: lookup dragonfly-scheduler-2.scheduler.dragonfly-system.svc.cluster.local. on 10.68.0.2:53: no such host\"","stacktrace":"d7y.io/dragonfly/v2/client/config.(*dynconfigManager).GetResolveSchedulerAddrs\n\t/go/src/d7y.io/dragonfly/v2/client/config/dynconfig_manager.go:115\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).ResolveNow\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:73\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).Build\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:64\ngoogle.golang.org/grpc.newCCResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/resolver_conn_wrapper.go:89\ngoogle.golang.org/grpc.(*ClientConn).initResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:2015\ngoogle.golang.org/grpc.(*ClientConn).exitIdleMode\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:393\ngoogle.golang.org/grpc.DialContext\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:263\nd7y.io/dragonfly/v2/pkg/rpc/scheduler/client.GetV1\n\t/go/src/d7y.io/dragonfly/v2/pkg/rpc/scheduler/client/client_v1.go:50\nd7y.io/dragonfly/v2/client/daemon.New\n\t/go/src/d7y.io/dragonfly/v2/client/daemon/daemon.go:226\nd7y.io/dragonfly/v2/cmd/dfget/cmd.runDaemon\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:194\nd7y.io/dragonfly/v2/cmd/dfget/cmd.glob..func1\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:80\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\nd7y.io/dragonfly/v2/cmd/dfget/cmd.Execute\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/root.go:120\nmain.main\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/main.go:24\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
{"level":"warn","ts":"2024-01-12 07:03:19.801","caller":"config/dynconfig_manager.go:125","msg":"scheduler 172.20.186.234 dragonfly-scheduler-2.scheduler.dragonfly-system.svc.cluster.local. 8002 has not reachable addresses","stacktrace":"d7y.io/dragonfly/v2/client/config.(*dynconfigManager).GetResolveSchedulerAddrs\n\t/go/src/d7y.io/dragonfly/v2/client/config/dynconfig_manager.go:125\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).ResolveNow\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:73\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).Build\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:64\ngoogle.golang.org/grpc.newCCResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/resolver_conn_wrapper.go:89\ngoogle.golang.org/grpc.(*ClientConn).initResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:2015\ngoogle.golang.org/grpc.(*ClientConn).exitIdleMode\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:393\ngoogle.golang.org/grpc.DialContext\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:263\nd7y.io/dragonfly/v2/pkg/rpc/scheduler/client.GetV1\n\t/go/src/d7y.io/dragonfly/v2/pkg/rpc/scheduler/client/client_v1.go:50\nd7y.io/dragonfly/v2/client/daemon.New\n\t/go/src/d7y.io/dragonfly/v2/client/daemon/daemon.go:226\nd7y.io/dragonfly/v2/cmd/dfget/cmd.runDaemon\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:194\nd7y.io/dragonfly/v2/cmd/dfget/cmd.glob..func1\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:80\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\nd7y.io/dragonfly/v2/cmd/dfget/cmd.Execute\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/root.go:120\nmain.main\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/main.go:24\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}

Answer 7 · 2024-01-17T06:59:25.000Z

The manager will evaluate based on attributes such as idc, location, etc, and then return the sorted clusters. The seed peer will check whether the scheduler address is accessible in the cluster. If it is not accessible, it will health check the scheduler address of the next cluster until it finds a cluster in which the scheduler address is accessible. The seed peer will use the scheduler of this cluster. It looks like the scheduler addresses of all your clusters are inaccessible.

{"level":"warn","ts":"2024-01-12 07:03:15.795","caller":"config/dynconfig_manager.go:110","msg":"scheduler ip address 172.20.140.124:8002 is unreachable: rpc error: code = DeadlineExceeded desc = context deadline exceeded","stacktrace":"d7y.io/dragonfly/v2/client/config.(*dynconfigManager).GetResolveSchedulerAddrs\n\t/go/src/d7y.io/dragonfly/v2/client/config/dynconfig_manager.go:110\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).ResolveNow\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:73\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).Build\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:64\ngoogle.golang.org/grpc.newCCResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/resolver_conn_wrapper.go:89\ngoogle.golang.org/grpc.(*ClientConn).initResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:2015\ngoogle.golang.org/grpc.(*ClientConn).exitIdleMode\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:393\ngoogle.golang.org/grpc.DialContext\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:263\nd7y.io/dragonfly/v2/pkg/rpc/scheduler/client.GetV1\n\t/go/src/d7y.io/dragonfly/v2/pkg/rpc/scheduler/client/client_v1.go:50\nd7y.io/dragonfly/v2/client/daemon.New\n\t/go/src/d7y.io/dragonfly/v2/client/daemon/daemon.go:226\nd7y.io/dragonfly/v2/cmd/dfget/cmd.runDaemon\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:194\nd7y.io/dragonfly/v2/cmd/dfget/cmd.glob..func1\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:80\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\nd7y.io/dragonfly/v2/cmd/dfget/cmd.Execute\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/root.go:120\nmain.main\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/main.go:24\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
{"level":"warn","ts":"2024-01-12 07:03:15.797","caller":"config/dynconfig_manager.go:115","msg":"scheduler host address dragonfly-scheduler-0.scheduler.dragonfly-system.svc.cluster.local.:8002 is unreachable: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp: lookup dragonfly-scheduler-0.scheduler.dragonfly-system.svc.cluster.local. on 10.68.0.2:53: no such host\"","stacktrace":"d7y.io/dragonfly/v2/client/config.(*dynconfigManager).GetResolveSchedulerAddrs\n\t/go/src/d7y.io/dragonfly/v2/client/config/dynconfig_manager.go:115\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).ResolveNow\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:73\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).Build\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:64\ngoogle.golang.org/grpc.newCCResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/resolver_conn_wrapper.go:89\ngoogle.golang.org/grpc.(*ClientConn).initResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:2015\ngoogle.golang.org/grpc.(*ClientConn).exitIdleMode\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:393\ngoogle.golang.org/grpc.DialContext\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:263\nd7y.io/dragonfly/v2/pkg/rpc/scheduler/client.GetV1\n\t/go/src/d7y.io/dragonfly/v2/pkg/rpc/scheduler/client/client_v1.go:50\nd7y.io/dragonfly/v2/client/daemon.New\n\t/go/src/d7y.io/dragonfly/v2/client/daemon/daemon.go:226\nd7y.io/dragonfly/v2/cmd/dfget/cmd.runDaemon\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:194\nd7y.io/dragonfly/v2/cmd/dfget/cmd.glob..func1\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:80\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\nd7y.io/dragonfly/v2/cmd/dfget/cmd.Execute\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/root.go:120\nmain.main\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/main.go:24\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
{"level":"warn","ts":"2024-01-12 07:03:15.797","caller":"config/dynconfig_manager.go:125","msg":"scheduler 172.20.140.124 dragonfly-scheduler-0.scheduler.dragonfly-system.svc.cluster.local. 8002 has not reachable addresses","stacktrace":"d7y.io/dragonfly/v2/client/config.(*dynconfigManager).GetResolveSchedulerAddrs\n\t/go/src/d7y.io/dragonfly/v2/client/config/dynconfig_manager.go:125\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).ResolveNow\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:73\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).Build\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:64\ngoogle.golang.org/grpc.newCCResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/resolver_conn_wrapper.go:89\ngoogle.golang.org/grpc.(*ClientConn).initResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:2015\ngoogle.golang.org/grpc.(*ClientConn).exitIdleMode\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:393\ngoogle.golang.org/grpc.DialContext\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:263\nd7y.io/dragonfly/v2/pkg/rpc/scheduler/client.GetV1\n\t/go/src/d7y.io/dragonfly/v2/pkg/rpc/scheduler/client/client_v1.go:50\nd7y.io/dragonfly/v2/client/daemon.New\n\t/go/src/d7y.io/dragonfly/v2/client/daemon/daemon.go:226\nd7y.io/dragonfly/v2/cmd/dfget/cmd.runDaemon\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:194\nd7y.io/dragonfly/v2/cmd/dfget/cmd.glob..func1\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:80\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\nd7y.io/dragonfly/v2/cmd/dfget/cmd.Execute\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/root.go:120\nmain.main\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/main.go:24\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
{"level":"warn","ts":"2024-01-12 07:03:17.797","caller":"config/dynconfig_manager.go:110","msg":"scheduler ip address 172.20.248.209:8002 is unreachable: rpc error: code = DeadlineExceeded desc = context deadline exceeded","stacktrace":"d7y.io/dragonfly/v2/client/config.(*dynconfigManager).GetResolveSchedulerAddrs\n\t/go/src/d7y.io/dragonfly/v2/client/config/dynconfig_manager.go:110\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).ResolveNow\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:73\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).Build\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:64\ngoogle.golang.org/grpc.newCCResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/resolver_conn_wrapper.go:89\ngoogle.golang.org/grpc.(*ClientConn).initResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:2015\ngoogle.golang.org/grpc.(*ClientConn).exitIdleMode\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:393\ngoogle.golang.org/grpc.DialContext\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:263\nd7y.io/dragonfly/v2/pkg/rpc/scheduler/client.GetV1\n\t/go/src/d7y.io/dragonfly/v2/pkg/rpc/scheduler/client/client_v1.go:50\nd7y.io/dragonfly/v2/client/daemon.New\n\t/go/src/d7y.io/dragonfly/v2/client/daemon/daemon.go:226\nd7y.io/dragonfly/v2/cmd/dfget/cmd.runDaemon\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:194\nd7y.io/dragonfly/v2/cmd/dfget/cmd.glob..func1\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:80\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\nd7y.io/dragonfly/v2/cmd/dfget/cmd.Execute\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/root.go:120\nmain.main\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/main.go:24\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
{"level":"warn","ts":"2024-01-12 07:03:17.799","caller":"config/dynconfig_manager.go:115","msg":"scheduler host address dragonfly-scheduler-1.scheduler.dragonfly-system.svc.cluster.local.:8002 is unreachable: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp: lookup dragonfly-scheduler-1.scheduler.dragonfly-system.svc.cluster.local. on 10.68.0.2:53: no such host\"","stacktrace":"d7y.io/dragonfly/v2/client/config.(*dynconfigManager).GetResolveSchedulerAddrs\n\t/go/src/d7y.io/dragonfly/v2/client/config/dynconfig_manager.go:115\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).ResolveNow\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:73\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).Build\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:64\ngoogle.golang.org/grpc.newCCResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/resolver_conn_wrapper.go:89\ngoogle.golang.org/grpc.(*ClientConn).initResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:2015\ngoogle.golang.org/grpc.(*ClientConn).exitIdleMode\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:393\ngoogle.golang.org/grpc.DialContext\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:263\nd7y.io/dragonfly/v2/pkg/rpc/scheduler/client.GetV1\n\t/go/src/d7y.io/dragonfly/v2/pkg/rpc/scheduler/client/client_v1.go:50\nd7y.io/dragonfly/v2/client/daemon.New\n\t/go/src/d7y.io/dragonfly/v2/client/daemon/daemon.go:226\nd7y.io/dragonfly/v2/cmd/dfget/cmd.runDaemon\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:194\nd7y.io/dragonfly/v2/cmd/dfget/cmd.glob..func1\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:80\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\nd7y.io/dragonfly/v2/cmd/dfget/cmd.Execute\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/root.go:120\nmain.main\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/main.go:24\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
{"level":"warn","ts":"2024-01-12 07:03:17.799","caller":"config/dynconfig_manager.go:125","msg":"scheduler 172.20.248.209 dragonfly-scheduler-1.scheduler.dragonfly-system.svc.cluster.local. 8002 has not reachable addresses","stacktrace":"d7y.io/dragonfly/v2/client/config.(*dynconfigManager).GetResolveSchedulerAddrs\n\t/go/src/d7y.io/dragonfly/v2/client/config/dynconfig_manager.go:125\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).ResolveNow\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:73\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).Build\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:64\ngoogle.golang.org/grpc.newCCResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/resolver_conn_wrapper.go:89\ngoogle.golang.org/grpc.(*ClientConn).initResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:2015\ngoogle.golang.org/grpc.(*ClientConn).exitIdleMode\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:393\ngoogle.golang.org/grpc.DialContext\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:263\nd7y.io/dragonfly/v2/pkg/rpc/scheduler/client.GetV1\n\t/go/src/d7y.io/dragonfly/v2/pkg/rpc/scheduler/client/client_v1.go:50\nd7y.io/dragonfly/v2/client/daemon.New\n\t/go/src/d7y.io/dragonfly/v2/client/daemon/daemon.go:226\nd7y.io/dragonfly/v2/cmd/dfget/cmd.runDaemon\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:194\nd7y.io/dragonfly/v2/cmd/dfget/cmd.glob..func1\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:80\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\nd7y.io/dragonfly/v2/cmd/dfget/cmd.Execute\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/root.go:120\nmain.main\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/main.go:24\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
{"level":"warn","ts":"2024-01-12 07:03:19.800","caller":"config/dynconfig_manager.go:110","msg":"scheduler ip address 172.20.186.234:8002 is unreachable: rpc error: code = DeadlineExceeded desc = context deadline exceeded","stacktrace":"d7y.io/dragonfly/v2/client/config.(*dynconfigManager).GetResolveSchedulerAddrs\n\t/go/src/d7y.io/dragonfly/v2/client/config/dynconfig_manager.go:110\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).ResolveNow\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:73\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).Build\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:64\ngoogle.golang.org/grpc.newCCResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/resolver_conn_wrapper.go:89\ngoogle.golang.org/grpc.(*ClientConn).initResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:2015\ngoogle.golang.org/grpc.(*ClientConn).exitIdleMode\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:393\ngoogle.golang.org/grpc.DialContext\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:263\nd7y.io/dragonfly/v2/pkg/rpc/scheduler/client.GetV1\n\t/go/src/d7y.io/dragonfly/v2/pkg/rpc/scheduler/client/client_v1.go:50\nd7y.io/dragonfly/v2/client/daemon.New\n\t/go/src/d7y.io/dragonfly/v2/client/daemon/daemon.go:226\nd7y.io/dragonfly/v2/cmd/dfget/cmd.runDaemon\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:194\nd7y.io/dragonfly/v2/cmd/dfget/cmd.glob..func1\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:80\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\nd7y.io/dragonfly/v2/cmd/dfget/cmd.Execute\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/root.go:120\nmain.main\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/main.go:24\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
{"level":"warn","ts":"2024-01-12 07:03:19.801","caller":"config/dynconfig_manager.go:115","msg":"scheduler host address dragonfly-scheduler-2.scheduler.dragonfly-system.svc.cluster.local.:8002 is unreachable: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp: lookup dragonfly-scheduler-2.scheduler.dragonfly-system.svc.cluster.local. on 10.68.0.2:53: no such host\"","stacktrace":"d7y.io/dragonfly/v2/client/config.(*dynconfigManager).GetResolveSchedulerAddrs\n\t/go/src/d7y.io/dragonfly/v2/client/config/dynconfig_manager.go:115\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).ResolveNow\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:73\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).Build\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:64\ngoogle.golang.org/grpc.newCCResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/resolver_conn_wrapper.go:89\ngoogle.golang.org/grpc.(*ClientConn).initResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:2015\ngoogle.golang.org/grpc.(*ClientConn).exitIdleMode\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:393\ngoogle.golang.org/grpc.DialContext\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:263\nd7y.io/dragonfly/v2/pkg/rpc/scheduler/client.GetV1\n\t/go/src/d7y.io/dragonfly/v2/pkg/rpc/scheduler/client/client_v1.go:50\nd7y.io/dragonfly/v2/client/daemon.New\n\t/go/src/d7y.io/dragonfly/v2/client/daemon/daemon.go:226\nd7y.io/dragonfly/v2/cmd/dfget/cmd.runDaemon\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:194\nd7y.io/dragonfly/v2/cmd/dfget/cmd.glob..func1\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:80\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\nd7y.io/dragonfly/v2/cmd/dfget/cmd.Execute\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/root.go:120\nmain.main\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/main.go:24\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
{"level":"warn","ts":"2024-01-12 07:03:19.801","caller":"config/dynconfig_manager.go:125","msg":"scheduler 172.20.186.234 dragonfly-scheduler-2.scheduler.dragonfly-system.svc.cluster.local. 8002 has not reachable addresses","stacktrace":"d7y.io/dragonfly/v2/client/config.(*dynconfigManager).GetResolveSchedulerAddrs\n\t/go/src/d7y.io/dragonfly/v2/client/config/dynconfig_manager.go:125\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).ResolveNow\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:73\nd7y.io/dragonfly/v2/pkg/resolver.(*SchedulerResolver).Build\n\t/go/src/d7y.io/dragonfly/v2/pkg/resolver/scheduler_resolver.go:64\ngoogle.golang.org/grpc.newCCResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/resolver_conn_wrapper.go:89\ngoogle.golang.org/grpc.(*ClientConn).initResolverWrapper\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:2015\ngoogle.golang.org/grpc.(*ClientConn).exitIdleMode\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:393\ngoogle.golang.org/grpc.DialContext\n\t/go/pkg/mod/google.golang.org/grpc@v1.60.0-dev/clientconn.go:263\nd7y.io/dragonfly/v2/pkg/rpc/scheduler/client.GetV1\n\t/go/src/d7y.io/dragonfly/v2/pkg/rpc/scheduler/client/client_v1.go:50\nd7y.io/dragonfly/v2/client/daemon.New\n\t/go/src/d7y.io/dragonfly/v2/client/daemon/daemon.go:226\nd7y.io/dragonfly/v2/cmd/dfget/cmd.runDaemon\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:194\nd7y.io/dragonfly/v2/cmd/dfget/cmd.glob..func1\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/daemon.go:80\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\nd7y.io/dragonfly/v2/cmd/dfget/cmd.Execute\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/cmd/root.go:120\nmain.main\n\t/go/src/d7y.io/dragonfly/v2/cmd/dfget/main.go:24\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}

Could you help me check if my values file is configured correctly? i have no idea, thanks!
issue.pdf

Answer 8 · 2024-01-17T07:02:44.000Z

@nicefuture2016 It looks like the scheduler addresses of all your clusters are inaccessible. Please check your network.

Answer 9 · 2024-01-17T07:13:35.000Z

@nicefuture2016 It looks like the scheduler addresses of all your clusters are inaccessible. Please check your network.

I logged into my container and found that the seed-peer cannot resolve the scheduler's address. Is this the reason? My Kubernetes version is v1.18.2.

# Kubernetes-managed hosts file.
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
172.20.89.191   dragonfly-seed-peer-0.seed-peer.dragonfly-system.svc.cluster.local.     dragonfly-seed-peer-0
/ # ping dragonfly-scheduler-0.scheduler.dragonfly-system.svc.cluster.local.
ping: bad address 'dragonfly-scheduler-0.scheduler.dragonfly-system.svc.cluster.local.'
/ # ping dragonfly-scheduler-0.scheduler.dragonfly-system.svc.cluster.local
ping: bad address 'dragonfly-scheduler-0.scheduler.dragonfly-system.svc.cluster.local'
/ # ping dragonfly-scheduler-0
ping: bad address 'dragonfly-scheduler-0'