actiontech/dtle

task restart will cause the number of goroutines to keep growing

LordofAvernus opened this issue · 2 comments

LordofAvernus commented

Description

common task restart will cause the number of goroutines to keep growing

Steps to reproduce the issue

  1. create normal dtle job(MySQL-MySQL)
***
  1. close source database
systemctl stop mysqld.service
  1. check groutine number
    http://ip:port/debug/pprof/goroutine?debug=1
goroutine profile: total 3579
2827 @ 0x43aad6 0x40862c 0x408098 0x122fe69 0x11c5713 0xb0a606 0xb0bde5 0xb05678 0x46b1a1
#	0x122fe68	github.com/hashicorp/nomad/plugins/drivers.(*driverPluginServer).TaskStats+0x268	/root/workspace/dtle/vendor/github.com/hashicorp/nomad/plugins/drivers/server.go:251
#	0x11c5712	github.com/hashicorp/nomad/plugins/drivers/proto._Driver_TaskStats_Handler+0xd2		/root/workspace/dtle/vendor/github.com/hashicorp/nomad/plugins/drivers/proto/driver.pb.go:4509
#	0xb0a605	google.golang.org/grpc.(*Server).processStreamingRPC+0xd05				/root/workspace/dtle/vendor/google.golang.org/grpc/server.go:1329
#	0xb0bde4	google.golang.org/grpc.(*Server).handleStream+0x9a4					/root/workspace/dtle/vendor/google.golang.org/grpc/server.go:1409
#	0xb05677	google.golang.org/grpc.(*Server).serveStreams.func1.1+0x97				/root/workspace/dtle/vendor/google.golang.org/grpc/server.go:746

668 @ 0x43aad6 0x407785 0x40733d 0xebbe78 0x46b1a1
#	0xebbe77	github.com/docker/libkv/store/consul.(*Consul).Watch.func1+0x197	/root/workspace/dtle/vendor/github.com/docker/libkv/store/consul/consul.go:314

8 @ 0x43aad6 0x4672dd 0x4672bd 0x480dcc 0x14e5f2e 0x46b1a1
#	0x4672bc	sync.runtime_notifyListWait+0x11c			/usr/local/go/src/runtime/sema.go:513
#	0x480dcb	sync.(*Cond).Wait+0x8b					/usr/local/go/src/sync/cond.go:56
#	0x14e5f2d	github.com/nats-io/nats%2ego.(*Conn).waitForMsgs+0xcd	/root/workspace/dtle/vendor/github.com/nats-io/nats.go/nats.go:2626

Output of ./dtle version:**

9.9.9.9-master-352256e
LordofAvernus commented

1. fix TaskStats goroutine leak

close taskstats channel when function is over

2. fix Watch.func1 goroutine leak

close the channel when watch consul kv function is return
LordofAvernus commented

Fixed pull request

 https://github.com/docker/libkv/pull/223