s12v/awsbeats

streams: Panic on retrying

Closed this issue · 0 comments

After #28, it seems like filebeat+awsbeats+kinesis data streams occasionally fail while retrying batch send to Kinesis:

2018-05-28T03:35:56.266Z	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":90,"time":97},"total":{"ticks":710,"time":723,"value":710},"user":{"ticks":620,"time":626}},"info":{"ephemeral_id":"6703632b-8559-4104-8e5a-5b9d64489e08","uptime":{"ms":60027}},"memstats":{"gc_next":18773184,"memory_alloc":13742984,"memory_total":136054552,"rss":27189248}},"filebeat":{"events":{"active":4568,"added":4578,"done":10},"harvester":{"open_files":30,"running":30,"started":30}},"libbeat":{"config":{"module":{"running":1,"starts":1},"reloads":2},"output":{"events":{"acked":4146,"batches":83,"total":4146}},"pipeline":{"clients":4,"events":{"active":4067,"filtered":404,"published":4166,"retry":50,"total":4570},"queue":{"acked":100}}},"registrar":{"states":{"current":70,"update":10},"writes":10},"system":{"load":{"1":8.43,"15":7.16,"5":7.46,"norm":{"1":4.215,"15":3.58,"5":3.73}}}}}}
2018-05-28T03:35:58.686Z	INFO	log/harvester.go:216	Harvester started for file: /var/lib/docker/containers/26bbe1f1660ab24647e5d0342d700fe820776312d3fb4603e07831a6eceb1dbf/26bbe1f1660ab24647e5d0342d700fe820776312d3fb4603e07831a6eceb1dbf-json.log
2018-05-28T03:36:12.918Z	INFO	log/harvester.go:216	Harvester started for file: /var/lib/docker/containers/26bbe1f1660ab24647e5d0342d700fe820776312d3fb4603e07831a6eceb1dbf/26bbe1f1660ab24647e5d0342d700fe820776312d3fb4603e07831a6eceb1dbf-json.log
2018-05-28T03:36:17.215Z	ERROR	registrar/registrar.go:246	Writing of registry returned error: rename /usr/share/filebeat/data/registry.new /usr/share/filebeat/data/registry: no such file or directory. Continuing...
2018-05-28T03:36:17.465Z	ERROR	registrar/registrar.go:246	Writing of registry returned error: rename /usr/share/filebeat/data/registry.new /usr/share/filebeat/data/registry: no such file or directory. Continuing...
2018-05-28T03:36:21.155Z	ERROR	registrar/registrar.go:246	Writing of registry returned error: rename /usr/share/filebeat/data/registry.new /usr/share/filebeat/data/registry: no such file or directory. Continuing...
2018-05-28T03:36:22.237Z	WARN	streams/client.go:144	Retrying 50 events
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xd74489]

goroutine 71 [running]:
github.com/elastic/beats/libbeat/publisher/pipeline.(*Batch).Drop(0xc42168a000)
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/batch.go:59 +0x29
github.com/elastic/beats/libbeat/publisher/pipeline.(*retryer).loop(0xc420139fc0)
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:140 +0x4ff
created by github.com/elastic/beats/libbeat/publisher/pipeline.newRetryer
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:70 +0x138