streamnative/pulsar-beat-output

beats stop collecting logs suddenly

SperChung-11 opened this issue · 2 comments

SperChung-11 commented

hi， i build pulsar-beat-output and get the binary file. When i use filebeat and metricbeat to collect logs, i found that they will stop collecting suddenly. Actually the logs are still printed by other service. There are no error log and i get the debug log as follows:

2020-12-22T04:00:57.538Z DEBUG [input] log/input.go:208 input states cleaned up. Before: 0, After: 0, Pending: 0
2020-12-22T04:00:57.538Z DEBUG [input] log/input.go:187 Start next scan
2020-12-22T04:00:57.538Z DEBUG [input] input/input.go:152 Run input
2020-12-22T04:00:57.538Z DEBUG [input] log/input.go:187 Start next scan
2020-12-22T04:00:57.539Z DEBUG [input] log/input.go:208 input states cleaned up. Before: 0, After: 0, Pending: 0
2020-12-22T04:00:57.539Z DEBUG [input] log/input.go:208 input states cleaned up. Before: 0, After: 0, Pending: 0
2020-12-22T04:00:57.539Z DEBUG [input] log/input.go:208 input states cleaned up. Before: 0, After: 0, Pending: 0
2020-12-22T04:00:57.539Z DEBUG [input] input/input.go:152 Run input
2020-12-22T04:00:57.539Z DEBUG [input] log/input.go:187 Start next scan
2020-12-22T04:00:57.539Z DEBUG [input] log/input.go:208 input states cleaned up. Before: 0, After: 0, Pending: 0
2020-12-22T04:00:57.540Z DEBUG [input] input/input.go:152 Run input
2020-12-22T04:00:57.540Z DEBUG [input] log/input.go:187 Start next scan
2020-12-22T04:00:57.540Z DEBUG [input] log/input.go:417 Check file for harvesting: /opt/container/edge/push/logs/push-running.log
2020-12-22T04:00:57.540Z DEBUG [input] log/input.go:507 Update existing file for harvesting: /opt/container/edge/push/logs/push-running.log, offset: 7733110
2020-12-22T04:00:57.540Z DEBUG [input] log/input.go:559 Harvester for file is still running: /opt/container/edge/push/logs/push-running.log
2020-12-22T04:00:57.540Z DEBUG [input] log/input.go:208 input states cleaned up. Before: 1, After: 1, Pending: 0
2020-12-22T04:00:57.540Z DEBUG [input] input/input.go:152 Run input
2020-12-22T04:00:57.540Z DEBUG [input] log/input.go:187 Start next scan
2020-12-22T04:00:57.540Z DEBUG [input] log/input.go:417 Check file for harvesting: /opt/container/ccops/metricbeat/logs/Metricbeat-runing.log
2020-12-22T04:00:57.540Z DEBUG [input] log/input.go:507 Update existing file for harvesting: /opt/container/ccops/metricbeat/logs/Metricbeat-runing.log, offset: 9928682
2020-12-22T04:00:57.540Z DEBUG [input] log/input.go:559 Harvester for file is still running: /opt/container/ccops/metricbeat/logs/Metricbeat-runing.log
2020-12-22T04:00:57.540Z DEBUG [input] log/input.go:417 Check file for harvesting: /opt/container/ccops/metricbeat/logs/metricbeat.log
2020-12-22T04:00:57.540Z DEBUG [input] log/input.go:507 Update existing file for harvesting: /opt/container/ccops/metricbeat/logs/metricbeat.log, offset: 4231944
2020-12-22T04:00:57.540Z DEBUG [input] log/input.go:559 Harvester for file is still running: /opt/container/ccops/metricbeat/logs/metricbeat.log
2020-12-22T04:00:57.540Z DEBUG [input] log/input.go:208 input states cleaned up. Before: 2, After: 2, Pending: 0
2020-12-22T04:01:09.725Z DEBUG [input] input/input.go:152 Run input
2020-12-22T04:01:09.725Z DEBUG [input] log/input.go:187 Start next scan
2020-12-22T04:01:09.725Z DEBUG [input] log/input.go:417 Check file for harvesting: /opt/filebeat/logs/Filebeat-runing.log
2020-12-22T04:01:09.725Z DEBUG [input] log/input.go:507 Update existing file for harvesting: /opt/filebeat/logs/Filebeat-runing.log, offset: 27382
2020-12-22T04:01:09.725Z DEBUG [input] log/input.go:559 Harvester for file is still running: /opt/filebeat/logs/Filebeat-runing.log
2020-12-22T04:01:09.725Z DEBUG [input] log/input.go:417 Check file for harvesting: /opt/filebeat/logs/filebeat.log
2020-12-22T04:01:09.725Z DEBUG [input] log/input.go:507 Update existing file for harvesting: /opt/iot/filebeat/logs/filebeat.log, offset: 112387
2020-12-22T04:01:09.725Z DEBUG [input] log/input.go:559 Harvester for file is still running: /opt/iot/filebeat/logs/filebeat.log
2020-12-22T04:01:09.725Z DEBUG [input] log/input.go:208 input states cleaned up. Before: 8, After: 8, Pending: 0
2020-12-22T04:01:09.725Z DEBUG [input] log/input.go:239 State for file not removed because harvester not finished: /opt/iot/filebeat/logs/filebeat.log
2020-12-22T04:01:09.725Z DEBUG [input] log/input.go:229 Remove state for file as file removed or renamed: /opt/iot/filebeat/logs/filebeat.log
2020-12-22T04:01:09.725Z DEBUG [input] log/input.go:239 State for file not removed because harvester not finished: /opt/iot/filebeat/logs/filebeat.log
2020-12-22T04:01:09.725Z DEBUG [input] log/input.go:229 Remove state for file as file removed or renamed: /opt/iot/filebeat/logs/filebeat.log
2020-12-22T04:01:09.725Z DEBUG [input] log/input.go:239 State for file not removed because harvester not finished: /opt/iot/filebeat/logs/filebeat.log
2020-12-22T04:01:09.725Z DEBUG [input] log/input.go:229 Remove state for file as file removed or renamed: /opt/iot/filebeat/logs/filebeat.log
2020-12-22T04:01:09.725Z DEBUG [input] log/input.go:239 State for file not removed because harvester not finished: /opt/iot/filebeat/logs/filebeat.log
2020-12-22T04:01:09.725Z DEBUG [input] log/input.go:229 Remove state for file as file removed or renamed: /opt/iot/filebeat/logs/filebeat.log
2020-12-22T04:01:09.725Z DEBUG [input] log/input.go:239 State for file not removed because harvester not finished: /opt/iot/filebeat/logs/filebeat.log
2020-12-22T04:01:09.725Z DEBUG [input] log/input.go:229 Remove state for file as file removed or renamed: /opt/iot/filebeat/logs/filebeat.log
2020-12-22T04:01:09.725Z DEBUG [input] log/input.go:239 State for file not removed because harvester not finished: /opt/iot/filebeat/logs/filebeat.log
2020-12-22T04:01:09.725Z DEBUG [input] log/input.go:229 Remove state for file as file removed or renamed: /opt/iot/filebeat/logs/filebeat.log

here are my filebeat.yml:
filebeat.inputs:
#=========================== ccops ELK log config ============================

  • type: log
    enabled: true
    paths:

    • /opt/container/ccops/metricbeat/logs/*.log
      fields:
      log_topic: ${LOG_TOPIC}
      logtype: ccops_metricbeat_log
      microservice: ccopsmetricbeat
      fields_under_root: true
      max_bytes: 512000
      close_timeout: 300m
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
      spool_size: 2048
      harvester_buffer_size: 102400
      scan_frequency: 30s
      idle_timeout: 20s
      registry_file: registry
      ignore_older: 1h

  • type: log
    enabled: true
    paths:

    • /opt/filebeat/logs/*.log
      fields:
      log_topic: ${LOG_TOPIC}
      logtype: ccops_filebeat_log
      microservice: ccopsfilebeat
      fields_under_root: true
      max_bytes: 512000
      close_timeout: 300m
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
      spool_size: 2048
      harvester_buffer_size: 102400
      scan_frequency: 30s
      idle_timeout: 20s
      registry_file: registry
      ignore_older: 1h

output.pulsar:
enabled: true
url: "pulsar://service-pulsar-ops:6650"
topic: ${LOG_TOPIC}

#================================ Logging ======================================
logging.level: debug
logging.to_files: true
logging.files:
path: /opt/filebeat/logs/
name: filebeat.log
keepfiles: 5
rotateeverybytes: 53687091
permissions: 0644
logging.json: false

may be too many logs leading to the problem ?

shoothzj commented

@SperChung-11 could you please try 0.3.0 https://github.com/streamnative/pulsar-beat-output/releases/tag/v0.3.0.
If you still have the problem, feel free to reopen it