td-agent within plugin "pgjson" lost connect with postgresql server
ttsrg opened this issue · 7 comments
ttsrg commented
If postgresql server restarts td-agent doesn't write any record at log and doesn't try to repeat connection to postgresql server, I have to do it manual
kenhys commented
Please describe your environment in details.
Without such information, it may be hard to answer for resolving the issue.
Distribution, td-agent version, flentd version, pgjson version, postgresql version and so on.
theasp commented
I think I'm seeing this too. I'm using Fluentd 1.14.2 from plugins docker image (specifically theasp/fluentd-plugins:edge-common) and PostgreSQL 14 and TimescaleBD (timescale/timescaledb:latest-pg14). When I restart the postgres container, fluentd stops sending data with the follow error every time it retries:
fluentd_fluentd.1.npvnhs7xtdfh@xxx | 2021-12-01 17:58:00 +0000 [warn]: #0 failed to flush the buffer. retry_times=8 next_retry_time=2021-12-01 18:00:04 +0000 chunk="5d219622fe0e44cbf6b5411026aa6f5e" error_class=PG::UnableToSend error="no connection to the server\n"
fluentd_fluentd.1.npvnhs7xtdfh@xxx | 2021-12-01 17:58:00 +0000 [warn]: #0 suppressed same stacktrace
Restarting fluentd fixes it. I'm guessing that something is not handling the loss of connection properly.
In fluent.conf I have:
<match **>
@type pgjson
host xxx_postgres
sslmode disable
database xxx
table fluentd_insert
user xxx
password xxx
<buffer>
@type file
path /fluentd/buffer/unmatched_pgjson
retry_max_interval 120
flush_interval 10s
</buffer>
</match>theasp commented
Stack trace is here:
fluentd_fluentd.1.t9l9xwwba6vy@xxx | 2021-12-01 14:57:25 +0000 [warn]: #0 failed to flush the buffer. retry_times=0 next_retry_time=2021-12-01 14:57:26 +0000 chunk="5d216e3acd8c08c8628cec5415d44d8e" error_class=PG::UnableToSend error="server closed the connection unexpectedly\n\tThis probably means the server terminated abnormally\n\tbefore or while processing the request.\n"
fluentd_fluentd.1.t9l9xwwba6vy@xxx | 2021-12-01 14:57:25 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-pgjson-1.0.1/lib/fluent/plugin/out_pgjson.rb:87:in `exec'
fluentd_fluentd.1.t9l9xwwba6vy@xxx | 2021-12-01 14:57:25 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-pgjson-1.0.1/lib/fluent/plugin/out_pgjson.rb:87:in `write'
fluentd_fluentd.1.t9l9xwwba6vy@xxx | 2021-12-01 14:57:25 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.14.2/lib/fluent/plugin/output.rb:1179:in `try_flush'
fluentd_fluentd.1.t9l9xwwba6vy@xxx | 2021-12-01 14:57:25 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.14.2/lib/fluent/plugin/output.rb:1491:in `flush_thread_run'
fluentd_fluentd.1.t9l9xwwba6vy@xxx | 2021-12-01 14:57:25 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.14.2/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start'
fluentd_fluentd.1.t9l9xwwba6vy@xxx | 2021-12-01 14:57:25 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.14.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
theasp commented
I think it works!
fluentd_fluentd.1.x4tffohknzr4@xxx | 2021-12-08 18:26:18 +0000 [warn]: #0 failed to flush the buffer. retry_times=0 next_retry_time=2021-12-08 18:26:19 +0000 chunk="5d2a69f9813388ca7ccafcf6eacb3793" error_class=PG::UnableToSend error="server closed the connection unexpectedly\n\tThis probably means the server terminated abnormally\n\tbefore or while processing the request.\n"
fluentd_fluentd.1.x4tffohknzr4@xxx | 2021-12-08 18:26:18 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-pgjson-1.0.2/lib/fluent/plugin/out_pgjson.rb:88:in `exec'
fluentd_fluentd.1.x4tffohknzr4@xxx | 2021-12-08 18:26:18 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-pgjson-1.0.2/lib/fluent/plugin/out_pgjson.rb:88:in `write'
fluentd_fluentd.1.x4tffohknzr4@xxx | 2021-12-08 18:26:18 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.14.3/lib/fluent/plugin/output.rb:1179:in `try_flush'
fluentd_fluentd.1.x4tffohknzr4@xxx | 2021-12-08 18:26:18 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.14.3/lib/fluent/plugin/output.rb:1491:in `flush_thread_run'
fluentd_fluentd.1.x4tffohknzr4@xxx | 2021-12-08 18:26:18 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.14.3/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start'
fluentd_fluentd.1.x4tffohknzr4@xxx | 2021-12-08 18:26:18 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.14.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
fluentd_fluentd.1.x4tffohknzr4@xxx | 2021-12-08 18:26:20 +0000 [warn]: #0 retry succeeded. chunk_id="5d2a69f9813388ca7ccafcf6eacb3793"
cosmo0920 commented
Thanks for your confirmation. Closing.