Buffered output doesn't work since 0.14

[njam]

Since fluentd 0.14 I can't make buffered outputs in a forest work anymore.

With a simple configuration like this:

<match **>
  @type forest
  subtype file
  <template>
    path /tmp/foo-__TAG__
    buffer_type memory
    flush_interval 1s
  </template>
</match>

It seems buffers aren't flushed anymore.
(all works fine in a non-buffered configuration, and the buffered version works fine without forest)

I'm running the latest fluentd 0.14.6 and fluent-plugin-forest 0.3.1.

[jottos]

I have exact same issue as @njam

What we are noticing is that on the inbound the buffer files are all being created along with their *.meta files BUT none of the buffered files are being drained to their respective output plugins. So far we have only been able to track the problem to the file_chunk.rb which sends errors like this to the td-agent.log file

2016-10-06 21:27:26 +0000 [warn]: fluent/root_agent.rb:277:handle_emits_error: emit transaction failed: error_class=Errno::ENOENT error="No such file or directory @ rb_sysopen - /var
/log/td-agent/kafka/useraccount/buffer.b53e38f5c98d48505b4479a415d2213bb.log" tag="filtered.staging.useraccount"

what we have determined is that the path /var/log/td-agent/kafka/useraccount exists and there are buffered logs in it that look fine, but the b53e38f5c98d48505b4479a415d2213bb part of the file is not correct, this specific buffer file does not exist and if we grep through all the files fluent has created we cannot find any reference to this specific ID.

[tagomoris]

As far as I tested, i reproduced this problem, but no Errono::ENOENT error raised.

<source>
  @type dummy
  tag test.dummy
  dummy {"message":"yaaaaaaaaaaaaay"}
  rate 3
</source>

<match test.*>
  @type forest
  subtype file
  <template>
    path /tmp/file_${tag}
    buffer_type file
    flush_interval 1s
  </template>
</match>

[tagomoris]

I found the root cause. Fluentd v0.14.6 introduced a new lifecycle control via #after_start method.
But fluent-plugin-forest doesn't call it.

[tagomoris]

I just released v0.3.2 to solve this problem. Thank you for reporting!

[jottos]

Thank you, Satoshi. I will test in the morning.
/jos

[njam]

Thank you! works for me! 🚲