Handle user-defined tags in #encode
cosmo0920 opened this issue · 1 comments
cosmo0920 commented
Currently, Fluentd Forward protocol cannot handle Array or Other classes' value in tag.
see: https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1
- Version: 3.1.5
- Operating System: macOS and Linux
- Config File (if you have sensitive info, please remove it):
input {
generator {
lines => [
"line 1",
"line 2",
"line 3"
]
count => 3
tags => ['test', 'logstash']
}
}
output {
tcp {
codec => fluent
host => localhost
port => 24224
}
}
- Sample Data and Steps to Reproduce:
- Install td-agent3
- execute td-agent with the following configuration and command:
fluent-logstash-in.conf:
<source>
@type forward
</source>
<match log>
@type stdout
</match>$ td-agent -c fluent-logstash-in.conf- Expected Result
JSON style output will be appeared in terminal which is running logstash:
2018-01-24 12:21:53.000000000 +0900 test.logstash: {"message":"line 3","@timestamp":"2018-01-24T03:21:53.008Z","host":"<hostname>.local","@version":"1","sequence":2}
2018-01-24 12:21:53.000000000 +0900 test.logstash: {"message":"line 2","@timestamp":"2018-01-24T03:21:53.006Z","host":"<hostname>.local","@version":"1","sequence":0}
2018-01-24 12:21:53.000000000 +0900 test.logstash: {"message":"line 1","@timestamp":"2018-01-24T03:21:53.007Z","host":"<hostname>.local","@version":"1","sequence":1}
2018-01-24 12:21:53.000000000 +0900 test.logstash: {"message":"line 3","@timestamp":"2018-01-24T03:21:53.007Z","host":"<hostname>.local","@version":"1","sequence":1}
2018-01-24 12:21:52.000000000 +0900 test.logstash: {"message":"line 1","@timestamp":"2018-01-24T03:21:52.980Z","host":"<hostname>.local","@version":"1","sequence":0}
2018-01-24 12:21:53.000000000 +0900 test.logstash: {"message":"line 2","@timestamp":"2018-01-24T03:21:53.007Z","host":"<hostname>.local","@version":"1","sequence":1}
2018-01-24 12:21:53.000000000 +0900 test.logstash: {"message":"line 2","@timestamp":"2018-01-24T03:21:53.008Z","host":"<hostname>.local","@version":"1","sequence":2}
2018-01-24 12:21:53.000000000 +0900 test.logstash: {"message":"line 3","@timestamp":"2018-01-24T03:21:53.007Z","host":"<hostname>.local","@version":"1","sequence":0}
2018-01-24 12:21:53.000000000 +0900 test.logstash: {"message":"line 1","@timestamp":"2018-01-24T03:21:53.007Z","host":"<hostname>.local","@version":"1","sequence":2}
- Actual Result
Fluentd complains cannot handle user-defined tag in logstash configuration:
2018-01-24 11:39:50 +0900 [warn]: #0 emit transaction failed: error_class=TypeError error="no implicit conversion of Array into String" location="/Users/cosmo/GitHub/fluentd/lib/fluent/match.rb:128:in `match'" tag=["test", "logstash"]
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/lib/fluent/match.rb:128:in `match'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/lib/fluent/match.rb:128:in `match'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/lib/fluent/event_router.rb:67:in `match?'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/lib/fluent/event_router.rb:244:in `block in find'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/lib/fluent/event_router.rb:243:in `each'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/lib/fluent/event_router.rb:243:in `each_with_index'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/lib/fluent/event_router.rb:243:in `find'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/lib/fluent/event_router.rb:111:in `block in match'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/lib/fluent/event_router.rb:129:in `get'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/lib/fluent/event_router.rb:110:in `match'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/lib/fluent/event_router.rb:96:in `emit_stream'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/lib/fluent/event_router.rb:87:in `emit'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/lib/fluent/plugin/in_forward.rb:337:in `on_message'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/lib/fluent/plugin/in_forward.rb:211:in `block in handle_connection'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/lib/fluent/plugin/in_forward.rb:248:in `block (3 levels) in read_messages'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/lib/fluent/plugin/in_forward.rb:247:in `feed_each'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/lib/fluent/plugin/in_forward.rb:247:in `block (2 levels) in read_messages'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/lib/fluent/plugin/in_forward.rb:256:in `block in read_messages'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/lib/fluent/plugin_helper/server.rb:591:in `on_read_without_connection'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/vendor/bundle/ruby/2.5.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:123:in `on_readable'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/vendor/bundle/ruby/2.5.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:186:in `on_readable'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/vendor/bundle/ruby/2.5.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/vendor/bundle/ruby/2.5.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
2018-01-24 11:39:50 +0900 [warn]: #0 /Users/cosmo/GitHub/fluentd/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2018-01-24 11:39:50 +0900 [error]: #0 unexpected error on reading data host="127.0.0.1" port=56161 error_class=TypeError error="no implicit conversion of Array into String"
2018-01-24 11:39:50 +0900 [error]: #0 suppressed same stacktrace
2018-01-24 11:39:50 +0900 [warn]: #0 emit transaction failed: error_class=TypeError error="no implicit conversion of Array into String" location="/Users/cosmo/GitHub/fluentd/lib/fluent/match.rb:128:in `match'" tag=["test", "logstash"]
- Fixing Candidates Patch
#21.
cosmo0920 commented
I've confirmed that this issue is fixed in 3.2.0.
Closing.