Linux fluend mogon "\xE7" to UTF-8 in conversion from ASCII-8BIT,Mongodb cannot be written in Chinese
qaz63700 opened this issue · 0 comments
qaz63700 commented
Describe the bug
Linux fluend mogon "\xE7" to UTF-8 in conversion from ASCII-8BIT,Mongodb cannot be written in Chinese
To Reproduce
Mongodb cannot be written in Chinese
Expected behavior
Chinese string successfully written to mongodb
Your Environment
- Fluentd version:1.4.6
- TD Agent version:center07
- Operating system:
- Kernel version:Your Configuration
<source>
@type tail
path /home/admin/logs/manage.json.log.%Y-%m-%d
<parse>
@type json
</parse>
tag mongo.json.access
</source>
<filter >
@type parser
key_name message
reserve_data true
remove_key_name_field true
<parse>
@type json
</parse>
</filter>
# <match >
# @type stdout
# </match>
<match mongo.**>
# plugin type
@type mongo
# mongodb db + collection
database apache
collection points
# mongodb host + port
host localhost
port 27017
# interval
<buffer>
flush_interval 30s
</buffer>
# # make sure to include the time key
<inject>
time_key time
</inject>
</match>Your Error Log
2022-05-19 14:06:39 +0800 [warn]: #0 got unrecoverable error in primary and no secondary error_class=Encoding::UndefinedConversionError error="\"\\xE7\" from ASCII-8BIT to UTF-8"
2022-05-19 14:06:39 +0800 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data 'got unrecoverable error in primary and no secondary error_class=Encoding::UndefinedConversionError error=\"\\\"\\\\xE7\\\" from ASCII-8BIT to UTF-8\"'" location=nil tag="fluent.warn" time=2022-05-19 14:06:39.439858832 +0800 record={"error"=>"#<Encoding::UndefinedConversionError: \"\\xE7\" from ASCII-8BIT to UTF-8>", "message"=>"got unrecoverable error in primary and no secondary error_class=Encoding::UndefinedConversionError error=\"\\\"\\\\xE7\\\" from ASCII-8BIT to UTF-8\""}
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/bson-4.15.0/lib/bson/string.rb:50:in `encode'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/bson-4.15.0/lib/bson/string.rb:50:in `put_string'
2022-05-19 14:06:39 +0800 [warn]: #0 no patterns matched tag="fluent.warn"
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/bson-4.15.0/lib/bson/string.rb:50:in `to_bson'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/bson-4.15.0/lib/bson/hash.rb:43:in `put_hash'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/bson-4.15.0/lib/bson/hash.rb:43:in `to_bson'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/serializers.rb:326:in `serialize'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/serializers.rb:287:in `block in serialize'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/serializers.rb:286:in `each'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/serializers.rb:286:in `serialize'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/serializers.rb:175:in `block in serialize'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/serializers.rb:168:in `each'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/serializers.rb:168:in `serialize'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/message.rb:241:in `block in serialize_fields'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/message.rb:229:in `each'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/message.rb:229:in `serialize_fields'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/message.rb:125:in `serialize'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/msg.rb:103:in `serialize'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/server/connection.rb:275:in `block in write'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/server/connection.rb:274:in `each'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/server/connection.rb:274:in `write'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/server/connection.rb:229:in `deliver'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/server/connection.rb:144:in `block in dispatch'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/monitoring/publishable.rb:48:in `publish_command'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/server/connection.rb:143:in `dispatch'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/operation/shared/executable.rb:34:in `block in dispatch_message'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/server/connection_pool.rb:110:in `with_connection'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/server.rb:251:in `with_connection'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/operation/shared/executable.rb:33:in `dispatch_message'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/operation/insert/op_msg.rb:42:in `execute'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/operation/shared/write.rb:59:in `bulk_execute'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/bulk_write.rb:210:in `insert_one'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/bulk_write.rb:178:in `execute_operation'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/bulk_write.rb:62:in `block (3 levels) in execute'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/retryable.rb:158:in `legacy_write_with_retry'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/retryable.rb:106:in `write_with_retry'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/bulk_write.rb:61:in `block (2 levels) in execute'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/bulk_write.rb:59:in `each'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/bulk_write.rb:59:in `block in execute'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/cluster.rb:532:in `with_session'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/client.rb:550:in `with_session'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/bulk_write.rb:58:in `execute'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/collection.rb:532:in `bulk_write'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/collection.rb:509:in `insert_many'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/fluent-plugin-mongo-1.5.0/lib/fluent/plugin/out_mongo.rb:334:in `operate'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/fluent-plugin-mongo-1.5.0/lib/fluent/plugin/out_mongo.rb:197:in `write'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start'
2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2022-05-19 14:06:39 +0800 [warn]: #0 bad chunk is moved to /tmp/fluent/backup/worker0/object_14a7738/5df572b61d8caab81b9cad3613c6bb26.log
2022-05-19 14:06:39 +0800 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data 'bad chunk is moved to /tmp/fluent/backup/worker0/object_14a7738/5df572b61d8caab81b9cad3613c6bb26.log'" location=nil tag="fluent.warn" time=2022-05-19 14:06:39.440568521 +0800 record={"message"=>"bad chunk is moved to /tmp/fluent/backup/worker0/object_14a7738/5df572b61d8caab81b9cad3613c6bb26.log"}Additional context
No response