send log to clickhouse db
zamanimehdi opened this issue · 2 comments
zamanimehdi commented
hello,
I want to send event to clickhouse, That's why I install fluentd and fluent-plugin-sql.
then i create table in clickhouse :
CREATE TABLE zamanitable1
(
dt DateTime,
ComputerName String,
RecordNumber Int32,
EventType Int32
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(dt)
ORDER BY dt
and install this gem:
gem install clickhouse-activerecord
my fluentd config :
<match **>
@type sql
host *.*.*.*
port 1317
database system
adapter clickhouse
username default
password ******
socket 1317
<table>
table zamanitable1
column_mapping 'TimeGenerated:dt,ComputerName:ComputerName,RecordNumber:RecordNumber,EventType:EventType'
</table>
</match>
i get this error on fluentd cli:
2020-03-27 07:51:16 +0000 [warn]: #0 fluent/log.rb:342:warn: got unrecoverable error in primary and no secondary error_class=NoMethodError error="undefined method `to_sym' for false:FalseClass"
2020-03-27 07:51:16 +0000 [warn]: #0 plugin/output.rb:452:block (2 levels) in start: /usr/lib/ruby/gems/2.5.0/gems/activerecord-import-0.28.2/lib/activerecord-import/import.rb:664:in `map'
2020-03-27 07:51:16 +0000 [warn]: #0 plugin/output.rb:452:block (2 levels) in start: /usr/lib/ruby/gems/2.5.0/gems/activerecord-import-0.28.2/lib/activerecord-import/import.rb:664:in `import_helper'
2020-03-27 07:51:16 +0000 [warn]: #0 plugin/output.rb:452:block (2 levels) in start: /usr/lib/ruby/gems/2.5.0/gems/activerecord-import-0.28.2/lib/activerecord-import/import.rb:526:in `bulk_import'
2020-03-27 07:51:16 +0000 [warn]: #0 plugin/output.rb:452:block (2 levels) in start: /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-sql-1.1.1/lib/fluent/plugin/out_sql.rb:98:in `import'
2020-03-27 07:51:16 +0000 [warn]: #0 plugin/output.rb:452:block (2 levels) in start: /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-sql-1.1.1/lib/fluent/plugin/out_sql.rb:238:in `block in write'
2020-03-27 07:51:16 +0000 [warn]: #0 plugin/output.rb:452:block (2 levels) in start: /usr/lib/ruby/gems/2.5.0/gems/activerecord-5.2.4.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
2020-03-27 07:51:16 +0000 [warn]: #0 plugin/output.rb:452:block (2 levels) in start: /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-sql-1.1.1/lib/fluent/plugin/out_sql.rb:231:in `write'
2020-03-27 07:51:16 +0000 [warn]: #0 plugin/output.rb:452:block (2 levels) in start: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/plugin/output.rb:1123:in `try_flush'
2020-03-27 07:51:16 +0000 [warn]: #0 plugin/output.rb:452:block (2 levels) in start: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/plugin/output.rb:1423:in `flush_thread_run'
2020-03-27 07:51:16 +0000 [warn]: #0 plugin/output.rb:452:block (2 levels) in start: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/plugin/output.rb:452:in `block (2 levels) in start'
2020-03-27 07:51:16 +0000 [warn]: #0 plugin/output.rb:452:block (2 levels) in start: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-03-27 07:51:16 +0000 [warn]: #0 fluent/log.rb:342:warn: bad chunk is moved to /tmp/fluent/backup/worker0/object_2ae0428ba298/5a1d15a2bc09ab3dcc3ca663644cb763.log
fluentd has successfully connected to clickhouse (Error message when table not exists) but cannot write to db, where is the problem from?
zamanimehdi commented
I found a line of code that caused the error
file: /usr/lib/ruby/gems/2.5.0/gems/activerecord-import-0.28.2/lib/activerecord-import/import.rb
...
# Force the primary key col into the insert if it's not on the list and we are using a sequence and stuff a nil value for it into each row so the sequencer will fire later
symbolized_column_names = Array(column_names).map(&:to_sym)
symbolized_primary_key = _Array(primary_key).map(&:to_sym) # this line
...
zamanimehdi commented
I solved the problem by change the out_sql code and add primery key in it.
my fluentd config change to :
<match **>
@type sql
host *.*.*.*
port 1317
database system
adapter clickhouse
username default
password ******
socket 1317
<table>
table zamanitable1
primary_key dt
column_mapping 'TimeGenerated:dt,ComputerName:ComputerName,RecordNumber:RecordNumber,EventType:EventType'
</table>
</match>