Unable to write timestamp to postgres
PACordonnier opened this issue · 1 comments
PACordonnier commented
Hi,
I'm trying to send my record to postgres with a field containing the timestamp.
Here is my schema:
CREATE TABLE simple_table (
id SERIAL PRIMARY KEY,
timestamp timestamptz,
);Here is a config sample
<filter record.**>
@type record_transformer
enable_ruby true
<record>
timestamp ${time}
</record>
</filter>
<filter record.**>
@type stdout
</filter>
<match record.**>
@type sql
host myhost
username myuser
password mypassword
port 5432
database mydb
adapter postgresql
<table>
table simple_table
column_mapping 'timestamp:timestamp'
</table>
flush_interval 1s
# disable_retry_limit
# num_threads 8
# slow_flush_log_threshold 40.0
</match>
No record is sent to SQL with such configuration. This stacktrace can be found in the log:
2019-05-13 13:55:55 +0000 [warn]: #0 emit transaction failed: error_class=NoMethodError error="undefined method `to_msgpack' for 2019-05-13 13:55:55 +0000:Time" location="/var/lib/gems/2.3.0/gems/msgpack-1.2.10/lib/msgpack.rb:44:in `write'" tag="argos.var.log.containers.log-app_argos_log-app-f99e0f0225be7f253b11d6744f1b6f5ced604b679701571dc4319138e56b587f.log"
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/msgpack-1.2.10/lib/msgpack.rb:44:in `write'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/msgpack-1.2.10/lib/msgpack.rb:44:in `pack'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/msgpack-1.2.10/lib/msgpack/core_ext.rb:11:in `to_msgpack'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-sql-1.1.1/lib/fluent/plugin/out_sql.rb:223:in `format'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.4.1/lib/fluent/plugin/output.rb:985:in `block in handle_stream_simple'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.4.1/lib/fluent/event.rb:193:in `block in each'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/activesupport-5.2.3/lib/active_support/core_ext/range/each.rb:9:in `each'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/activesupport-5.2.3/lib/active_support/core_ext/range/each.rb:9:in `each'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.4.1/lib/fluent/event.rb:192:in `each'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.4.1/lib/fluent/plugin/output.rb:984:in `handle_stream_simple'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.4.1/lib/fluent/plugin/output.rb:870:in `execute_chunking'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.4.1/lib/fluent/plugin/output.rb:793:in `emit_buffered'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.4.1/lib/fluent/event_router.rb:159:in `emit_events'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.4.1/lib/fluent/event_router.rb:96:in `emit_stream'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.4.1/lib/fluent/plugin/in_tail.rb:402:in `receive_lines'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.4.1/lib/fluent/plugin/in_tail.rb:520:in `wrap_receive_lines'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.4.1/lib/fluent/plugin/in_tail.rb:747:in `block in handle_notify'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.4.1/lib/fluent/plugin/in_tail.rb:791:in `with_io'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.4.1/lib/fluent/plugin/in_tail.rb:725:in `handle_notify'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.4.1/lib/fluent/plugin/in_tail.rb:721:in `block in on_notify'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.4.1/lib/fluent/plugin/in_tail.rb:721:in `synchronize'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.4.1/lib/fluent/plugin/in_tail.rb:721:in `on_notify'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.4.1/lib/fluent/plugin/in_tail.rb:550:in `on_notify'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.4.1/lib/fluent/plugin/in_tail.rb:636:in `on_timer'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/cool.io-1.5.4/lib/cool.io/loop.rb:88:in `run_once'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/cool.io-1.5.4/lib/cool.io/loop.rb:88:in `run'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.4.1/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
2019-05-13 13:55:55 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.4.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2019-05-13 13:56:05.878816961 +0000 argos.var.log.containers.log-app_argos_log-app-f99e0f0225be7f253b11d6744f1b6f5ced604b679701571dc4319138e56b587f.log: {"data":"data","timestamp":"2019-05-13T13:56:05.878+00:00"}
I'm customising the fluentd image available in the kubernetes repo. Gemfile:
gem 'activesupport', '~>5.2.2'
gem 'fluentd', '<=1.4.1'
gem 'fluent-plugin-concat', '~>2.3.0'
gem 'fluent-plugin-detect-exceptions', '~>0.0.12'
gem 'fluent-plugin-elasticsearch', '~>3.3.3'
gem 'fluent-plugin-kubernetes_metadata_filter', '~>2.1.6'
gem 'fluent-plugin-multi-format-parser', '~>1.0.0'
gem 'fluent-plugin-prometheus', '~>1.3.0'
gem 'fluent-plugin-systemd', '~>1.0.2'
gem 'oj', '~>3.7.9'
gem 'fluent-plugin-sql', '~>1.1.1'
gem 'pg', '0.21.0'
# I have tried to install msgpack and mspack-rails separately but it did not solve the error
gem 'msgpack'
gem 'msgpack-rails', :git => 'git://github.com/nzifnab/msgpack-rails.git'
When writing a text or varchar it's working fine, only when the destination type is a timestamp I can't write it and this error shows up.
I can probably write the timestamp as a 'text' but it's not the same. Is there a dependency I'm missing ?
Thank you for your help
PACordonnier commented
Just like most of the issue I ever created, I found the fix just a few minutes creating it (even though I worked on the problem for a few hours).
Anyway it's kind of the same issue as this one fluent/fluentd#2072
Since the timestamp field is created with the 'enable_ruby' flag it's not a String but a Time where to_msgpack method is not defined (if I understand correctly)
I disabled the enable_ruby flag in my and it works correctly.
For future users, here is the full configuration:
<filter record.**>
@type record_transformer
<record>
timestamp ${time}
</record>
</filter>
<filter record.**>
@type stdout
</filter>
<match record.**>
@type sql
host myhost
username myuser
password mypassword
port 5432
database mydb
adapter postgresql
<table>
table simple_table
column_mapping 'timestamp:timestamp'
</table>
flush_interval 1s
# disable_retry_limit
# num_threads 8
# slow_flush_log_threshold 40.0
</match>
Thank you for your work on fluentd