fluent/fluent-plugin-sql

How to populate timestamp column

bschooly opened this issue · 2 comments

What is the proper/recommended way to populate the timestamp?

If have:

<source>
  @type tail
  path /tmp/example.log
  pos_file /var/log/td-agent/example.log.pos
  tag example.*
 <parse>
        @type json
        keep_time_key true
  </parse>
</source>
...

<match example.**>
  @type sql
  host localhost
  port 3306
  database example
  adapter mysql2
  username example
  password XXXXXX
  socket /var/lib/mysql/mysql.sock
  remove_tag_prefix my.rdb # optional, dual of tag_prefix in in_sql

  <table>
    table logs
    column_mapping 'time,host,size,method'
    # This is the default table because it has no "pattern" argument in <table>
    # The logic is such that if all non-default <table> blocks
    # do not match, the default one is chosen.
    # The default table is required.
  </table>
</match>

The above works but I couldn't get the time value reformatted. How do I do this? Also how do I use the implicit timestamp generated by fluentd? Ie. the timestamp that is not part of the json payload.

Here is a input record:

{"time":"1362020444","host":"localhost","size":777,"method":"PUT"}
This works. I get exacly the data shown above in the db but I want to know if I can reformat the time and how to populate the timestamp if it is not part of the json data.

Thanks in advance for a reply...

Bill

Any luck on the issue? Facing the same thing at the moment

Maybe you can use filter

<filter app.*>
  @type record_transformer
  enable_ruby true
  <record>
    @timestamp ${time.strftime('%Y-%m-%dT%H:%M:%S.%L')}
  </record>
</filter>