Support for ${uuid}

[gtrevg]

I really like the flexibility of your plugin as it allows me to modify tags as well as do lots of other helpful processing of the logs. One feature I was looking to be able to add in is a UUID identifier.

I saw that there is a mixin that gives support for the hostname and uuid variables ( https://github.com/tagomoris/fluent-mixin-config-placeholders ) which I see the fluent-plugin-record-modifier that you reference is using. Right now I use that plugin to add in the uuid, then pass it to your plugin to do more processing:

<match uuid.**>
  type record_reformer

  remove_keys l_tag
  output_tag out.${l_tag}

  source ${hostname}
  foo bar
</match>

<match **>
  type record_modifier

  include_tag_key true

  lid ${uuid}
  tag_key l_tag
  tag uuid
</match>

It seems like that's a very clunky solution just to get the uuid into the data so that I can use your plugin to do much more processing.

Would it be hard to add in the uuid feature? Or if you have a better solution on how to do that, that would be great.

Thanks

[gtrevg]

I forked & made an update, but (since I don't know Ruby), I haven't been able to get the ${uuid:hostname} type format to work. I did get the ${uuid_hostname} format to work, though:

https://github.com/gtrevg/fluent-plugin-record-reformer/blob/master/lib/fluent/plugin/out_record_reformer.rb

Ideally it'd be the same format as config-placeholders.

Thanks

[sonots]

@gtrevg Well, let me make clear. In your code, you are generating uuid in #configure, thus, @uuid_r = UUIDTools::UUID.random_create.to_s will be executed only once on starting up of fluentd, which results in the random uuid will be common in each record emit. Is it what you want to do?

[gtrevg]

Ah, no that's not the behavior that I wanted.

I would want each log item to be able generate a new UUID. Ideally, the plugin would also only generate the needed UUID if that variable were passed in (optimization to not do any additional work if it wasn't necessary).

[sonots]

only generate the needed UUID if that variable were passed in

hmm, current code structure is not supporting ways like that. It seems big refactoring is required to support it.

I may do someday when I have free time, but pull request is always welcome!

[gtrevg]

yeah, i think that would be an optimization only if performance weren't good and that would still need to be tested to see if there's actually a performance hit.

I guess initially I could just move the UUID generation to the emit function and then that would work for generating UUIDs.

[gtrevg]

I've updated the code to now support these variables:

<match **>
  type record_reformer

  output_tag out.${tag}

  host ${hostname}
  uuid ${uuid}
  uuid_random ${uuid:random}
  uuid_hostname ${uuid:hostname}
  uuid_timestamp ${uuid:timestamp}

  foo bar
</match>

I'll send you a pull request for the changes. In my testing, I actually found that record_modifier plugin suffers from the "only generate one UUID for all logs" issue that you described before. It looks like I wouldn't even be able to use that plugin as an alternative.

As for performance penalty, we're only generating two UUIDs per record so I don't think at this moment it is a huge concern.

Thanks

[bmpandrade]

above ruby 1.9.3, if you requires securerandom, you can use SecureRandom.uuid to generate uuid inside the plugin.