encrypt / suppress sensitive data in the log
samar-elsayed opened this issue · 2 comments
samar-elsayed commented
Describe the bug
I am using KAFKA_OPTS as a var for rdkafka options and it is working fine but the issue is I need to either :
- suppress the config from the log while logging other INFO
- make the var appear as it is in the log
To Reproduce
create var KAFKA_OPTS = "sasl.mechanism: SCRAM-SHA-512,security.protocol: sasl_ssl,sasl.username: user,sasl.password: pass"
and then use it in the config as shown in Your Configuration section
Expected behavior
the config shouldn't appear in the log or at least the var value will be shown as encrypted ****
Your Environment
- Fluentd version: 1.14.0
- fluent-plugin-kafka version: 0.17.5
- ruby-kafka version: 1.5.0
- rdkafka: 0.12.0
- Operating system: Alpine Linux v3.13
- Kernel version: 5.15.58-flatcarYour Configuration
<source>
@type tail
path /logs/quarkus.log
tag file.all
<parse>
@type regexp
expression /^(?<datetime>[0-9- :,]+) (?<host>[0-9a-zA-Z\-\.\+]+) (?<processname>.+?) (?<loglevel>.+) +\[(?<logger>[a-zA-Z-.]+?)\] \((?<thread>.+?)\) (?<logmessage>.+)$/
</parse>
</source>
<match file.all>
@type rewrite_tag_filter
<rule>
key logmessage
pattern /ULFFRecord\:\ (?<ulffrecord>.+)$/
tag file.ulff
</rule>
<rule>
key logmessage
pattern /./
tag file.generic
</rule>
</match>
<filter file.ulff>
@type parser
key_name logmessage
<parse>
@type regexp
expression /^ULFFRecord\:\ (?<ulffrecord>.+)$/
</parse>
</filter>
<filter file.ulff>
@type parser
format json
key_name ulffrecord
</filter>
<match file.ulff>
@type rdkafka2
brokers "<broker>"
get_kafka_client_log true
default_topic ulff
flush_interval 3s
use_event_time true
**rdkafka_options "#{ENV["KAFKA_OPTS"]}"**
<buffer>
flush_mode interval
flush_interval 2s
</buffer>
<format>
@type "json"
</format>
</match>
<match file.generic>
@type rdkafka2
enable_ruby
brokers "<broker>"
get_kafka_client_log true
default_topic custom
use_event_time true
rdkafka_options "#{ENV["KAFKA_OPTS"]}"
<buffer>
flush_mode interval
flush_interval 2s
</buffer>
<format>
@type "json"
</format>
</match>
Your Error Log
the var value is logged along with other infoAdditional context
No response
ashie commented
It's already fixed in master branch (by #452) but not released yet.
You can use password parameter instead of placing it in rdkafka_options.
It will be masked in the log.
I'll release it in the near future to ship it in the next major version of td-agent.