[update-collection-v3] Multiline strings are serialized as single line string
Closed this issue · 0 comments
andrzej-stencel commented
Version: v2.16.0
Multiline strings are migrated as single-line strings.
The output values file still seems to work with Helm, but the multiline properties are impossible to read.
Steps to reproduce
-
Download input values file from https://raw.githubusercontent.com/SumoLogic/sumologic-kubernetes-collection/v2.19.1/examples/multiline_logs/values_containerd.yaml
-
Run the migration tool
./update-collection-v3 -in values_containerd.yaml -out v3.yaml
Expected output
The migrated file v3.yaml should contain multiline entries in e.g. fluent-bit.config.inputs etc.
fluent-bit:
luaScripts:
parse_logs.lua: |
local function split(s, delimiter)
result = {};
for match in (s..delimiter):gmatch('(.-)'..delimiter) do
table.insert(result, match);
end
return result;
end
function get_log_content(line)
-- remove elements specific containerd log format and get log as single string
table.remove(line, 1) -- remove date
table.remove(line, 1) -- remove stream
table.remove(line, 1) -- remove log tag
return table.concat(line, " ")
end
function adjust_first_line(record, first_line)
-- adjust the first line to containerd format, it comes without date, stream and logtag
-- 'fake-date' string at the beginning is used only to have proper log format
first_line = 'fake-date' .. ' ' .. record['stream'] .. ' ' .. record['logtag'] .. ' ' .. first_line
return first_line
end
function parse_log(tag, timestamp, record)
if record['log'] == nil or record['stream'] == nil or record['logtag'] == nil then
-- log does not contain required attributes ('log', 'stream', 'logtag') to be processed by parse_log function
-- the record will not be modified
return 0, timestamp, record
end
log_lines = split(record['log'], '\n')
log_lines[1] = adjust_first_line(record, log_lines[1])
new_lines = {}
buffer = ''
for k, v in pairs(log_lines) do
line = split(v, ' ')
log_tag = line[3]
buffer = buffer .. get_log_content(line)
if log_tag == 'F' then
table.insert(new_lines, buffer)
buffer = ""
end
end
new_log = table.concat(new_lines, "\n")
record['log'] = new_log
return 2, timestamp, record
end
config:
service: |
[SERVICE]
Flush 1
Daemon Off
Log_Level info
Parsers_File parsers.conf
Parsers_File custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
inputs: |
[INPUT]
Name tail
Path /var/log/containers/*.log
Read_from_head true
Multiline On
Parser_Firstline containerd_multiline_pattern
Multiline_Flush 60
Tag containers.*
Refresh_Interval 1
Rotate_Wait 60
Mem_Buf_Limit 30MB
Skip_Long_Lines On
DB /tail-db/tail-containers-state-sumo.db
DB.Sync Normal
[INPUT]
Name systemd
Tag host.*
DB /tail-db/systemd-state-sumo.db
Systemd_Filter _SYSTEMD_UNIT=addon-config.service
Systemd_Filter _SYSTEMD_UNIT=addon-run.service
Systemd_Filter _SYSTEMD_UNIT=cfn-etcd-environment.service
Systemd_Filter _SYSTEMD_UNIT=cfn-signal.service
Systemd_Filter _SYSTEMD_UNIT=clean-ca-certificates.service
Systemd_Filter _SYSTEMD_UNIT=containerd.service
Systemd_Filter _SYSTEMD_UNIT=coreos-metadata.service
Systemd_Filter _SYSTEMD_UNIT=coreos-setup-environment.service
Systemd_Filter _SYSTEMD_UNIT=coreos-tmpfiles.service
Systemd_Filter _SYSTEMD_UNIT=dbus.service
Systemd_Filter _SYSTEMD_UNIT=docker.service
Systemd_Filter _SYSTEMD_UNIT=efs.service
Systemd_Filter _SYSTEMD_UNIT=etcd-member.service
Systemd_Filter _SYSTEMD_UNIT=etcd.service
Systemd_Filter _SYSTEMD_UNIT=etcd2.service
Systemd_Filter _SYSTEMD_UNIT=etcd3.service
Systemd_Filter _SYSTEMD_UNIT=etcdadm-check.service
Systemd_Filter _SYSTEMD_UNIT=etcdadm-reconfigure.service
Systemd_Filter _SYSTEMD_UNIT=etcdadm-save.service
Systemd_Filter _SYSTEMD_UNIT=etcdadm-update-status.service
Systemd_Filter _SYSTEMD_UNIT=flanneld.service
Systemd_Filter _SYSTEMD_UNIT=format-etcd2-volume.service
Systemd_Filter _SYSTEMD_UNIT=kube-node-taint-and-uncordon.service
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Systemd_Filter _SYSTEMD_UNIT=ldconfig.service
Systemd_Filter _SYSTEMD_UNIT=locksmithd.service
Systemd_Filter _SYSTEMD_UNIT=logrotate.service
Systemd_Filter _SYSTEMD_UNIT=lvm2-monitor.service
Systemd_Filter _SYSTEMD_UNIT=mdmon.service
Systemd_Filter _SYSTEMD_UNIT=nfs-idmapd.service
Systemd_Filter _SYSTEMD_UNIT=nfs-mountd.service
Systemd_Filter _SYSTEMD_UNIT=nfs-server.service
Systemd_Filter _SYSTEMD_UNIT=nfs-utils.service
Systemd_Filter _SYSTEMD_UNIT=node-problem-detector.service
Systemd_Filter _SYSTEMD_UNIT=ntp.service
Systemd_Filter _SYSTEMD_UNIT=oem-cloudinit.service
Systemd_Filter _SYSTEMD_UNIT=rkt-gc.service
Systemd_Filter _SYSTEMD_UNIT=rkt-metadata.service
Systemd_Filter _SYSTEMD_UNIT=rpc-idmapd.service
Systemd_Filter _SYSTEMD_UNIT=rpc-mountd.service
Systemd_Filter _SYSTEMD_UNIT=rpc-statd.service
Systemd_Filter _SYSTEMD_UNIT=rpcbind.service
Systemd_Filter _SYSTEMD_UNIT=set-aws-environment.service
Systemd_Filter _SYSTEMD_UNIT=system-cloudinit.service
Systemd_Filter _SYSTEMD_UNIT=systemd-timesyncd.service
Systemd_Filter _SYSTEMD_UNIT=update-ca-certificates.service
Systemd_Filter _SYSTEMD_UNIT=user-cloudinit.service
Systemd_Filter _SYSTEMD_UNIT=var-lib-etcd2.service
Max_Entries 1000
Read_From_Tail true
outputs: |
[OUTPUT]
Name forward
Match *
Host ${FLUENTD_LOGS_SVC}.${NAMESPACE}.svc.cluster.local.
Port 24321
Retry_Limit False
tls off
tls.verify on
tls.debug 1
# Disable keepalive for better load balancing
net.keepalive off
customParsers: |
[PARSER]
Name containerd_multiline_pattern
Format regex
Regex (?<time>^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}[.]\d+Z) (?<stream>stdout|stderr) (?<logtag>[P|F]) (?<log>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.* .*)
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%LZ
filters: |
[FILTER]
Name lua
Match containers.*
script /fluent-bit/scripts/parse_logs.lua
call parse_log
fluentd:
logs:
containers:
multiline:
enabled: falseActual output
The properties in migrated file v3.yaml contain multiline strings rendered as single line with newlines encoded as \n.
fluent-bit:
luaScripts:
parse_logs.lua: "local function split(s, delimiter)\n result = {};\n for match in (s..delimiter):gmatch('(.-)'..delimiter) do\n table.insert(result, match);\n end\n return result;\nend\n\nfunction get_log_content(line)\n -- remove elements specific containerd log format and get log as single string\n table.remove(line, 1) -- remove date\n table.remove(line, 1) -- remove stream\n table.remove(line, 1) -- remove log tag\n return table.concat(line, \" \")\nend\n\nfunction adjust_first_line(record, first_line)\n -- adjust the first line to containerd format, it comes without date, stream and logtag\n -- 'fake-date' string at the beginning is used only to have proper log format\n first_line = 'fake-date' .. ' ' .. record['stream'] .. ' ' .. record['logtag'] .. ' ' .. first_line\n return first_line\nend\n\nfunction parse_log(tag, timestamp, record)\n if record['log'] == nil or record['stream'] == nil or record['logtag'] == nil then\n -- log does not contain required attributes ('log', 'stream', 'logtag') to be processed by parse_log function\n -- the record will not be modified\n return 0, timestamp, record\n end\n\n log_lines = split(record['log'], '\\n')\n log_lines[1] = adjust_first_line(record, log_lines[1])\n\n new_lines = {}\n buffer = ''\n\n for k, v in pairs(log_lines) do\n line = split(v, ' ')\n log_tag = line[3]\n buffer = buffer .. get_log_content(line)\n\n if log_tag == 'F' then\n table.insert(new_lines, buffer)\n buffer = \"\"\n end\n end\n\n new_log = table.concat(new_lines, \"\\n\")\n record['log'] = new_log\n return 2, timestamp, record\nend\n"
config:
service: "[SERVICE]\n Flush 1\n Daemon Off\n Log_Level info\n Parsers_File parsers.conf\n Parsers_File custom_parsers.conf\n HTTP_Server On\n HTTP_Listen 0.0.0.0\n HTTP_Port 2020\n"
inputs: "[INPUT]\n Name tail\n Path /var/log/containers/*.log\n Read_from_head true\n Multiline On\n Parser_Firstline containerd_multiline_pattern\n Multiline_Flush 60\n Tag containers.*\n Refresh_Interval 1\n Rotate_Wait 60\n Mem_Buf_Limit 30MB\n Skip_Long_Lines On\n DB /tail-db/tail-containers-state-sumo.db\n DB.Sync Normal\n[INPUT]\n Name systemd\n Tag host.*\n DB /tail-db/systemd-state-sumo.db\n Systemd_Filter _SYSTEMD_UNIT=addon-config.service\n Systemd_Filter _SYSTEMD_UNIT=addon-run.service\n Systemd_Filter _SYSTEMD_UNIT=cfn-etcd-environment.service\n Systemd_Filter _SYSTEMD_UNIT=cfn-signal.service\n Systemd_Filter _SYSTEMD_UNIT=clean-ca-certificates.service\n Systemd_Filter _SYSTEMD_UNIT=containerd.service\n Systemd_Filter _SYSTEMD_UNIT=coreos-metadata.service\n Systemd_Filter _SYSTEMD_UNIT=coreos-setup-environment.service\n Systemd_Filter _SYSTEMD_UNIT=coreos-tmpfiles.service\n Systemd_Filter _SYSTEMD_UNIT=dbus.service\n Systemd_Filter _SYSTEMD_UNIT=docker.service\n Systemd_Filter _SYSTEMD_UNIT=efs.service\n Systemd_Filter _SYSTEMD_UNIT=etcd-member.service\n Systemd_Filter _SYSTEMD_UNIT=etcd.service\n Systemd_Filter _SYSTEMD_UNIT=etcd2.service\n Systemd_Filter _SYSTEMD_UNIT=etcd3.service\n Systemd_Filter _SYSTEMD_UNIT=etcdadm-check.service\n Systemd_Filter _SYSTEMD_UNIT=etcdadm-reconfigure.service\n Systemd_Filter _SYSTEMD_UNIT=etcdadm-save.service\n Systemd_Filter _SYSTEMD_UNIT=etcdadm-update-status.service\n Systemd_Filter _SYSTEMD_UNIT=flanneld.service\n Systemd_Filter _SYSTEMD_UNIT=format-etcd2-volume.service\n Systemd_Filter _SYSTEMD_UNIT=kube-node-taint-and-uncordon.service\n Systemd_Filter _SYSTEMD_UNIT=kubelet.service\n Systemd_Filter _SYSTEMD_UNIT=ldconfig.service\n Systemd_Filter _SYSTEMD_UNIT=locksmithd.service\n Systemd_Filter _SYSTEMD_UNIT=logrotate.service\n Systemd_Filter _SYSTEMD_UNIT=lvm2-monitor.service\n Systemd_Filter _SYSTEMD_UNIT=mdmon.service\n Systemd_Filter _SYSTEMD_UNIT=nfs-idmapd.service\n Systemd_Filter _SYSTEMD_UNIT=nfs-mountd.service\n Systemd_Filter _SYSTEMD_UNIT=nfs-server.service\n Systemd_Filter _SYSTEMD_UNIT=nfs-utils.service\n Systemd_Filter _SYSTEMD_UNIT=node-problem-detector.service\n Systemd_Filter _SYSTEMD_UNIT=ntp.service\n Systemd_Filter _SYSTEMD_UNIT=oem-cloudinit.service\n Systemd_Filter _SYSTEMD_UNIT=rkt-gc.service\n Systemd_Filter _SYSTEMD_UNIT=rkt-metadata.service\n Systemd_Filter _SYSTEMD_UNIT=rpc-idmapd.service\n Systemd_Filter _SYSTEMD_UNIT=rpc-mountd.service\n Systemd_Filter _SYSTEMD_UNIT=rpc-statd.service\n Systemd_Filter _SYSTEMD_UNIT=rpcbind.service\n Systemd_Filter _SYSTEMD_UNIT=set-aws-environment.service\n Systemd_Filter _SYSTEMD_UNIT=system-cloudinit.service\n Systemd_Filter _SYSTEMD_UNIT=systemd-timesyncd.service\n Systemd_Filter _SYSTEMD_UNIT=update-ca-certificates.service\n Systemd_Filter _SYSTEMD_UNIT=user-cloudinit.service\n Systemd_Filter _SYSTEMD_UNIT=var-lib-etcd2.service\n Max_Entries 1000\n Read_From_Tail true\n"
outputs: "[OUTPUT]\n Name forward\n Match *\n Host ${FLUENTD_LOGS_SVC}.${NAMESPACE}.svc.cluster.local.\n Port 24321\n Retry_Limit False\n tls off\n tls.verify on\n tls.debug 1\n # Disable keepalive for better load balancing\n net.keepalive off\n"
customParsers: "[PARSER]\n Name containerd_multiline_pattern\n Format regex\n Regex (?<time>^\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}[.]\\d+Z) (?<stream>stdout|stderr) (?<logtag>[P|F]) (?<log>\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}.* .*)\n Time_Key time\n Time_Format %Y-%m-%dT%H:%M:%S.%LZ\n"
filters: "[FILTER]\n Name lua\n Match containers.*\n script /fluent-bit/scripts/parse_logs.lua\n call parse_log\n"
fluentd:
logs:
containers:
multiline:
enabled: false