variables work at the token level instead of line
triorbonk opened this issue · 1 comments
I am working on parsing a syslog and trying to enter a variable for the date and time. The date format is "Mmm dd hh:mm:ss" the spaces in the date are split out as tokens then the tokens are replaced. This causes a problem when trying to replace the day "dd" as the pattern is to simple and replaces data I don't want to.
Can the variable functionality be applied before the line is tokenized?
command using
cat system.log | logmine -p'*' -v time:"/\d{2}:\d{2}:\d{2}/" month:"/Mar/" day:"/[1-3]?[0-9]/" -c
Log example
Mar 26 18:07:28 --- last message repeated 1 time ---
Mar 26 18:07:28 xxx Google Chrome Helper[56961]: Libnotify: notify_register_coalesced_registration failed with code 9 on line 2835
Mar 26 18:08:06 xxx com.apple.xpc.launchd[1] (com.apple.mdworker.shared.0D000000-0700-0000-0000-000000000000[15944]): Service exited due to SIGKILL | sent by mds[92]
Mar 26 18:08:07 xxx com.apple.xpc.launchd[1] (com.apple.mdworker.shared.0B000000-0700-0000-0000-000000000000[15957]): Service exited due to SIGKILL | sent by mds[92]
Mar 26 18:08:16 xxx Google Chrome Helper[15966]: Libnotify: notify_register_coalesced_registration failed with code 9 on line 2835
Mar 26 18:08:28 --- last message repeated 16 times ---
Mar 26 18:08:28 xxx Google Chrome Helper[56961]: Libnotify: notify_register_coalesced_registration failed with code 9 on line 2835
Mar 26 18:08:45 xxx com.apple.xpc.launchd[1] (com.apple.mdworker.shared.08000000-0400-0000-0000-000000000000[15955]): Service exited due to SIGKILL | sent by mds[92]
Mar 26 18:08:47 xxx com.apple.xpc.launchd[1] (com.apple.mdworker.shared.06000000-0000-0000-0000-000000000000[15951]): Service exited due to SIGKILL | sent by mds[92]
Mar 26 18:08:50 xxx com.apple.xpc.launchd[1] (com.apple.mdworker.shared.0C000000-0200-0000-0000-000000000000[15964]): Service exited due to SIGKILL | sent by mds[92]
Mar 26 18:08:55 xxx com.apple.xpc.launchd[1] (com.apple.mdworker.shared.0A000000-0000-0000-0000-000000000000[15960]): Service exited due to SIGKILL | sent by mds[92]
Mar 26 18:09:16 xxx com.apple.xpc.launchd[1] (com.apple.mdworker.shared.0F000000-0200-0000-0000-000000000000[15962]): Service exited due to SIGKILL | sent by mds[92]
Mar 26 18:09:25 xxx com.apple.xpc.launchd[1] (com.apple.mdworker.shared.0D000000-0000-0000-0000-000000000000[15963]): Service exited due to SIGKILL | sent by mds[92]
Mar 26 18:09:28 xxx Google Chrome Helper[56961]: Libnotify: notify_register_coalesced_registration failed with code 9 on line 2835
Mar 26 18:09:40 xxx Google Chrome Helper[1567]: Libnotify: notify_register_coalesced_registration failed with code 9 on line 2835
Mar 26 18:09:53 xxx com.apple.xpc.launchd[1] (com.apple.mdworker.shared.0C000000-0300-0000-0000-000000000000[15970]): Service exited due to SIGKILL | sent by mds[92]
Mar 26 18:09:54 xxx com.apple.xpc.launchd[1] (com.apple.mdworker.shared.0F000000-0300-0000-0000-000000000000[15971]): Service exited due to SIGKILL | sent by mds[92]
Mar 26 18:10:14 xxx com.apple.xpc.launchd[1] (com.apple.mdworker.shared.07000000-0300-0000-0000-000000000000[15954]): Service exited due to SIGKILL | sent by mds[92]
Mar 26 18:10:24 xxx Google Chrome Helper[1567]: Libnotify: notify_register_coalesced_registration failed with code 9 on line 2835
Mar 26 18:10:28 xxx Google Chrome Helper[56961]: Libnotify: notify_register_coalesced_registration failed with code 9 on line 2835
Mar 26 18:11:01 xxx Google Chrome Helper[1567]: Libnotify: notify_register_coalesced_registration failed with code 9 on line 2835
Mar 26 18:11:28 xxx Google Chrome Helper[56961]: Libnotify: notify_register_coalesced_registration failed with code 9 on line 2835
Mar 26 18:11:28 xxx syslogd[63]: ASL Sender Statistics
Mar 26 18:11:34 xxx Google Chrome Helper[1567]: Libnotify: notify_register_coalesced_registration failed with code 9 on line 2835
Mar 26 18:12:06 xxx com.apple.xpc.launchd[1] (com.apple.mdworker.shared.0F000000-0400-0000-0000-000000000000[15979]): Service exited due to SIGKILL | sent by mds[92]
Mar 26 18:12:13 xxx com.apple.xpc.launchd[1] (com.apple.mdworker.shared.0A000000-0100-0000-0000-000000000000[15975]): Service exited due to SIGKILL | sent by mds[92]
Mar 26 18:12:13 xxx com.apple.xpc.launchd[1] (com.apple.mdworker.shared.07000000-0400-0000-0000-000000000000[15977]): Service exited due to SIGKILL | sent by mds[92]
Mar 26 18:12:28 xxx Google Chrome Helper[56961]: Libnotify: notify_register_coalesced_registration failed with code 9 on line 2835
Mar 26 18:12:50 xxx Google Chrome Helper[1567]: Libnotify: notify_register_coalesced_registration failed with code 9 on line 2835
Mar 26 18:13:24 xxx com.apple.xpc.launchd[1] (com.apple.mdworker.shared.08000000-0500-0000-0000-000000000000[15982]): Service exited due to SIGKILL | sent by mds[92]
Mar 26 18:13:28 xxx Google Chrome Helper[56961]: Libnotify: notify_register_coalesced_registration failed with code 9 on line 2835
Mar 26 18:14:03 xxx Google Chrome Helper[1567]: Libnotify: notify_register_coalesced_registration failed with code 9 on line 2835
Mar 26 18:14:09 --- last message repeated 1 time ---
Mar 26 18:14:09 xxx com.apple.xpc.launchd[1] (com.apple.mdworker.shared.0C000000-0400-0000-0000-000000000000[15987]): Service exited due to SIGKILL | sent by mds[92]
Mar 26 18:14:09 xxx com.apple.xpc.launchd[1] (com.apple.mdworker.shared.0F000000-0500-0000-0000-000000000000[15986]): Service exited due to SIGKILL | sent by mds[92]
Mar 26 18:14:28 xxx Google Chrome Helper[56961]: Libnotify: notify_register_coalesced_registration failed with code 9 on line 2835
Mar 26 18:14:29 xxx com.apple.xpc.launchd[1] (com.apple.mdworker.shared.0D000000-0100-0000-0000-000000000000[15973]): Service exited due to SIGKILL | sent by mds[92]
Hi there! This is not possible with the algorithm. However, you can preprocess your log before piping it to logmine. For example:
cat system.log | sed -En "s/Mar [[:digit:]]{2} [[:digit:]]{2}:[[:digit:]]{2}:[[:digit:]]{2}/<date-time>/p" | logmine
I got this output:
18 <date-time> xxx com.apple.xpc.launchd[1] (com.apple.mdworker.shared.0D000000-0700-0000-0000-000000000000[15944]): Service exited due to SIGKILL | sent by mds[92]
15 <date-time> xxx Google Chrome Helper[56961]: Libnotify: notify_register_coalesced_registration failed with code 9 on line 2835
3 <date-time> --- last message repeated 1 time ---