y2log-util: Utilities for YaST Logs

Author: Stefan Hundhammer shundhammer@suse.com

License: GPL V2

This contains some scripts for handling and analyzing y2logs as generated by YaST and the save_y2logs script.

They are provided in the hope that they may be useful, but there is absolutely no warranty and no support. Use at your own risk.

Merging and Splitting

y2log-merge

This merges all the parts of y2logs into one single file, so you no longer have to deal with y2log, y2log-1.gz, y2log-2.gz etc. separately.

It generates one single file y2log-full.log from them.

Usage:

cd YaST2
y2log-merge

You will get a new file y2log-full.log.

y2log-split

This splits one y2log (by default y2log-full.log) into several parts, one for each y2base call (i.e. YaST start). The resulting files include a numeric part for the chronological order of invocation and the name of the YaST module called.

Usage:

cd YaST2
y2log-merge
y2log-split

cd YaST2
y2log-split y2log

You will get files like

y2-00-installation.log
y2-01-menu.log
y2-02-sw_single.log
y2-03-repositories.log
y2-04-repositories.log
y2-05-repositories.log
y2-06-repositories.log
y2-07-dasd.log
y2-08-dasd.log
y2-09-dasd.log

y2log-headers

This shows the headers (the full y2base call) from split logs (that were created by y2log-split):

% y2log-headers

y2-00-installation.log:  2021-02-26 22:29:37  y2base called with ["installation", "--arg", "initial", "ncurses"]
y2-01-menu.log:  2021-02-26 17:42:37  y2base called with ["menu", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2-02-sw_single.log:  2021-02-26 17:42:40  y2base called with ["sw_single", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2-03-repositories.log:  2021-02-26 17:42:52  y2base called with ["repositories", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2-04-repositories.log:  2021-02-26 17:48:14  y2base called with ["repositories", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2-05-repositories.log:  2021-02-26 17:51:45  y2base called with ["repositories", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2-06-repositories.log:  2021-02-26 17:58:07  y2base called with ["repositories", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2-07-dasd.log:  2021-02-26 18:06:22  y2base called with ["dasd", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2-08-dasd.log:  2021-02-27 00:07:52  y2base called with ["dasd", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2-09-dasd.log:  2021-02-27 00:11:57  y2base called with ["dasd", "ncurses", "-name", "YaST2", "-icon", "yast"]

y2log-y2base-calls

This shows y2base calls from the raw (gzipped and not split) y2log and y2log-[1-9].gz files:

% y2log-y2base-calls

y2log-1.gz:  2021-02-26 22:29:37  y2base called with ["installation", "--arg", "initial", "ncurses"]
y2log:       2021-02-26 17:42:37  y2base called with ["menu", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2log:       2021-02-26 17:42:40  y2base called with ["sw_single", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2log:       2021-02-26 17:42:52  y2base called with ["repositories", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2log:       2021-02-26 17:48:14  y2base called with ["repositories", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2log:       2021-02-26 17:51:45  y2base called with ["repositories", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2log:       2021-02-26 17:58:07  y2base called with ["repositories", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2log:       2021-02-26 18:06:22  y2base called with ["dasd", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2log:       2021-02-27 00:07:52  y2base called with ["dasd", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2log:       2021-02-27 00:11:57  y2base called with ["dasd", "ncurses", "-name", "YaST2", "-icon", "yast"]

y2log-rm-split-logs

This removes the files created by y2log-split, i.e. all files that match y2-[0-9][0-9]*.log.

Caution: This does not ask for confirmation, it just does it.

Cleaning and Stripping

The scripts in this section are intended for simplifying y2logs.

y2log-clean

This is a wrapper script to strip off common cruft in y2logs, to simplify the line header, and to downgrade misleading Cheetah error log lines from <3> to <1>.

All operations are done in-place, so make sure to copy the original file if you want to keep the untouched version.

This script can be used commonly for y2logs with or without debug logging.

Usage:

y2log-clean y2-??-*.log

y2log-strip-irregular

This strips off irregular lines from one or more y2log files in-place. It leaves only those that match the standard log line pattern:

2024-02-30 14:07:42 <0> myhost(4711) [Ruby]

This is a remedy against all that cruft from random places that fills up the y2logs like this:

// global string String::Quote (string)
// global string String::UnQuote (string)

-------------------------------- accept ----------------------------

// filename: "/usr/share/YaST2/data/languages/language_af_ZA.ycp"
textdomain "languages_db";
return $["af_ZA":["Afrikaans", "Afrikaans", ".UTF-8", "", _("Afrikaans")], "keyboard":"english-us"];

It is hopeless that those who thought it would be a good idea to just dump that stuff into the y2log will ever clean up their mess; that hasn't happened over the last 15-20 years, so it won't happen in the next 15-20 years.

This y2log-strip-irregular script gets rid of it.

This is included in the more general y2log-clean script.

y2log-strip-date-hostname

This strips off the date part of the log line headers, the hostname and the PID, so it considerably shortens the line header:

From

2024-02-30 14:07:42 <1> myhost(4711) [Ruby] foobar.rb:42: Hello world!
2024-02-30 14:07:42 <1> myhost(4711) [Ruby] foobar.rb:43: ...
2024-02-30 14:07:42 <1> myhost(4711) [Ruby] foobar.rb:44: ...

14:07:42 <1> [Ruby] foobar.rb:42: Hello world!
14:07:42 <1> [Ruby] foobar.rb:43: ...
14:07:42 <1> [Ruby] foobar.rb:44: ...

This is included in the more general y2log-clean script.

The rationale is that those parts are always the same anyway for each log line in all normal cases, so we can save some screen space; and, more importantly, give Bugzilla less opportunities to butcher any pasted content, like it loves to do.

y2log-strip-date

This does only the date stripping part of y2log-strip-date-hostname:

From

2024-02-30 14:07:42 <1> myhost(4711) [Ruby] foobar.rb:42: Hello world!
2024-02-30 14:07:42 <1> myhost(4711) [Ruby] foobar.rb:43: ...
2024-02-30 14:07:42 <1> myhost(4711) [Ruby] foobar.rb:44: ...

14:07:42 <1> myhost(4711) [Ruby] foobar.rb:42: Hello world!
14:07:42 <1> myhost(4711) [Ruby] foobar.rb:43: ...
14:07:42 <1> myhost(4711) [Ruby] foobar.rb:44: ...

This might be useful if the hostname actually changes, or if multiple YaST processes are logging into the same file at the same time.

y2log-strip-hostname

This does only the hostname/PID stripping part of y2log-strip-date-hostname:

From

2024-02-30 14:07:42 <1> myhost(4711) [Ruby] foobar.rb:42: Hello world!
2024-02-30 14:07:42 <1> myhost(4711) [Ruby] foobar.rb:43: ...
2024-02-30 14:07:42 <1> myhost(4711) [Ruby] foobar.rb:44: ...

2024-02-30 14:07:42 <1> [Ruby] foobar.rb:42: Hello world!
2024-02-30 14:07:42 <1> [Ruby] foobar.rb:43: ...
2024-02-30 14:07:42 <1> [Ruby] foobar.rb:44: ...

This might be useful if the date actually changes in the same y2log, or if multiple y2log files stretching over multiple dates are involved.

y2log-strip-cruft

This strips common completely useless messages that only obscur important ones:

[Pkg] modules/PackageCallbacks.rb:9999 Pkg Builtin called: Callback...
[libstorage] SystemCmd.cc(doExecute):999 stopwatch ...
[libstorage] SystemCmd.cc(execute):999 timestamp ...
[libstorage] SystemCmd.cc(step_fork_and_exec):999 child_pid:9999
[libstorage] SystemCmd.cc(SystemCmd):999 constructor SystemCmd(...)
[libstorage] SystemCmd.cc(add_line):713 line stdout[99] 'loop99'
[agent-ini] IniParser.cc(getTimeStamp):999 Unable to stat '/etc/sysctl.conf'
[agent-ini] IniParser.cc(getTimeStamp):999 Unable to stat '/etc/install.inf'

This also shortens the boilerplate stdout output for libstorage and cheetah shell commands:

2024-04-16 17:36:56 <1> myhost(9999) [libstorage] SystemCmd.cc(add_line):713 line stdout[0] 'foo'
2024-04-16 17:36:56 <1> myhost(9999) [libstorage] SystemCmd.cc(add_line):713 line stdout[1] 'bar'
2024-04-16 17:36:56 <1> myhost(9999) [libstorage] SystemCmd.cc(add_line):713 line stdout[2] 'baz'

2024-04-16 17:36:56 <1> myhost(9999)  'foo'
2024-04-16 17:36:56 <1> myhost(9999)  'bar'
2024-04-16 17:36:56 <1> myhost(9999)  'baz'

2024-04-16 15:03:48 <1> myhost(9999) [Ruby] lib/cheetah.rb(log_stream_line):211 Standard output: foo
2024-04-16 15:03:48 <1> myhost(9999) [Ruby] lib/cheetah.rb(log_stream_line):211 Standard output: bar
2024-04-16 15:03:48 <1> myhost(9999) [Ruby] lib/cheetah.rb(log_stream_line):211 Standard output: baz

2024-04-16 15:03:48 <1> myhost(9999)  foo
2024-04-16 15:03:48 <1> myhost(9999)  bar
2024-04-16 15:03:48 <1> myhost(9999)  baz

The context is still clear from the line immediately preceding the block of output which contains the executed command.

y2log-clean-cheetah

This cleans up misleading error log lines written by Cheetah which just logs all regular stdout of commands that it invokes as errors. Because reasons. And despite the bug report about this that was filed years ago.

This makes searching for real errors a royal PITA because a simple dracut run will easily flood the y2log with some hundred error log lines, drowning out all real errors.

This script simply downgrades all <3> (error) log lines to <1> (milestone) if cheetah is found in the same line. This operation is in-place in the file.

Usage:

y2log-clean-cheetah y2log

This is included in the more general y2log-clean script.

y2log-strip-debug

This strips off debug lines from one or more y2log files in-place; it removes lines with log level <0> like these:

2024-02-30 14:07:42 <0> myhost(4711) [Ruby]
14:07:42 <0> myhost(4711) [Ruby]
14:07:42 <0> [Ruby]
14:07:42 <0> myhost(4711) [Ruby]

Sometimes it's useful to have debug logging, but mostly it's just additional noise that drowns out useful information.

Since this operates in-place, it is advised to copy the original file with debug logging away to a safe place before using this script:

cp y2log y2log-debug
y2log-strip-debug y2log

It is advised to do this together with the other scripts here:

y2log-clean y2log
cp y2log y2log-debug
y2log-strip-debug y2log

Now there are two cleaned-up y2logs: y2log without debug logging and y2log-debug with.

Extracting Embedded Information

y2log-get-xml-profile

This extracts an XML AutoYaST profile from an y2log and formats it in a human-readable way, with \n translated to newlines etc. Typically, this needs to be an y2log with debug logging.

Usage:

y2log-get-xml-profile y2log-debug >profile.xml

At the time of this writing, this uses the first XML profile that it finds. This might change in the future.

y2log-grep-xml-profile

This searches one or more y2log files for XML AutoYaST profiles. It cuts off the output after a reasonable amount of characters to avoid flooding the terminal window.

% y2log-grep-xml-profile y2log-debug

02:25:14 ScriptingAgent.cc(executeSubagentCommand):592 opt: "<?xml version=\"1.0\"?>\n<!DOCTYPE
profile>\n<profile xmlns=\"http://www.suse.com/1.0/yast2ns\"
xmlns:config=\"http://www.suse.com/1.0/configns\">\n <bootloader t=\"map\">\n <global t=\"map\">\n
<append>showopts</append>\n <t

02:25:14 ../../libscr/src/include/scr/Y2AgentComponent.h(evaluate):95 evaluate (`Write (.string,
"/tmp/YaST2-06833-60VX5P/base_profile.xml", "<?xml version=\"1.0\"?>\n<!DOCTYPE profile>\n<profile
xmlns=\"http://www.suse.com/1.0/yast2ns\" xmlns:config=\"http://www.suse.com/1.0/configns\">\n
<bootloa

02:25:14 ../../libscr/src/include/scr/Y2AgentComponent.h(evaluate):100 Going to evaluate `Write
(.string, "/tmp/YaST2-06833-60VX5P/base_profile.xml", "<?xml version=\"1.0\"?>\n<!DOCTYPE
profile>\n<profile xmlns=\"http://www.suse.com/1.0/yast2ns\"
xmlns:config=\"http://www.suse.com/1.0/configns\">\n

...
...
...

21:26:15 ../../libscr/src/include/scr/Y2AgentComponent.h(evaluate):100 Going to evaluate `Write
(.string, "/tmp/YaST2-06833-60VX5P/autoinst.xml", "<?xml version=\"1.0\"?>\n<!DOCTYPE
profile>\n<profile xmlns=\"http://www.suse.com/1.0/yast2ns\"
xmlns:config=\"http://www.suse.com/1.0/configns\">\n <bo

21:26:15 ../../libscr/src/include/scr/Y2AgentComponent.h(evaluate):121 After code evaluation: `Write
(.string, "/tmp/YaST2-06833-60VX5P/autoinst.xml", "<?xml version=\"1.0\"?>\n<!DOCTYPE
profile>\n<profile xmlns=\"http://www.suse.com/1.0/yast2ns\"
xmlns:config=\"http://www.suse.com/1.0/configns\">\n


*** Number of profiles: 24

y2log-grep-systemcmd

This extracts the external commands executed with the libstorage SystemCmd class to get a good impression of storage operations:

y2log-grep-systemcmd

By default, this uses y2log-full.log (the result of y2log-merge), and if there is no file with that name, y2log. You can also specify a file on the command line.

Since libstorage executes a lot of udevadm settle commands that are normally not very relevant, it is usually a good idea to filter those out:

% y2log-grep-systemcmd | grep -v 'udevadm settle'

02:25:13  /usr/bin/uname -m
02:25:13  /usr/bin/test -d /sys/firmware/efi/efivars
02:25:13  /usr/bin/getconf PAGESIZE
02:25:13  /bin/ls -1 --sort=none /sys/block
02:25:13  /usr/bin/stat --format %f /dev/sda
02:25:13  /usr/bin/udevadm info /dev/sda
02:25:14  /usr/bin/uname -m
02:25:14  /usr/bin/test -d /sys/firmware/efi/efivars
02:25:14  /usr/bin/getconf PAGESIZE
21:26:15  /usr/bin/uname -m
21:26:15  /usr/bin/test -d /sys/firmware/efi/efivars
21:26:15  /usr/bin/getconf PAGESIZE
21:30:00  /sbin/multipath -d -v 2
21:30:00  /sbin/mdadm --assemble --scan
21:30:00  /sbin/dmraid --activate yes --no_partitions
21:30:00  /sbin/lvs --reportformat json --config log { command_names = 0 prefix = "" } --units b --nosuffix --all --options lv_name,lv_uuid,vg_name,vg_uuid,lv_role,lv_attr,lv_size,origin_size,segtype,stripes,stripe_size,chunk_size,pool_lv,pool_lv_uuid,origin,origin_uuid,data_lv,data_lv_uuid,metadata_lv,metadata_lv_uuid
21:30:01  /sbin/blkid -c /dev/null
21:30:01  /usr/bin/uname -m
21:30:01  /usr/bin/test -d /sys/firmware/efi/efivars
21:30:01  /usr/bin/getconf PAGESIZE
21:30:01  /bin/ls -1 --sort=none /sys/block
21:30:01  /usr/bin/stat --format %f /dev/sda
21:30:01  /usr/bin/udevadm info /dev/sda
21:30:01  /usr/bin/lsscsi --version
21:30:01  /usr/bin/lsscsi --transport
21:30:01  /sbin/multipath -d -v 2 -ll
21:30:01  /sbin/dmraid --sets=active -ccc
21:30:01  /sbin/blkid -c /dev/null
21:30:01  /usr/sbin/parted --version
21:30:01  /usr/sbin/parted --script --machine /dev/sda unit s print

shundhammer/y2log-util

y2log-util: Utilities for YaST Logs

Merging and Splitting

y2log-merge

y2log-split

y2log-headers

y2log-y2base-calls

y2log-rm-split-logs

Cleaning and Stripping

y2log-clean

y2log-strip-irregular

y2log-strip-date-hostname

y2log-strip-date

y2log-strip-hostname

y2log-strip-cruft

y2log-clean-cheetah

y2log-strip-debug

Extracting Embedded Information

y2log-get-xml-profile

y2log-grep-xml-profile

y2log-grep-systemcmd