logclean wants to also clean out the log of the most recent successful install
equaeghe opened this issue · 12 comments
As far as I understood, logclean should leave the log of the most recent successful install of a package. However, if I run logclean -s logs, I see that first all ‘old’ logs are rmed, but then logclean seems to go over the whole tree again and also rms the ‘recent’ logs. For example, for the kitty package, I get in the first part of the output
rm /var/log/portage/build/x11-terms/kitty-0.12.3:20181001-195923.log
rm /var/log/portage/build/x11-terms/kitty-9999:20180616-123918.log
rm /var/log/portage/build/x11-terms/kitty-0.13.1-r1:20181216-215738.log
rm /var/log/portage/build/x11-terms/kitty-0.11.2:20181001-200010.log
rm /var/log/portage/build/x11-terms/kitty-0.10.1:20180608-202438.log
rm /var/log/portage/build/x11-terms/kitty-0.11.2:20180726-082006.log
rm /var/log/portage/build/x11-terms/kitty-0.13.3:20190123-201330.log
rm /var/log/portage/build/x11-terms/kitty-0.10.1:20180609-201339.log
rm /var/log/portage/build/x11-terms/kitty-9999:20180616-124130.log
rm /var/log/portage/build/x11-terms/kitty-0.13.2:20190106-203835.log
rm /var/log/portage/build/x11-terms/kitty-0.13.1:20181212-215158.log
rm /var/log/portage/build/x11-terms/kitty-0.12.3:20181212-215234.log
rm /var/log/portage/build/x11-terms/kitty-0.13.1:20181216-215818.log
rm /var/log/portage/build/x11-terms/kitty-0.13.1-r1:20190106-203919.log
rm /var/log/portage/build/x11-terms/kitty-0.13.2:20190123-201411.log
rm /var/log/portage/build/x11-terms/kitty-0.14.2:20190616-173456.log
rm /var/log/portage/build/x11-terms/kitty-0.13.3:20190616-173542.log
rm /var/log/portage/build/x11-terms/kitty-0.14.4:20190901-111109.log
rm /var/log/portage/build/x11-terms/kitty-0.14.2:20190901-111149.log
rm /var/log/portage/build/x11-terms/kitty-0.14.6:20190929-150027.log
rm /var/log/portage/build/x11-terms/kitty-0.14.4:20190929-150125.log
rm /var/log/portage/build/x11-terms/kitty-0.14.6-r1:20191119-191303.log
rm /var/log/portage/build/x11-terms/kitty-0.14.6:20191119-191351.log
rm /var/log/portage/build/x11-terms/kitty-0.15.0:20191203-204758.log
rm /var/log/portage/build/x11-terms/kitty-0.14.6-r1:20191203-204849.log
rm /var/log/portage/build/x11-terms/kitty-0.15.0:20200109-195618.log
In the second part of the output, I get
rm /var/log/portage/build/x11-terms/kitty-0.15.1:20200109-195521.log
I would expect that the second part of the output is not present. What is going on? How can I make sure that the logs for the most recent successful installation will be kept?
N.B.: For firefox, the first part of the output contains the logs for old package versions and the second part contains the logs for the currently installed version, but there are two logs, as it had been reinstalled. That may give a clue to why there are apparently two ‘runs’ through the tree.
FYI, the directory listing:
# ls -alh build/x11-terms/
total 956K
drwxr-xr-x 1 root root 1.8K 9 jan 20:56 .
drwxr-xr-x 1 root root 1.7K 31 jan 14:26 ..
-rw-rw---- 1 portage portage 47K 8 jun 2018 kitty-0.10.1:20180608-202438.log
-rw-rw---- 1 portage portage 36 9 jun 2018 kitty-0.10.1:20180609-201339.log
-rw-rw---- 1 portage portage 58K 26 jul 2018 kitty-0.11.2:20180726-082006.log
-rw-rw---- 1 portage portage 19K 1 okt 2018 kitty-0.11.2:20181001-200010.log
-rw-rw---- 1 portage portage 53K 1 okt 2018 kitty-0.12.3:20181001-195923.log
-rw-rw---- 1 portage portage 14K 12 dec 2018 kitty-0.12.3:20181212-215234.log
-rw-rw---- 1 portage portage 37K 12 dec 2018 kitty-0.13.1:20181212-215158.log
-rw-rw---- 1 portage portage 14K 16 dec 2018 kitty-0.13.1:20181216-215818.log
-rw-rw---- 1 portage portage 56K 16 dec 2018 kitty-0.13.1-r1:20181216-215738.log
-rw-rw---- 1 portage portage 14K 6 jan 2019 kitty-0.13.1-r1:20190106-203919.log
-rw-rw---- 1 portage portage 55K 6 jan 2019 kitty-0.13.2:20190106-203835.log
-rw-rw---- 1 portage portage 14K 23 jan 2019 kitty-0.13.2:20190123-201411.log
-rw-rw---- 1 portage portage 55K 23 jan 2019 kitty-0.13.3:20190123-201330.log
-rw-rw---- 1 portage portage 14K 16 jun 2019 kitty-0.13.3:20190616-173542.log
-rw-rw---- 1 portage portage 60K 16 jun 2019 kitty-0.14.2:20190616-173456.log
-rw-rw---- 1 portage portage 28K 1 sep 13:11 kitty-0.14.2:20190901-111149.log
-rw-rw---- 1 portage portage 48K 1 sep 13:11 kitty-0.14.4:20190901-111109.log
-rw-rw---- 1 portage portage 28K 29 sep 17:01 kitty-0.14.4:20190929-150125.log
-rw-rw---- 1 portage portage 48K 29 sep 17:01 kitty-0.14.6:20190929-150027.log
-rw-rw---- 1 portage portage 28K 19 nov 20:13 kitty-0.14.6:20191119-191351.log
-rw-rw---- 1 portage portage 48K 19 nov 20:13 kitty-0.14.6-r1:20191119-191303.log
-rw-rw---- 1 portage portage 28K 3 dec 21:48 kitty-0.14.6-r1:20191203-204849.log
-rw-rw---- 1 portage portage 50K 3 dec 21:48 kitty-0.15.0:20191203-204758.log
-rw-rw---- 1 portage portage 35K 9 jan 20:56 kitty-0.15.0:20200109-195618.log
-rw-rw---- 1 portage portage 50K 9 jan 20:56 kitty-0.15.1:20200109-195521.log
-rw-rw---- 1 portage portage 1.6K 16 jun 2018 kitty-9999:20180616-123918.log
-rw-rw---- 1 portage portage 20K 16 jun 2018 kitty-9999:20180616-124130.log
You can add -d (--debug) to see the reasons why something is cleaned.
Indeed, first all logs for non-installed packages and versions are deleted, and then in a second phase at most the latest log of the installed package is kept.
You can add -d (--debug) to see the reasons why something is cleaned.
Using the debug option, I can see that
logclean: debug: no installation log: build/x11-terms/kitty-0.15.1:20200109-195521.log
but I have no clue why logclean would think it is not an installation log. I have no perl experience, so I do not see why is_install_log thinks it is not an installation log.
Can you provide some insight? Feel free to ask me to supply specific extra information (file contents or properties).
This function essentially checks whether the file contains somewhere the textCompleted installing.
Unless something has changed recently, portage prints this message at the end of the install phase: If this phase was not called within the log, the log is probably just an “uninstall” message or a message for merging a tbz2 file. Or it might be a failed compilation (which did not reach the end of the install phase).
This function essentially checks whether the file contains somewhere the text
Completed installing.
Unless something has changed recently, portage prints this message at the end of the install phase: If this phase was not called within the log, the log is probably just an “uninstall” message or a message for merging a tbz2 file. Or it might be a failed compilation (which did not reach the end of the install phase).
This is very strange. The compiles have not failed, but the logs themselves are clearly not in the expected form:
* Package: x11-terms/kitty-0.15.1
* Repository: gentoo
* Maintainer: pabloorduna98@gmail.com proxy-maint@gentoo.org
* USE: abi_x86_64 amd64 elibc_glibc kernel_linux python_single_target_python3_6 python_targets_python3_6 userland_GNU wayland
* FEATURES: network-sandbox preserve-libs sandbox userpriv usersandbox
* Applying kitty-0.15.1-flags.patch ...
[ ok ]
* Applying kitty-0.14.4-svg-icon.patch ...
[ ok ]
CC: x86_64-pc-linux-gnu-gcc (9, 2)
/usr/bin/wayland-scanner client-header //usr/share/wayland-protocols/stable/xdg-shell/xdg-shell.xml glfw/wayland-xdg-shell-client-protocol.h
/usr/bin/wayland-scanner private-code //usr/share/wayland-protocols/stable/xdg-shell/xdg-shell.xml glfw/wayland-xdg-shell-client-protocol.c
[EDITED OUT MANY SIMILAR LINES]
/usr/bin/wayland-scanner private-code //usr/share/wayland-protocols/unstable/primary-selection/primary-selection-unstable-v1.xml glfw/wayland-primary-selection-unstable-v1-client-protocol.c
x86_64-pc-linux-gnu-gcc -MMD -DNDEBUG -DPRIMARY_VERSION=4000 -DSECONDARY_VERSION=15 -Wextra -Wfloat-conversion -Wno-missing-field-initializers -Wall -Wstrict-prototypes -std=c11 -pedantic-errors -fwrapv -fstack-protector-strong -fvisibility=hidden -fPIC -march=native -O2 -pipe -flto -pthread -I/usr/include/libpng16 -I/usr/include/freetype2 -I/usr/include/uuid -I/usr/include/harfbuzz -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/python3.6m -c kitty/unicode-data.c -o build/fast_data_types-unicode-data.c.o
x86_64-pc-linux-gnu-gcc -MMD -DNDEBUG -D_GLFW_X11 -D_GLFW_BUILD_DLL -Wextra -Wfloat-conversion -Wno-missing-field-initializers -Wall -Wstrict-prototypes -std=c11 -pedantic-errors -fwrapv -fstack-protector-strong -fvisibility=hidden -fPIC -march=native -O2 -pipe -flto -fpic -pthread -I/usr/include/dbus-1.0 -I/usr/lib64/dbus-1.0/include -c glfw/x11_window.c -o build/glfw-x11-x11_window.c.o
[EDITED OUT MANY SIMILAR LINES]
x86_64-pc-linux-gnu-gcc -Wextra -Wfloat-conversion -Wno-missing-field-initializers -Wall -Wstrict-prototypes -std=c11 -fwrapv -fstack-protector-strong -fvisibility=hidden -fPIC -march=native -O2 -pipe -flto -Ikitty -I/usr/include/python3.6m -Wall -shared -Wl,-O1 -Wl,--as-needed -flto build/subseq_matcher-charsets.c.o build/subseq_matcher-main.c.o build/subseq_matcher-output.c.o build/subseq_matcher-score.c.o build/subseq_matcher-unix_compat.c.o -lpthread -ldl -lutil -lm -L/usr/lib64 -lpython3.6m -Xlinker -export-dynamic -o build/kittens/choose/subseq_matcher.so
* Fixing shebang in /usr/lib64/kitty/__main__.py.
* Fixing shebang in /usr/lib64/kitty/kitty/borders.py.
[EDITED OUT MANY SIMILAR LINES]
* Fixing shebang in /usr/lib64/kitty/kittens/unicode_input/main.py.
* Final size of build directory: 47940 KiB (46.8 MiB)
* Final size of installed tree: 12176 KiB (11.8 MiB)
* Updating icons cache ...
[ ok ]
But for logclean and its dependency, the log files are in the expected shape:
�[32;01m * �[39;49;00mPackage: app-portage/logclean-18.3
�[32;01m * �[39;49;00mRepository: mv
�[32;01m * �[39;49;00mMaintainer: martin@mvath.de
�[32;01m * �[39;49;00mUpstream: martin@mvath.de
�[32;01m * �[39;49;00mUSE: abi_x86_64 amd64 elibc_glibc kernel_linux userland_GNU
�[32;01m * �[39;49;00mFEATURES: network-sandbox preserve-libs sandbox userpriv usersandbox
>>> Unpacking source...
>>> Unpacking logclean-18.3.tar.gz to /var/tmp/portage/app-portage/logclean-18.3/work
>>> Source unpacked in /var/tmp/portage/app-portage/logclean-18.3/work
>>> Preparing source in /var/tmp/portage/app-portage/logclean-18.3/work/logclean-18.3 ...
>>> Source prepared.
>>> Configuring source in /var/tmp/portage/app-portage/logclean-18.3/work/logclean-18.3 ...
>>> Source configured.
>>> Compiling source in /var/tmp/portage/app-portage/logclean-18.3/work/logclean-18.3 ...
>>> Source compiled.
>>> Test phase [not enabled]: app-portage/logclean-18.3
>>> Install app-portage/logclean-18.3 into /var/tmp/portage/app-portage/logclean-18.3/image
>>> Completed installing app-portage/logclean-18.3 into /var/tmp/portage/app-portage/logclean-18.3/image
�[32;01m*�[0m Final size of build directory: 52 KiB
�[32;01m*�[0m Final size of installed tree: 52 KiB
Is there some portage FEATURE or PORTAGE_ELOG_CLASSES setting that can influence this? (I was playing around with those when installing logclean.) Package successfully installed more recently than logclean also do not contain the expected Completed installing. Moreover, some logs of installations seem to be spread over multiple files (for different phases of the install), with nowhere Completed installing present.
Sorry for the late reply: I am very busy in RL these days.
some logs of installations seem to be spread over multiple files (for different phases of the install)
That's the reason for the Completed installing check: I never liked it, as it is obviously very hackish, but it was the only way I found to distinguish the “main” log from the trivial logs sometimes generated for merging or unmerging the old version etc (which I do not want to keep and which have usually a newer date).
I had emerged kitty-0.15.1 for testing, and the log contained Completed installing. In my make.conf there is PORTAGE_ELOG_SYSTEM="echo save".
My FEATURES variable contains
assume-digests -buildsyspkg -candy ccache -clean-logs -compress-build-logs -compressdebug -compress-index config-protect-if-modified -digests -distcc -distcc-pump -downgrade-backup -fakeroot -fail-clean fixlafiles -force-mirror -force-prefix -getbinpkg -installsources ipc-sandbox -keeptemp -keepwork -lmirror -merge-sync -metadata-transfer -mirror multilib-strict network-sandbox news -noauto -noclean -nodoc -noinfo -noman -nostrip -notitles -parallel-fetch parallel-install -prelink-checksums -python-trace sandbox -severe sfperms -sign -skiprocheck -split-elog split-log -splitdebug strict -stricter -suidctl -test test-fail-continue -unmerge-backup -unmerge-logs unmerge-orphans -unknown-features-filter unknown-features-warn userfetch userpriv usersandbox usersync -xattr collision-protect protect-owned cgroup
I had added split-logs relatively late, so I doubt that this is related.
Maybe FEATURES=-split-elog is the difference
Maybe FEATURES=-split-elog is the difference
No, that is about splitting into per-category directories.
I had emerged kitty-0.15.1 for testing, and the log contained
Completed installing. In my make.conf there isPORTAGE_ELOG_SYSTEM="echo save".
Could you copy-paste the kitty log here? Then I can compare.
My FEATURES variable contains
Can you post the output of portageq envvar FEATURES | xargs -n 1
This gives the active features. That will be easier to compare. Mine is
assume-digests
binpkg-docompress
binpkg-dostrip
binpkg-logs
candy
cgroup
config-protect-if-modified
distlocks
ebuild-locks
fail-clean
fixlafiles
ipc-sandbox
merge-sync
multilib-strict
network-sandbox
news
parallel-fetch
pid-sandbox
preserve-libs
protect-owned
sandbox
sfperms
split-elog
split-log
strict
unknown-features-warn
unmerge-logs
unmerge-orphans
userfetch
userpriv
usersandbox
usersync
xattr
Maybe a difference is that I do not use any portage parallelism (except a job number in MAKEOPTS)
Could you copy-paste the kitty log here?
Available for 1 month (lines with PII manually removed, and for simpler comparison with exported empty MAKEOPTS, *FLAGS) on https://pastebin.com/1K1UQjbX
I am not sure (and was too lazy to check) whether portageq does take portage defaults correctly into account. Anyway, here is the output
binpkg-docompress
binpkg-dostrip
ccache
cgroup
collision-protect
config-protect-if-modified
distlocks
ebuild-locks
fixlafiles
ipc-sandbox
mount-sandbox
multilib-strict
network-sandbox
news
parallel-install
pid-sandbox
preserve-libs
protect-owned
qa-unresolved-soname-deps
sandbox
sfperms
split-log
strict
test-fail-continue
unknown-features-warn
unmerge-orphans
userfetch
userpriv
usersandbox
usersync```
Maybe a difference is that I do not use any portage parallelism (except a job number in MAKEOPTS)
My impression is that I use even less parallelism (no parallel-install in my FEATURES).
Could you copy-paste the kitty log here?
Available for 1 month (lines with PII manually removed, and for simpler comparison with exported empty MAKEOPTS, *FLAGS) on https://pastebin.com/1K1UQjbX
I had a look. The log is clearly different. Namely, I'm missing all the lines that start with >>> or --- (and some whitespace lines). But the why is still a mystery to me.
I am not sure (and was too lazy to check) whether portageq does take portage defaults correctly into account. Anyway, here is the output
List of FEATURES I have which you don't:
assume-digests
binpkg-logs
candy
fail-clean
merge-sync
parallel-fetch
split-elog
unmerge-logs
xattr
(I've since disabled candy, without effect.)
List of FEATURES you have which I don't:
ccache
collision-protect
mount-sandbox
parallel-install
qa-unresolved-soname-deps
test-fail-continue
None of the FEATURES seem to have to do with the missing >>> lines, AFAICT from the make.conf man page. (qa-unresolved-soname-deps seems undocumented there, but from the name it doesn't seem to be related.)
I think I need to get someone's attention who's involved in portage development…
In portage we could make the Completed installing message independent of the emerge --quiet option. This message is emitted by the bash ebuild process, so we can assume that anyone who sees it is willing to see some potentially verbose output.
@vaeth :
My use of --quiet during emerges is what causes the issue I experience. (Thanks to zmedico for helping me realize this.) Now that I know the reason, I have come to the conclusion that this issue would be best resolved for me by adding a feature to logclean that allows one to not do the second stage, i.e., to only clean based on filename, not on file content.
@zmedico :
In portage we could make the
Completed installingmessage independent of the emerge --quiet option.
That may not completely fix the issue here, as with --quiet, apparently also the logs for a single emerge are split across multiple files. It also feels like a hack, because it is singularly aimed at fixing this issue with logclean caused by a fragile way of detecting a successful emerge. It would be more useful to add something in the logs that is there in all cases, independent of --quiet, so that logclean does not need to do something fragile.
Furthermore, now that I am aware of it, I'd like to make sure somehow to get all the __vecho output in the build logs, irrespective of whether I use --quiet or not. But I guess that should go in a gentoo bug report; I should first learn about the differences between --quiet and --quiet-build.
This message is emitted by the bash ebuild process, so we can assume that anyone who sees it is willing to see some potentially verbose output.
I have no idea what this means exactly, so I cannot give an opinion.