abrt/faf

Retrace cannot find packages even if downloaded

Closed this issue · 9 comments

djuarezg commented

TL;DR: Retrace fails as it says that it cannot find source or binary packages (but it does find the debug ones).

This one is a bit long but here are all steps on how to reproduce:

  • Take a CentOS machine, configure /etc/libreport/plugins/ureport.conf with your server URL (URL = https://abrtanalytics.web.cern.ch/faf in our case).

  • Add CentOS 7 release:

faf releaseadd -o centos --opsys-release 7 --status ACTIVE
  • Configure repos that provide all involved packages:
sh-5.0$ faf repolist | awk '{print $1}' | xargs -L 1 faf repoinfo
Name: updates_debug
Type: dnf
URL: http://linuxsoft.cern.ch/cern/centos/7/updates/Debug/x86_64/
GPG check enabled: False
Name: updates_basic
Type: dnf
URL: http://linuxsoft.cern.ch/cern/centos/7/updates/x86_64/
GPG check enabled: False
Name: updates_source
Type: dnf
URL: http://linuxsoft.cern.ch/cern/centos/7/updates/Source/
GPG check enabled: False
Name: update_sources
Type: dnf
URL: http://linuxsoft.cern.ch/cern/centos/7/updates/Sources/
GPG check enabled: False
Name: os
Type: dnf
URL: http://linuxsoft.cern.ch/cern/centos/7/os/x86_64/
GPG check enabled: False
Name: os_source
Type: dnf
URL: http://linuxsoft.cern.ch/cern/centos/7/os/Source/
URL: http://linuxsoft.cern.ch/cern/centos/7/os/Sources/
GPG check enabled: False
  • Assign repos to the previous release
faf repoassign <each_repo> "CentOS 7" x86_64

  • Pull components

  • Reposync only packages that will be relevant to the crash we will send:

faf reposync --name-prefix glibc -d -v
faf reposync --name-prefix coreutils -d -v
# This will start printing that debuginfo, source rpms and binary rpms are being downloaded, stored afterwards on /var/spool/faf/lob/Package/package/...
  • Force a crash on the CentOS machine:
sleep 10m &
kill -SIGSEGV <ID>

  • This will send the ureport to your faf server, run faf save-reports and faf create-problems.

  • Run faf retrace -v -d:

[2020-05-19 09:38:22] INFO:faf.Retrace: Processing 'Crash of user-space binary' problem type
[2020-05-19 09:38:23] DEBUG:faf.Retrace: [1 / 4] Processing '__nanosleep' @ '/lib64/libc.so.6'
[2020-05-19 09:38:23] DEBUG:faf.CoredumpProblem: Build-id: 398944d32cf16a67af51067a326e6c0cc14f90ed
[2020-05-19 09:38:23] DEBUG:faf.CoredumpProblem: File names: /usr/lib/debug/.build-id/39/8944d32cf16a67af51067a326e6c0cc14f90ed.debug, /usr/lib/.build-id/39/8944d32cf16a67af51067a326e6c0cc14f90ed
[2020-05-19 09:38:23] DEBUG:faf.CoredumpProblem: Debug Package: glibc-debuginfo-2.17-292.el7.x86_64
[2020-05-19 09:38:23] DEBUG:faf.CoredumpProblem: Binary Package: Not found
[2020-05-19 09:38:23] DEBUG:faf.CoredumpProblem: Source Package: Not found
[2020-05-19 09:38:23] DEBUG:faf.Retrace: [2 / 4] Processing 'rpl_nanosleep' @ '/usr/bin/sleep'
[2020-05-19 09:38:23] DEBUG:faf.CoredumpProblem: Build-id: 32370fcda191ea2f76c58551ac330fdd5080fda1
[2020-05-19 09:38:23] DEBUG:faf.CoredumpProblem: File names: /usr/lib/debug/.build-id/32/370fcda191ea2f76c58551ac330fdd5080fda1.debug, /usr/lib/.build-id/32/370fcda191ea2f76c58551ac330fdd5080fda1
[2020-05-19 09:38:23] DEBUG:faf.CoredumpProblem: Debug Package: coreutils-debuginfo-8.22-24.el7.x86_64
[2020-05-19 09:38:23] DEBUG:faf.CoredumpProblem: Binary Package: Not found
[2020-05-19 09:38:23] DEBUG:faf.CoredumpProblem: Source Package: Not found
[2020-05-19 09:38:23] DEBUG:faf.Retrace: [3 / 4] Processing 'xnanosleep' @ '/usr/bin/sleep'
[2020-05-19 09:38:23] DEBUG:faf.CoredumpProblem: Build-id: 32370fcda191ea2f76c58551ac330fdd5080fda1
[2020-05-19 09:38:23] DEBUG:faf.CoredumpProblem: File names: /usr/lib/debug/.build-id/32/370fcda191ea2f76c58551ac330fdd5080fda1.debug, /usr/lib/.build-id/32/370fcda191ea2f76c58551ac330fdd5080fda1
[2020-05-19 09:38:23] DEBUG:faf.CoredumpProblem: Debug Package: coreutils-debuginfo-8.22-24.el7.x86_64
[2020-05-19 09:38:23] DEBUG:faf.CoredumpProblem: Binary Package: Not found
[2020-05-19 09:38:23] DEBUG:faf.CoredumpProblem: Source Package: Not found
[2020-05-19 09:38:23] DEBUG:faf.Retrace: [4 / 4] Processing 'main' @ '/usr/bin/sleep'
[2020-05-19 09:38:23] DEBUG:faf.CoredumpProblem: Build-id: 32370fcda191ea2f76c58551ac330fdd5080fda1
[2020-05-19 09:38:23] DEBUG:faf.CoredumpProblem: File names: /usr/lib/debug/.build-id/32/370fcda191ea2f76c58551ac330fdd5080fda1.debug, /usr/lib/.build-id/32/370fcda191ea2f76c58551ac330fdd5080fda1
[2020-05-19 09:38:23] DEBUG:faf.CoredumpProblem: Debug Package: coreutils-debuginfo-8.22-24.el7.x86_64
[2020-05-19 09:38:23] DEBUG:faf.CoredumpProblem: Binary Package: Not found
[2020-05-19 09:38:23] DEBUG:faf.CoredumpProblem: Source Package: Not found
[2020-05-19 09:38:23] INFO:faf.Retrace: Starting the retracing process
[2020-05-19 09:38:23] INFO:faf.Retrace: All done

NOTE: This sample report corresponds in our case to https://abrtanalytics.web.cern.ch/faf/reports/1/

select_from_packages.txt

djuarezg commented

For your information, I could replicate it with other reports, always getting the same behaviour. I even retried on a fresh deployment.

ernestask commented

Yeah, I see what the problem is now. Someone botched the DNF code incredibly badly.

faf/src/pyfaf/repos/dnf.py

Line 106 in 5bc42f3

base_package_name=package.name,

This results in, for example, coreutils and coreutils-debuginfo having distinct builds, even though the base is identical.

Then we have this beauty:

faf/src/pyfaf/problemtypes/core.py

Line 495 in 5bc42f3

db_bin_package = get_package_by_file_build_arch(db, path,

It uses the build of the debuginfo package, which only has the debuginfo package associated with it.

ernestask commented

I’ll see how YUM defined the base name property. I can either yell at DNF devs or just do some suffix stripping.

ernestask commented

Went with the latter to save some brain cells.

djuarezg commented

@ernestask You sure this fixes the issue?

[2020-05-25 12:17:14] DEBUG:faf.CoredumpProblem: Build-id: 398944d32cf16a67af51067a326e6c0cc14f90ed
[2020-05-25 12:17:14] DEBUG:faf.CoredumpProblem: File names: /usr/lib/debug/.build-id/39/8944d32cf16a67af51067a326e6c0cc14f90ed.debug, /usr/lib/.build-id/39/8944d32cf16a67af51067a326e6c0cc14f90ed
[2020-05-25 12:17:14] DEBUG:faf.CoredumpProblem: Debug Package: glibc-debuginfo-2.17-292.el7.x86_64
[2020-05-25 12:17:14] DEBUG:faf.CoredumpProblem: Binary Package: glibc-2.17-292.el7.x86_64
[2020-05-25 12:17:14] DEBUG:faf.CoredumpProblem: Source Package: Not found
[2020-05-25 12:17:14] DEBUG:faf.Retrace: [2 / 4] Processing 'rpl_nanosleep' @ '/usr/bin/sleep'
[2020-05-25 12:17:14] DEBUG:faf.CoredumpProblem: Build-id: 32370fcda191ea2f76c58551ac330fdd5080fda1
[2020-05-25 12:17:14] DEBUG:faf.CoredumpProblem: File names: /usr/lib/debug/.build-id/32/370fcda191ea2f76c58551ac330fdd5080fda1.debug, /usr/lib/.build-id/32/370fcda191ea2f76c58551ac330fdd5080fda1
[2020-05-25 12:17:14] DEBUG:faf.CoredumpProblem: Debug Package: coreutils-debuginfo-8.22-24.el7.x86_64
[2020-05-25 12:17:14] DEBUG:faf.CoredumpProblem: Binary Package: coreutils-8.22-24.el7.x86_64
[2020-05-25 12:17:14] DEBUG:faf.CoredumpProblem: Source Package: Not found
[2020-05-25 12:17:14] DEBUG:faf.Retrace: [3 / 4] Processing 'xnanosleep' @ '/usr/bin/sleep'
[2020-05-25 12:17:14] DEBUG:faf.CoredumpProblem: Build-id: 32370fcda191ea2f76c58551ac330fdd5080fda1
[2020-05-25 12:17:14] DEBUG:faf.CoredumpProblem: File names: /usr/lib/debug/.build-id/32/370fcda191ea2f76c58551ac330fdd5080fda1.debug, /usr/lib/.build-id/32/370fcda191ea2f76c58551ac330fdd5080fda1
[2020-05-25 12:17:14] DEBUG:faf.CoredumpProblem: Debug Package: coreutils-debuginfo-8.22-24.el7.x86_64
[2020-05-25 12:17:14] DEBUG:faf.CoredumpProblem: Binary Package: coreutils-8.22-24.el7.x86_64
[2020-05-25 12:17:14] DEBUG:faf.CoredumpProblem: Source Package: Not found
[2020-05-25 12:17:14] DEBUG:faf.Retrace: [4 / 4] Processing 'main' @ '/usr/bin/sleep'
[2020-05-25 12:17:14] DEBUG:faf.CoredumpProblem: Build-id: 32370fcda191ea2f76c58551ac330fdd5080fda1
[2020-05-25 12:17:14] DEBUG:faf.CoredumpProblem: File names: /usr/lib/debug/.build-id/32/370fcda191ea2f76c58551ac330fdd5080fda1.debug, /usr/lib/.build-id/32/370fcda191ea2f76c58551ac330fdd5080fda1
[2020-05-25 12:17:14] DEBUG:faf.CoredumpProblem: Debug Package: coreutils-debuginfo-8.22-24.el7.x86_64
[2020-05-25 12:17:14] DEBUG:faf.CoredumpProblem: Binary Package: coreutils-8.22-24.el7.x86_64
[2020-05-25 12:17:14] DEBUG:faf.CoredumpProblem: Source Package: Not found
[2020-05-25 12:17:14] DEBUG:faf.Retrace: [1 / 2] Creating task for 'glibc-debuginfo-2.17-292.el7.x86_64'
[2020-05-25 12:17:14] DEBUG:faf.Retrace: [2 / 2] Creating task for 'coreutils-debuginfo-8.22-24.el7.x86_64'
[2020-05-25 12:17:14] INFO:faf.Retrace: Starting the retracing process
[2020-05-25 12:17:14] INFO:faf.thread.RetracePool(Worker_0): [1 / 2] Retracing glibc-debuginfo-2.17-292.el7.x86_64
[2020-05-25 12:17:14] INFO:faf.thread.RetracePool(Worker_1): [2 / 2] Retracing coreutils-debuginfo-8.22-24.el7.x86_64
[2020-05-25 12:17:14] WARNING:faf.thread.RetracePool(Worker_0): Retracing task encountered an exception: [Errno 2] No such file or directory: 'cpio'
[2020-05-25 12:17:14] WARNING:faf.thread.RetracePool(Worker_1): Retracing task encountered an exception: [Errno 2] No such file or directory: 'cpio'
[2020-05-25 12:17:14] INFO:faf.Retrace: All done

A reposync actually stated that the Source rpm is known:

[2020-05-25 12:15:55] DEBUG:faf.RepoSync: [589 / 3176] Processing package glibc
[2020-05-25 12:15:55] DEBUG:faf.RepoSync: Known package glibc-2.17-292.el7.src.rpm
# and
[2020-05-25 12:21:00] DEBUG:faf.RepoSync: [290 / 3176] Processing package coreutils
[2020-05-25 12:21:00] DEBUG:faf.RepoSync: Known package coreutils-8.22-23.el7.src.rpm
[2020-05-25 12:21:00] DEBUG:faf.RepoSync: [291 / 3176] Processing package coreutils
[2020-05-25 12:21:00] DEBUG:faf.RepoSync: Known package coreutils-8.22-24.el7.src.rpm

Although I guess that I will have to open a new issue regarding the lack of cpio

ernestask commented

Oh, yeah, I forgot the cpio bit. It’s supposed to be included in the standard comps group, but the way the Fedora container images are built is a complete mystery to me. Please do create an issue for it.

djuarezg commented

I will check also if there is anything else missing before opening it and build and test it myself, proposing a PR. I would also like to thank you for all the help regarding these issues!

ernestask commented

SRPM lookup is apparently disabled by default and I have no idea why (as usual). See /etc/faf/plugins/retrace.conf.

ernestask commented

In any case, the spec file now has an explicit dependency on cpio: e876d2e.

  • 👍1