aquasecurity/btfhub

btfgen.sh produces empty output

saschagrunert opened this issue · 3 comments

saschagrunert commented

When running the btfgen.sh script for my bpf objects, then the output will be only empty directories.

> ./btfgen.sh -a x86_64 -o recorder.bpf.o.amd64
centos/7/x86_64
centos/8/x86_64
fedora/29/x86_64
fedora/30/x86_64
fedora/31/x86_64
fedora/32/x86_64
fedora/33/x86_64
fedora/34/x86_64
ubuntu/18.04/x86_64
ubuntu/20.04/x86_64

> find ../custom-archive/
../custom-archive/
../custom-archive/centos
../custom-archive/centos/8
../custom-archive/centos/8/x86_64
../custom-archive/centos/7
../custom-archive/centos/7/x86_64
../custom-archive/ubuntu
../custom-archive/ubuntu/20.04
../custom-archive/ubuntu/20.04/x86_64
../custom-archive/ubuntu/18.04
../custom-archive/ubuntu/18.04/x86_64
../custom-archive/fedora
../custom-archive/fedora/33
../custom-archive/fedora/33/x86_64
../custom-archive/fedora/34
../custom-archive/fedora/34/x86_64
../custom-archive/fedora/32
../custom-archive/fedora/32/x86_64
../custom-archive/fedora/30
../custom-archive/fedora/30/x86_64
../custom-archive/fedora/29
../custom-archive/fedora/29/x86_64
../custom-archive/fedora/31
../custom-archive/fedora/31/x86_64
../custom-archive/.gitignore

Did I miss anything?

rafaeldtinoco commented

Oops, have you ran "extract.sh" before ?

It will extract all .tar.xz files within the cloned repository. The extracted files are .gitignored so there will be no problem in coexisting in the same directory. The btfgen.sh will use the extracted files. The reasoning why they're separate scripts is that you may extract a single time and re-run btfgen.sh multiple times (so you save time using already extracted files).

Let me know if it doesn't work.

rafaeldtinoco commented

@saschagrunert

[rafaeldtinoco@fujitsu:~/.../ebpf/btfhub][main]$ ./tools/extract.sh -a x86_64
...

[rafaeldtinoco@fujitsu:~/.../ebpf/btfhub][main]$ ./tools/btfgen.sh -a x86_64 -o ./objs/recorder.bpf.o.amd64
centos/7/x86_64
generating btf from ./archive/centos/7/x86_64/3.10.0-693.2.2.el7.x86_64.btf
processing ./objs/recorder.bpf.o.amd64 object
generating btf from ./archive/centos/7/x86_64/3.10.0-1062.4.2.el7.x86_64.btf
processing ./objs/recorder.bpf.o.amd64 object
generating btf from ./archive/centos/7/x86_64/3.10.0-123.el7.x86_64.btf
processing ./objs/recorder.bpf.o.amd64 object
generating btf from ./archive/centos/7/x86_64/3.10.0-957.5.1.el7.x86_64.btf
processing ./objs/recorder.bpf.o.amd64 object

[rafaeldtinoco@fujitsu:~/.../ebpf/btfhub][main]$ find custom-archive
[rafaeldtinoco@fujitsu:~/.../ebpf/btfhub][main]$ find custom-archive | more
custom-archive
custom-archive/fedora
custom-archive/fedora/29
custom-archive/fedora/29/x86_64
custom-archive/fedora/29/x86_64/5.3.11-100.fc29.x86_64.btf
custom-archive/fedora/29/x86_64/4.18.16-300.fc29.x86_64.btf
custom-archive/fedora/30
custom-archive/fedora/30/x86_64
custom-archive/fedora/30/x86_64/5.6.13-100.fc30.x86_64.btf
custom-archive/fedora/30/x86_64/5.0.9-301.fc30.x86_64.btf
custom-archive/fedora/33
custom-archive/fedora/33/x86_64
custom-archive/fedora/33/x86_64/5.8.15-301.fc33.x86_64.btf
custom-archive/fedora/34
custom-archive/fedora/34/x86_64
custom-archive/fedora/34/x86_64/5.11.12-300.fc34.x86_64.btf
custom-archive/fedora/31
custom-archive/fedora/31/x86_64
custom-archive/fedora/31/x86_64/5.8.18-100.fc31.x86_64.btf
custom-archive/fedora/31/x86_64/5.3.7-301.fc31.x86_64.btf
custom-archive/fedora/32
custom-archive/fedora/32/x86_64
custom-archive/fedora/32/x86_64/5.6.6-300.fc32.x86_64.btf
custom-archive/.gitignore
custom-archive/centos
custom-archive/centos/8
custom-archive/centos/8/x86_64
custom-archive/centos/8/x86_64/4.18.0-80.el8.x86_64.btf
custom-archive/centos/8/x86_64/4.18.0-240.1.1.el8_3.x86_64.btf
custom-archive/centos/8/x86_64/4.18.0-147.8.1.el8_1.x86_64.btf
custom-archive/centos/8/x86_64/4.18.0-305.19.1.el8_4.x86_64.btf
custom-archive/centos/8/x86_64/4.18.0-193.1.2.el8_2.x86_64.btf
custom-archive/centos/8/x86_64/4.18.0-193.6.3.el8_2.x86_64.btf
custom-archive/centos/8/x86_64/4.18.0-305.12.1.el8_4.x86_64.btf
custom-archive/centos/8/x86_64/4.18.0-147.3.1.el8_1.x86_64.btf
custom-archive/centos/8/x86_64/4.18.0-305.17.1.el8_4.x86_64.btf
custom-archive/centos/8/x86_64/4.18.0-305.25.1.el8_4.x86_64.btf
custom-archive/centos/8/x86_64/4.18.0-305.10.2.el8_4.x86_64.btf
custom-archive/centos/8/x86_64/4.18.0-193.el8.x86_64.btf
...

AND, if you want to "dedup" resulting BTFs (won't work when embedding to GO as it does not support symlinks) you can run:

[rafaeldtinoco@fujitsu:~/.../ebpf/btfhub][main]$ ./tools/dedup.sh
-- CENTOS: ./custom-archive/centos/7/x86_64
previous: 3.10.0-123.el7.x86_64.btf () (3.10.0-123), current: 3.10.0-123.1.2.el7.x86_64.btf () (3.10.0-123) = CHANGED ? 0
previous: 3.10.0-123.1.2.el7.x86_64.btf () (3.10.0-123), current: 3.10.0-123.4.2.el7.x86_64.btf () (3.10.0-123) = CHANGED ? 0
previous: 3.10.0-123.4.2.el7.x86_64.btf () (3.10.0-123), current: 3.10.0-123.4.4.el7.x86_64.btf () (3.10.0-123) = CHANGED ? 0
previous: 3.10.0-123.4.4.el7.x86_64.btf () (3.10.0-123), current: 3.10.0-123.6.3.el7.x86_64.btf () (3.10.0-123) = CHANGED ? 0
previous: 3.10.0-123.6.3.el7.x86_64.btf () (3.10.0-123), current: 3.10.0-123.8.1.el7.x86_64.btf () (3.10.0-123) = CHANGED ? 0
previous: 3.10.0-123.8.1.el7.x86_64.btf () (3.10.0-123), current: 3.10.0-123.9.2.el7.x86_64.btf () (3.10.0-123) = CHANGED ? 0
previous: 3.10.0-123.9.2.el7.x86_64.btf () (3.10.0-123), current: 3.10.0-123.9.3.el7.x86_64.btf () (3.10.0-123) = CHANGED ? 0
previous: 3.10.0-123.9.3.el7.x86_64.btf () (3.10.0-123), current: 3.10.0-123.13.1.el7.x86_64.btf () (3.10.0-123) = CHANGED ? 0
previous: 3.10.0-123.13.1.el7.x86_64.btf () (3.10.0-123), current: 3.10.0-123.13.2.el7.x86_64.btf () (3.10.0-123) = CHANGED ? 0
previous: 3.10.0-123.13.2.el7.x86_64.btf () (3.10.0-123), current: 3.10.0-123.20.1.el7.x86_64.btf () (3.10.0-123) = CHANGED ? 0
previous: 3.10.0-229.el7.x86_64.btf () (3.10.0-229), current: 3.10.0-229.1.2.el7.x86_64.btf () (3.10.0-229) = CHANGED ? 0
previous: 3.10.0-229.1.2.el7.x86_64.btf () (3.10.0-229), current: 3.10.0-229.4.2.el7.x86_64.btf () (3.10.0-229) = CHANGED ? 0
previous: 3.10.0-229.4.2.el7.x86_64.btf () (3.10.0-229), current: 3.10.0-229.7.2.el7.x86_64.btf () (3.10.0-229) = CHANGED ? 0
previous: 3.10.0-229.7.2.el7.x86_64.btf () (3.10.0-229), current: 3.10.0-229.11.1.el7.x86_64.btf () (3.10.0-229) = CHANGED ? 0
previous: 3.10.0-229.11.1.el7.x86_64.btf () (3.10.0-229), current: 3.10.0-229.14.1.el7.x86_64.btf () (3.10.0-229) = CHANGED ? 0
previous: 3.10.0-229.14.1.el7.x86_64.btf () (3.10.0-229), current: 3.10.0-229.20.1.el7.x86_64.btf () (3.10.0-229) = CHANGED ? 0
...
saschagrunert commented

Awesome, thank you for your help. It was the missing extraction.