aquasecurity/btfhub

btfgen.sh should support multiple objects as input (just like btfgen binary does)

rafaeldtinoco opened this issue · 2 comments

rafaeldtinoco commented

Opening this issue per @mauriciovasquezbernal request. It makes sense that btfgen knows how to handle multiple objects for the tailored BTF files generation (so one can tailor the BTFs based on multiple objects).

rafaeldtinoco commented

@mauriciovasquezbernal,

I have made 2 changes based on recent feedback you've given:

  1. btfgen.sh needed to support multiple objects (this issue)
  2. extracting all BTF files consumed too much disk space

Together with changes proposed at:

https://github.com/kinvolk/btfgen/pull/12/commits

I have generated 2 new btfgen binaries inside tools/bin/ directory. btfgen tool supports now having file or directories as input/output arguments, as well as deleting result BTF files that were poisoned (the default option set in btfgen.sh).

I'm pushing this change and you will be able to test your use case by doing something like:

Note that you don't need to run extract.sh in order to run newbtfgen.sh script.

[rafaeldtinoco@fujitsu:~/.../ebpf/btfhub]$ sudo ./tools/newbtfgen.sh -a $(uname -m) -o /home/rafaeldtinoco/one.bpf.o -o /home/rafaeldtinoco/two.bpf.o
SBTF: ./3.10.0-327.4.5.el7.x86_64.btf
OBJ : /home/rafaeldtinoco/one.bpf.o
OBJ : /home/rafaeldtinoco/two.bpf.o
DBTF: ./custom-archive/centos/7/x86_64/3.10.0-327.4.5.el7.x86_64.btf
WARN: generated btf (./custom-archive/centos/7/x86_64/3.10.0-327.4.5.el7.x86_64.btf) is poisoned (deleting)
SBTF: ./3.10.0-1160.6.1.el7.x86_64.btf
OBJ : /home/rafaeldtinoco/one.bpf.o
OBJ : /home/rafaeldtinoco/two.bpf.o
DBTF: ./custom-archive/centos/7/x86_64/3.10.0-1160.6.1.el7.x86_64.btf
WARN: generated btf (./custom-archive/centos/7/x86_64/3.10.0-1160.6.1.el7.x86_64.btf) is poisoned (deleting)
SBTF: ./3.10.0-693.1.1.el7.x86_64.btf
OBJ : /home/rafaeldtinoco/one.bpf.o
OBJ : /home/rafaeldtinoco/two.bpf.o
DBTF: ./custom-archive/centos/7/x86_64/3.10.0-693.1.1.el7.x86_64.btf
WARN: generated btf (./custom-archive/centos/7/x86_64/3.10.0-693.1.1.el7.x86_64.btf) is poisoned (deleting)
...
SBTF: ./5.4.0-1012-azure.btf
OBJ : /home/rafaeldtinoco/one.bpf.o
OBJ : /home/rafaeldtinoco/two.bpf.o
DBTF: ./custom-archive/ubuntu/20.04/x86_64/5.4.0-1012-azure.btf
SBTF: ./5.4.0-1047-aws.btf
OBJ : /home/rafaeldtinoco/one.bpf.o
OBJ : /home/rafaeldtinoco/two.bpf.o
DBTF: ./custom-archive/ubuntu/20.04/x86_64/5.4.0-1047-aws.btf
SBTF: ./5.4.0-1059-aws.btf
OBJ : /home/rafaeldtinoco/one.bpf.o
OBJ : /home/rafaeldtinoco/two.bpf.o
DBTF: ./custom-archive/ubuntu/20.04/x86_64/5.4.0-1059-aws.btf
SBTF: ./5.4.0-59-generic.btf
OBJ : /home/rafaeldtinoco/one.bpf.o
OBJ : /home/rafaeldtinoco/two.bpf.o
DBTF: ./custom-archive/ubuntu/20.04/x86_64/5.4.0-59-generic.btf
SBTF: ./5.4.0-67-generic.btf
OBJ : /home/rafaeldtinoco/one.bpf.o
OBJ : /home/rafaeldtinoco/two.bpf.o
DBTF: ./custom-archive/ubuntu/20.04/x86_64/5.4.0-67-generic.btf
SBTF: ./5.4.0-1021-aws.btf
OBJ : /home/rafaeldtinoco/one.bpf.o
OBJ : /home/rafaeldtinoco/two.bpf.o
DBTF: ./custom-archive/ubuntu/20.04/x86_64/5.4.0-1021-aws.btf
SBTF: ./5.11.0-1020-aws.btf
OBJ : /home/rafaeldtinoco/one.bpf.o
OBJ : /home/rafaeldtinoco/two.bpf.o
DBTF: ./custom-archive/ubuntu/20.04/x86_64/5.11.0-1020-aws.btf
SBTF: ./5.8.0-1038-aws.btf
OBJ : /home/rafaeldtinoco/one.bpf.o
OBJ : /home/rafaeldtinoco/two.bpf.o
DBTF: ./custom-archive/ubuntu/20.04/x86_64/5.8.0-1038-aws.btf
SBTF: ./5.11.0-1017-azure.btf
OBJ : /home/rafaeldtinoco/one.bpf.o
OBJ : /home/rafaeldtinoco/two.bpf.o
DBTF: ./custom-archive/ubuntu/20.04/x86_64/5.11.0-1017-azure.btf
SBTF: ./5.4.0-74-generic.btf
OBJ : /home/rafaeldtinoco/one.bpf.o
OBJ : /home/rafaeldtinoco/two.bpf.o
DBTF: ./custom-archive/ubuntu/20.04/x86_64/5.4.0-74-generic.btf

[rafaeldtinoco@fujitsu:~/.../ebpf/btfhub]$ find custom-archive | wc -l
289

Note that the initial BTFs (for centos7, or kernels 4.15) will likely be poisoned due to the features we are currently using in our eBPF objects nowadays. This does not mean centos7 BTF files will only generate poisoned BTF files to ANY given bpf object, but likely most of them (more recent kernel BTFs are likely to have all relocations satisfied).

  • 👀1
rafaeldtinoco commented

@mtcherni95

You should consider using newbtfgen.sh, instead of btfgen.sh, for your release at the end of this week. You don't need to run "extract.sh" before using it AND it has better features (like not generating BTF files for kernels that wouldn't work).