cilium/pwru

Failed to inject filter ebpf for kprobe_skb_2: register r8 used twice

kdrag0n opened this issue · 11 comments

kdrag0n commented

With the static arm64 build of pwru v1.0.1, trying to use a filter results in the following error:

# ./pwru  'dst host 192.168.207.2'
2023/09/04 22:24:06 Failed to inject filter ebpf for kprobe_skb_2: register r8 used twice
# ./pwru  'port 80'
2023/09/04 22:26:28 Failed to inject filter ebpf for kprobe_skb_5: register r8 used twice

Kernel version: 6.4.14
Distro: Alpine edge

brb commented

Thanks for the issue. Does the same appear with v1.0.0?

kdrag0n commented

v1.0.0 seems to get past the filter generation, but fails while attaching kprobes because it doesn't have 5e6011d.

brb commented

@jschwinger233 Mind taking a look?

  • 👍1
kdrag0n commented

Update: Just built v1.0.0 with the rlimit increased (as a workaround) and it seems to work fine.

  • 👍2
dakehero commented

The pwru v1.0.1 amd64 build also has this issue on kernel 5.4

brb commented

Interesting, if I build v1.0.1 pwru from scratch (Archlinux, LLVM 16, gcc 13), I get it working. But with the released version (built by https://github.com/cilium/pwru/actions/runs/6071926301/job/16470942060, LLVM 13, gcc 12) I'm bumping into the same problem:

> sudo ./pwru 'port 80'
2023/09/05 09:49:58 Failed to inject filter ebpf for kprobe_skb_3: register r8 used twice
jschwinger233 commented

Checking out upstream locally and running make local-release seems fine.

jschwinger233 commented

I think it could be caused by wrong registers found for skb->data and skb->date_end.

In a good kprobemultipwru_bpfel_x86.o, we have

; 	bpf_printk("%d %d", data, data_end);
      79:	r1 = 16 ll
      81:	r2 = 6
      82:	r3 = r9
      83:	r4 = r8
      84:	call 6

In a bad one, we have

      76:	r3 = r8
      77:	r3 += r6
; 	void *data_end = skb_head + l4_off + len;
      78:	r8 += r7
; 	bpf_printk("%d %d", data, data_end);
      79:	r1 = 16 ll
      81:	r2 = 6
      82:	r4 = r8
      83:	call 6

In the pwru, the searching for registers is implemented inflexibly:

pwru/internal/libpcap/inject.go

Lines 126 to 145 in 80d7973

var (
dataReg asm.Register = 255
dataEndReg asm.Register = 255
)
for idx := injectIdx - 1; idx >= 0; idx-- {
inst := program.Instructions[idx]
if inst.OpCode.ALUOp() == asm.Mov {
if inst.Dst == asm.R3 {
dataReg = inst.Src
} else if inst.Dst == asm.R4 {
dataEndReg = inst.Src
}
}
if dataReg != 255 && dataEndReg != 255 {
break
}
}
if dataReg == 255 || dataEndReg == 255 {
return errors.New("Cannot find the data / data_end registers")
}

So pwru found r8 as both skb->data and skb->data_end, which explains error register r8 used twice

kdrag0n commented

v1.0.2 works for me on arm64. Thanks for the quick fix!

  • 🎉1
dakehero commented

v1.0.2 amd64 build also works for me

  • 🎉1