google/nftables

GetSetElements fails unexpected header type: got error, want unknown(2572)

Xniveres opened this issue · 4 comments

Xniveres commented

hey, i tried to get the size of a table, however GetSetElements returns an error:

code I've used:

tables, _ := clientNFT.ListTables()
	for _, i := range tables {
		if i.Name == "main" {
			t, err := clientNFT.GetSetByName(i, "blackhole")
			if err != nil {
				fmt.Println(err.Error())
				return 0
			}
			fmt.Println("found table")
			fmt.Println(t.Name)
			data, err := clientNFT.GetSetElements(t)
			if err != nil {
				fmt.Println(err)
				return 0
			}
			return len(data)
		}
	}

output:

found table
blackhole
unexpected header type: got error, want unknown(2572)

testing on go 1.18.3 on AlmaLinux 8.6

stapelberg commented

Unfortunately I don’t have any experience with sets (cc @sbezverk who contributed set support).

If you can make your configuration work with the nft tool, can you compare what nft does and what the nftables package does? For an example, see

nftables/nftables_test.go

Line 270 in 2eca001

func TestConfigureNAT(t *testing.T) {

That should hopefully make the problem relatively obvious.

turekt commented

Hi,

I have managed to investigate this issue a bit better and I succeeded in reproducing this error when using nftables lasting connection. Here is example code:

func test() {
    c, _ := nftables.New(nftables.AsLasting())
    filter := &nftables.Table{
        Family: nftables.TableFamilyIPv4,
        Name:   "filter",
    }   
    c.AddTable(filter)
    c.Flush()
    testSet := &nftables.Set{
        Table:   filter,
        Name:    "test_set",
        KeyType: nftables.TypeIFName,
    }   
    c.AddSet(testSet, nil)
    c.Flush()

    c.GetSetByName(filter, "test_set")
    if _, err := c.GetSetElements(testSet); err != nil {
        panic(err)
    }   

    elements := []nftables.SetElement{{Key: []byte("wg1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00")}}
    if err := c.SetAddElements(testSet, elements); err != nil {
        panic(err)
    }   
    c.Flush()
}

When I convert the first line to c, _ := nftables.New(), the issue no longer persists. It is also interesting to note that if any of the calls after first Flush is removed, the error is no longer there (therefore, to reproduce, one must have particular set of calls made).

I also managed to investigate a bit what is happening under the hood.

Running one of the test examples we see the following messages being received:

[{Header:{Length:0 Type:unknown(2569) Flags:request|acknowledge|0x400 Sequence:0 PID:0} Data:[2 0 0 0 11 0 1 0 102 105 108 116 101 114 0 0 13 0 2 0 116 101 115 116 95 115 101 116 0 0 0 0 8 0 3 0 0 0 0 0 8 0 4 0 0 0 0 41 8 0 5 0 0 0 0 16 8 0 10 0 0 0 0 1 10 0 13 0 0 4 1 0 0 0 0 0]}]
msg: {Header:{Length:92 Type:unknown(2569) Flags:0 Sequence:315476426 PID:7201} Data:[2 0 0 46 11 0 1 0 102 105 108 116 101 114 0 0 13 0 2 0 116 101 115 116 95 115 101 116 0 0 0 0 12 0 16 0 0 0 0 0 0 0 0 4 8 0 4 0 0 0 0 41 8 0 5 0 0 0 0 16 10 0 13 0 0 4 1 0 0 0 0 0 4 0 9 0]}
type: unknown(2569)
msg: {Header:{Length:36 Type:error Flags:0x100 Sequence:315476426 PID:7201} Data:[0 0 0 0 48 0 0 0 10 10 5 0 202 201 205 18 33 28 0 0]}
type: error
unexpected header type: got error, want unknown(2569)

The most important message being the NLM_F_CAPPED message:

msg: {Header:{Length:36 Type:error Flags:0x100 Sequence:315476426 PID:7201} Data:[0 0 0 0 48 0 0 0 10 10 5 0 202 201 205 18 33 28 0 0]}

The netlink error message that we get is not an actual error when observing the first 4 bytes in the Data section of the message indicating a success (https://github.com/mdlayher/netlink/blob/main/message.go#L145-L147, this is also confirmed by netlink man page which states that errno set to 0 is not an error). The message seen here only informs that the message is being capped (again, matched by the Flags field in the message struct https://git.netfilter.org/nftables/tree/include/netlink.h?id=187c6d01d35722618c2711bbc49262c286472c8f#n31) and I don't see this case being handled by the nftables go lib or the netlink library at the moment (see TODO https://github.com/mdlayher/netlink/blob/53a1c10065e51077659ceedf921c8f0807abe8c0/message.go#L299).

Additionally, I see identical response with strace when using nft cmdline tool:

recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{nlmsg_len=36, nlmsg_type=NLMSG_ERROR, nlmsg_flags=NLM_F_CAPPED, nlmsg_seq=0, nlmsg_pid=6378}, {error=0, msg={nlmsg_len=48, nlmsg_type=NFNL_SUBSYS_NFTABLES<<8|NFT_MSG_GETSET, nlmsg_flags=NLM_F_REQUEST|NLM_F_ACK, nlmsg_seq=0, nlmsg_pid=0}}], iov_len=69631}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 36

Therefore, this is not something out of the ordinary. I went further to debug send and receive calls (note: NL_DEBUG env set) and I see a lot of these NLM_F_CAPPED messages as part of netlink message exchange, e.g.:

EX: creating filter table done
nl: send msgs: {Header:{Length:20 Type:unknown(16) Flags:request Sequence:3852203812 PID:4969} Data:[0 0 0 10]}
nl: send msgs: {Header:{Length:40 Type:unknown(2560) Flags:request|acknowledge|0x400 Sequence:3852203813 PID:4969} Data:[2 0 0 0 11 0 1 0 102 105 108 116 101 114 0 0 8 0 2 0 0 0 0 0]}
nl: send msgs: {Header:{Length:20 Type:unknown(17) Flags:request Sequence:3852203814 PID:4969} Data:[0 0 0 10]}
nl: recv: {Header:{Length:36 Type:error Flags:0x100 Sequence:3852203813 PID:4969} Data:[0 0 0 0 40 0 0 0 0 10 5 4 37 247 155 229 105 19 0 0]}
EX: creating filter table done, flushed

Maybe this is just some form of an ack? I am not sure.

I have made fixes locally that perform conn.Receive once more if message type error is received with errno equal to 0 and added a test case for this particular scenario. I can create a PR with a fix for review and further discussion since I'm not really sure what this NLM_F_CAPPED message means and why it happens specifically in lasting connections. Other than that, maybe this can be fixed in conn.go directly rather than re-initiating recv in set.go. Until this is fixed, the quickest fix for this right now would be to use "normal" nftables connection instead of a lasting one.

Let me know what you think.

joewilliams commented

I am experiencing a similar problem with nftables.AsLasting(). Creating a counter object works fine but getting the object results in a similar error to the OP.

	counter, err := c.GetObj(myCounter)
	if err != nil {
		log.Fatalf("nftables.GetObj() failed: %v", err)
	}

2022/09/20 16:57:58 nftables.GetObj() failed: unexpected header type: got error, want unknown(2578)

Removing nftables.AsLasting() solves the error.

stapelberg commented

Removing nftables.AsLasting() solves the error.

That’s typically a hint that error handling or response handling is incorrect (too many or too few responses).