sflow/sflowtool

GRE/IPv6/TCP parsing seems to be incorrect

vovinacci opened this issue · 6 comments

vovinacci commented

Hi,

First of all thanks very much for such wonderful tool, it powers up our OVS based telemetry for a while and we absolutely love it!

We've used it with GRE/Ethernet/IPv6/TCP encapsulation and it was just perfect. Recently we've stripped Ethernet headers to have GRE/IPv6/TCP (we had a request to reduce amount headers a bit) and unfortunately it doesn't work anymore.

Sample JSON (IP addresses are obfuscated in the clear output):

{
  "sampleType_tag": "0:1",
  "sampleType": "FLOWSAMPLE",
  "sampleSequenceNo": "493",
  "sourceId": "2:1000",
  "meanSkipCount": "512",
  "samplePool": "252416",
  "dropEvents": "0",
  "inputPort": "0",
  "outputPort": "multiple 1",
  "elements": [
    {
      "flowBlock_tag": "0:1001",
      "extendedType": "SWITCH",
      "in_vlan": "0",
      "in_priority": "0",
      "out_vlan": "0",
      "out_priority": "0"
    },
    {
      "flowBlock_tag": "0:1",
      "flowSampleType": "HEADER",
      "headerProtocol": "1",
      "sampledPacketSize": "108",
      "strippedBytes": "4",
      "headerLen": "104",
      "headerBytes": "60-06-B7-F3-00-40-3A-40-FD-9D-3A-19-DE-FA-D6-A7-20-FE-71-DB-87-F2-E5-12-FD-9D-3A-19-DE-FA-B0-40-30-B9-F8-32-4E-5F-0C-8A-80-00-21-48-65-03-00-07-43-0B-D7-5D-00-00-00-00-E4-F5-03-00-00-00-00-00-10-11-12-13-14-15-16-17-18-19-1A-1B-1C-1D-1E-1F-20-21-22-23-24-25-26-27-28-29-2A-2B-2C-2D-2E-2F-30-31-32-33-34-35-36-37",
      "dstMAC": "6006b7f30040",
      "srcMAC": "3a40fd9d3a19"
    },
    {
      "flowBlock_tag": "0:1024",
      "flowSampleType": "tunnel_ipv4_in_IPV4",
      "tunnel_ipv4_in_sampledPacketSize": "0",
      "tunnel_ipv4_in_IPSize": "0",
      "tunnel_ipv4_in_srcIP": "15.145.185.129",
      "tunnel_ipv4_in_dstIP": "10.0.3.49",
      "tunnel_ipv4_in_IPProtocol": "47",
      "tunnel_ipv4_in_IPTOS": "0"
    }
  ]
}

So, it seems that sflowtool:

  • recognises proto 47 inside IPv4 and concludes that this is GRE - correct
  • then strips 4 bytes of GRE header - correct
  • then parses next two fields as Ethernet dst and src - not correct

Checking same flow with ovs-dpctl dump-flows shows everything correctly.

Is it possible to have such situation handled?

Thanks again for the great tool and ready to provide any input necessary.

sflow commented

sflowtool currently only has a rudimentary packet-decoder - it doesn't even look for a GRE header. That tunnel structure you see is an sFlow annotation, included by the OVS sFlow agent to indicate that the packet was or is going to be encapsulated in that way on the wire. So this may actually be a bug we should fix in OVS. Specifically, we may need to change the "headerProtocol" field from 1==SFLHEADER_ETHERNET_ISO8023 to something else (such as 11==SFLHEADER_IPv4) based on what is going with the forwarding rules inside OVS. Right now it is hard-coded here:

https://github.com/openvswitch/ovs/blob/master/ofproto/ofproto-dpif-sflow.c#L1317

But the hex "headerBytes" in your message doesn't look like an IPv4 header either. What is "60-06-B7-F3-" really the beginning of? Did you obfuscate there too? Are you in fact seeing headerBytes that start with 0x45? I appreciate that you may not be able to send raw captured packets, but to fix this we will need to know enough to reproduce it.

vovinacci commented

Thanks for your prompt reply!

The only obfuscated parts in the original message are tunnel_ipv4_in* fields.
With regards to headerBytes, it looks like IPv6 header to me, where:

"headerBytes":  "60-06-B7-F3-00-40-3A-40-FD-9D-3A-19-DE-FA-D6-A7-20-FE-71-DB-87-F2-E5-12-FD-9D-3A-19-DE-FA-B0-40-30-B9-F8-32-4E-5F-0C-8A-...",
                 ab bc cc cc dd dd ee ff +-------------------  src  -------------------+ +-------------------  dst  -------------------+
  • (a) 0x6 is Version (IPv6) (4 bits)
  • (b) 0x00 is Traffic Class (8 bits)
  • (c) 0x6B7F3 is Flow Label (20 bits)
  • (d) 0x0040 is Payload Length
  • (e) 0x3A is Next Header (58, ICMP for IPv6)
  • (f) 0x40 is Hop Limit (64, a.k.a. TTL)
    etc., etc.

Makes sense?

I'll try to gather raw captured packets for you as well this evening. Thanks again and have a great day!

sflow commented

Seeing raw packets will be helpful, but it seems clear that the headerProtocol field should have been set to SFLHEADER_IPv6, so we'll need to fix this in OVS.

Please send me what you did at the OpenFlow level to make this IPv6-over-GRE happen. For example, if you consider this sFlow tunneling test...

https://github.com/openvswitch/ovs/blob/master/tests/ofproto-dpif.at#L6734-L6834

can you see how to adjust that test setup to mirror what is happening in your system? That way we could supply a bugfix and a new test in the same pull-request, which will help it to be accepted promptly.

vovinacci commented

Thanks much for taking this forward, will do and get back once done.

vovinacci commented

Merry Christmas and a Happy New Year!

We found ourselves not that familiar with m4 to be sure that we wrote the test correctly, so if you don't mind - just plain commands that we've used. Would it be hard for you to wrap this to m4 as outlined in your link above?

  • Connection diagram:
Left host (172.17.0.48) <— L3 GRE tunnel —> OVS (172.17.0.22) <— L3 GRE tunnel —> Right host (172.17.0.23)
  • Left host config (L3 GRE tunnel)
ip tunnel add gre_ovs mode gre remote 172.17.0.22 local 172.17.0.48
ip -6 addr add 2001:0db8:0:f101::1/64 dev gre_ovs 
ip link set gre_ovs up
  • Right host config (L3 GRE tunnel)
ip tunnel add gre_ovs mode gre remote 172.17.0.22 local 172.17.0.23
ip -6 addr add 2001:0db8:0:f101::2/64 dev gre_ovs
ip link set gre_ovs up
  • OVS config
ovs-vsctl add-br br_test
ovs-vsctl set bridge br_test protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15

ovs-vsctl add-port br_test p1 -- set interface p1 type=gre options:remote_ip=172.17.0.48 options:local_ip=172.17.0.22 ofport_request=101 options:packet_type=legacy_l3
ovs-vsctl add-port br_test p2 -- set interface p2 type=gre options:remote_ip=172.17.0.23 options:local_ip=172.17.0.22 ofport_request=102 options:packet_type=legacy_l3
  • Add flows
ovs-ofctl add-flow br_test 'in_port=p1,priority=4000,actions=set_field:10:d7:b7:8f:00:00->eth_dst,move:in_port[0..15]->eth_dst[0..15],set_field:10:d7:b7:8f:00:01->eth_src,decap(),output:p2' -O Openflow15
ovs-ofctl add-flow br_test 'in_port=p2,priority=4000,actions=set_field:10:d7:b7:8f:00:00->eth_dst,move:in_port[0..15]->eth_dst[0..15],set_field:10:d7:b7:8f:00:01->eth_src,decap(),output:p1' -O Openflow15
  • Ping test from Left host to the Right host
ping -6 2001:0db8:0:f101::2
PING 2001:0db8:0:f101::2(2001:db8:0:f101::2) 56 data bytes
64 bytes from 2001:db8:0:f101::2: icmp_seq=1 ttl=64 time=1.41 ms
64 bytes from 2001:db8:0:f101::2: icmp_seq=2 ttl=64 time=1.00 ms
64 bytes from 2001:db8:0:f101::2: icmp_seq=3 ttl=64 time=1.09 ms
64 bytes from 2001:db8:0:f101::2: icmp_seq=4 ttl=64 time=1.07 ms
...
  • Check flows on OVS with ovs-dpctl dump-flows
recirc_id(0),tunnel(src=172.17.0.48,dst=172.17.0.22,flags(-df-csum)),in_port(3),eth_type(0x86dd),ipv6(tclass=0/0x3,frag=no), packets:8, bytes:832, used:0.424s, actions:sample(sample=0.2%,actions(userspace(pid=3663069814,sFlow(vid=0,pcp=0,output=2147483649),actions))),push_eth(src=10:d7:b7:8f:00:01,dst=10:d7:b7:8f:00:65),set(tunnel(src=172.17.0.22,dst=172.17.0.23,ttl=64,flags(df))),pop_eth,3
recirc_id(0),tunnel(src=172.17.0.23,dst=172.17.0.22,flags(-df-csum)),in_port(3),eth_type(0x86dd),ipv6(tclass=0/0x3,frag=no), packets:8, bytes:832, used:0.424s, actions:sample(sample=0.2%,actions(userspace(pid=3663069814,sFlow(vid=0,pcp=0,output=2147483649),actions))),push_eth(src=10:d7:b7:8f:00:01,dst=10:d7:b7:8f:00:0f),set(tunnel(src=172.17.0.22,dst=172.17.0.48,ttl=64,flags(df))),pop_eth,3

recirc_id(0),tunnel(src=172.17.0.48,dst=172.17.0.22,flags(-df-csum)),in_port(3),eth_type(0x86dd),ipv6(tclass=0/0x3,frag=no), packets:47, bytes:4888, used:0.593s, actions:sample(sample=0.2%,actions(userspace(pid=3663069814,sFlow(vid=0,pcp=0,output=2147483649),actions))),push_eth(src=10:d7:b7:8f:00:01,dst=10:d7:b7:8f:00:65),set(tunnel(src=172.17.0.22,dst=172.17.0.23,ttl=64,flags(df))),pop_eth,3
recirc_id(0),tunnel(src=172.17.0.23,dst=172.17.0.22,flags(-df-csum)),in_port(3),eth_type(0x86dd),ipv6(tclass=0/0x3,frag=no), packets:47, bytes:4888, used:0.593s, actions:sample(sample=0.2%,actions(userspace(pid=3663069814,sFlow(vid=0,pcp=0,output=2147483649),actions))),push_eth(src=10:d7:b7:8f:00:01,dst=10:d7:b7:8f:00:0f),set(tunnel(src=172.17.0.22,dst=172.17.0.48,ttl=64,flags(df))),pop_eth,3
vovinacci commented

Small update. This is not required anymore, so issue could be closed if not further actions are planned. Thanks!

P.S. Also many thanks for your help with troubleshooting.