RIverbed Netflow support (Was: Can’t yet decode flow set id)
venki522 opened this issue · 16 comments
- Version:6.3
- Operating System:red hat7.1
- Config File (if you have sensitive info, please remove it):
- Sample Data:
[2018-07-03T07:17:16,724][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 300 from source id 3203342338, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2018-07-03T07:17:16,724][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 300 from source id 3203342338, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2018-07-03T07:17:16,724][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 300 from source id 3203342338, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2018-07-03T07:17:16,724][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 302 from source id 3203342338, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2018-07-03T07:17:16,724][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 301 from source id 3203342338, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2018-07-03T07:17:16,724][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 300 from source id 3203342338, because no template to decode it with has been received. This message will usually go away after 1 minute.
- Steps to Reproduce:
We r running elastiflow on elk 6.3 and the logstash-plain.log has all the above messages and it is filling our disk space..could you help us to figure out what is the issue and why we are seeing these messages
Can you check the changelog to see if perhaps this was already fixed in a newer version of the codec?
If not: can you post a pcap of your netflow traffic so I can look at it?
@jorritfolmer I have the PCAP in question here. It looks like there are unknown fields...
I will check all of the templates in the PCAP and workout what needs to be added to netflow.yml.
These are the IDs that need to be identified, and what I could find online (lengths are from templates).
100 - DEPRECATED (was className) - 1 Byte
102 - layer2packetSectionOffset - 4 Bytes
103 - layer2packetSectionSize - 4 Bytes
104 - layer2packetSectionData - 2 Bytes
105 - Reserved for future use by cisco - 2 Bytes
106 - Reserved for future use by cisco - 4 Bytes
107 - Reserved for future use by cisco - 2 Bytes
108 - Reserved for future use by cisco - 4 Bytes
109 - Reserved for future use by cisco - 4 Bytes
110 - Reserved for future use by cisco - 4 Bytes
111 - Reserved for future use by cisco - 1 Byte
51130 - I couldn't find anything for this ID - 1 Byte
@jorritfolmer if you have any info on these fields it would be helpful. I assume you don't or it would already be in the codec, but I wanted to double check.
BTW, as I have researched this, I also searched through the implementations of other netflow projects here on GitHub, and I am convinced that the logstash netflow codec has the most comprehensive support for various vendor-specific fields.
I did find that nfdump had a few NProbe fields that aren't supported here. I was going to submit a PR to include them, but I noticed that they don't seem consistent with the NProbe docs, so I will need to do some testing first.
Other than that one exception, there is nothing supported by other projects (regardless of language) that isn't supported here. Great work @jorritfolmer !
I figured out these are from Riverbed Cascade Sensor. More info when I get it.
Hi James..
If u see in the above logs the we don’t see the IP address of source for which template is not getting decoded..is it possible for future release to add that feature where we can see the IP address if source in the log
Current status as I understand it:
- Rob has a pcap (needed for our rspec tests)
- but we lack documentation to properly name the Netflow fields
Actually I now have the flows decoded properly by adding the needed fields to netflow.yml.
The question I have is whether you want to include them in the codec, as they overlap with an area that is used by other fields, or is "reserved"? I just worry about conflicts here. Of course this is alway a challenge with Netflow, which IPFIX handles with enterprises.
Because of concerns related to conflicting IDs, I have added a new directory in ElastiFlow that contains an alternate field definition files, which the user can optionally use by uncommenting a line in the input. The commit with all of the ElastiFlow changes is here: robcowart/elastiflow@1399df8
This is what I have added to netflow.yml:
100:
- :uint8
- :rb_passthru_reason
101:
- :uint8
- :rb_wan_visibility
102:
- :ip4_addr
- :rb_cfe_inpath_addr
103:
- :ip4_addr
- :rb_sfe_inpath_addr
104:
- :uint16
- :rb_cfe_tcp_port
105:
- :uint16
- :rb_sfe_tcp_port
106:
- :ip4_addr
- :rb_outer_inpath_addr
107:
- :uint16
- :rb_outer_tcp_port
108:
- :uint32
- :rb_retrans_pkts
109:
- :uint32
- :rb_retrans_bytes
110:
- :uint32
- :rb_tcp_conn_rtt
111:
- :uint8
- :rb_fe_type
51130:
- :uint8
- :rb_unk_51130
Note the name of that last one. It still remains a mystery, but at least it is generically handled this way.
If you are not concerned about the potential ID conflicts, I can submit a PR with the needed changes.
I like your definitions directory solution!
For the time being I'll mark Riverbed unsupported in the codec documentation, pointing to Elastiflow for a workaround.
Docs updated
Hello @jorritfolmer, I'm facing some issue with a Riverbed appliance (a WAN Optimizer) sending a template as the following:
FlowSet 5 [id=0] (Data Template): 304
FlowSet Id: Data Template (V9) (0)
FlowSet Length: 116
Template (Id = 304, Count = 27)
Template Id: 304
Field Count: 27
Field (1/27): IP_SRC_ADDR
Field (2/27): IP_DST_ADDR
Field (3/27): IP_NEXT_HOP
Field (4/27): PKTS
Field (5/27): BYTES
Field (6/27): FIRST_SWITCHED
Field (7/27): LAST_SWITCHED
Field (8/27): L4_SRC_PORT
Field (9/27): L4_DST_PORT
Field (10/27): INPUT_SNMP
Field (11/27): OUTPUT_SNMP
Field (12/27): TCP_FLAGS
Field (13/27): PROTOCOL
Field (14/27): IP_TOS
Field (15/27): DIRECTION
Field (16/27): SRC_VLAN
Field (17/27): IP TTL MINIMUM
Field (18/27): IP TTL MAXIMUM
Field (19/27): Unknown(51103)
Type: Unknown (51103)
Length: 4
Field (20/27): Unknown(51102)
Type: Unknown (51102)
Length: 4
Field (21/27): Unknown(51105)
Type: Unknown (51105)
Length: 2
Field (22/27): Unknown(51104)
Type: Unknown (51104)
Length: 2
Field (23/27): Unknown(51110)
Type: Unknown (51110)
Length: 4
Field (24/27): Unknown(51108)
Type: Unknown (51108)
Length: 4
Field (25/27): Unknown(51109)
Type: Unknown (51109)
Length: 4
Field (26/27): Unknown(51111)
Type: Unknown (51111)
Length: 1
Field (27/27): Unknown(51101)
Type: Unknown (51101)
Length: 1
Do we expect to add those definitions in the codec or we should apply the Elastiflow workaround?
Best regards,
Luca
There are a number of issues with the Riverbed Netflow:
- they use fields that are already taken or reserved (id 100-110)
- they use fields that we lack field names for (51102-51111 in your wireshark )
If you can provide documentation or other clues as to how we should name the 51102-51111 fields, I'd be happy to include them in the codec.
Any update on the fields (51102-51111)?
Any update on the fields (51102-51111)?
Hi Jorrit,
I think I have the same problem with my Riverbed Steelhead appliances. I took at look at the Riverbed knowlege base, and found this:
| Type | Description | Length | Values |
|---|---|---|---|
| 51100 | Pass-through reason | 1 | Unknown |
| 51101 | WAN visibility | 1 | 0: correct addressing1: port transparency2: full transparency |
| 51102 | CFE in-path IP address | 4 | |
| 51103 | SFE in-path IP address | 4 | |
| 51104 | CFE TCP port | 2 | |
| 51105 | SFE TCP port | 2 | Typically 7800,or7810 |
| 51106 | Outer in-path IP address | 4 | |
| 51107 | Outer TCP port number | 2 | |
| 51108 | Retransmitted packets | 4 | 0 or greater |
| 51109 | Retransmitted bytes | 4 | 0 or greater |
| 51110 | TCP connection RTT | 4 | |
| 51111 | FE_TYPE | 1 | 1: CFE, 2: SFE |
Now a little explanation for non-Riverbed geeks.
- SFE = server file engine (aka server-side Steelhead)
- CFE = client file engine (aka client-side Steelhead)
- Riverbeds NAT optimized traffic so when they talk about "outer" they are referring to the traffic "as-is" over the WAN (natted), typically SFE and CFE inpath-IP address.
- Inner is the "original" client/server IP addresses
- TCP connection RTT, is the round-trip-time between the two Riverbeds
Let me know if there are questions, and I will try and see if I can answer them..
Ps. Im pretty new to logstash and elastic search.. So, any pointers on how I can make this work with my installation is much appreciated.. :-)
Esben