No matching template and Unsupported field type.

Question

No matching template and Unsupported field type.

zhaohangbo opened this issue 9 years ago · 15 comments

My td-agent.log when receiving netflow data with fluent-plugin-netflow

2016-03-10 06:30:11 -0800 [warn]: No matching template for flow id 263
2016-03-10 06:31:12 -0800 [warn]: No matching template for flow id 263
2016-03-10 06:31:12 -0800 [warn]: No matching template for flow id 256
2016-03-10 06:31:12 -0800 [warn]: No matching template for flow id 256
2016-03-10 06:31:35 -0800 [warn]: Unsupported field type=148 length=4
2016-03-10 06:31:35 -0800 [warn]: Unsupported field type=148 length=4
2016-03-10 06:31:35 -0800 [warn]: Unsupported field type=176 length=1
2016-03-10 06:31:35 -0800 [warn]: Unsupported field type=176 length=1

Can you please give me some help ?

What's wrong and how to figure it out ?

Find similar issues in Logstash discussion:

It's because data template includes a field that Logstash doesn't understand.

elastic/logstash#2002
https://groups.google.com/forum/#!topic/logstash-users/rvBIYZwWY-c
https://groups.google.com/forum/#!topic/logstash-users/hicR5rugjiQ

Answer 1 · 2016-03-14T22:40:52.000Z

Using logstash-plugins/logstash-codec-netflow#12 (comment) approach doesn't work?

Answer 2 · 2016-03-15T00:35:58.000Z

Is that mean I need to configure the Network Device by myself ?

Either define that field type with a suitable data type and field name in Network Device:

225:

:uint32
:name_of_field

Or mark it to be skipped in Network Device:

225:

:skip

Answer 3 · 2016-03-15T01:10:17.000Z

Use definition parameter with your rules.

<source>
  @type netflow
  #...
  definitions /path/to/your_netflow_def.yml
</source>

Answer 4 · 2016-03-18T04:55:26.000Z

Thank you very much for your quick reply and help.

I've tried many times to add different skip fields in definitions /some/path/to/my_netflow_def.yml
like,

148:
- :skip
176:
- :skip
177:
- :skip
.......
.......
.......

But there are always new "Unsupported field type" coming out in the fluentd log.
So if there is a way that
I can get the netflow data first then I can generate a corresponding netflow_def.yml without having access to Network Devices.

Answer 5 · 2016-03-18T05:14:24.000Z

Hi

But there are always new "Unsupported field type" coming out in the fluentd log.

Does it keep showing for a long time? Or just a few seconds after you reboot fluentd?
It's an expected behavior to report "Unsupported field type" until this plugin learns about netflow v9 template. (Network device usually sends template flowset periodically, which is something like data schema)

So if there is a way that
I can get the netflow data first then I can generate a corresponding netflow_def.yml without having access to Network Devices.

I hope tshark -V on your collector is going to help you.

Answer 6 · 2016-03-18T05:14:46.000Z

And still having the 2 warnings:
No matching template for flow id 263
No matching template for flow id 256

Answer 7 · 2016-03-18T05:21:00.000Z

@codeout Thanks,

but what tshark captured is binary,
I am using

 tshark -f "udp port 5140" -i any -V

Am I using it in the wrong way ?

And attached is my current log warnings.

Answer 8 · 2016-03-18T05:42:57.000Z

You may want to decode by -d option like:

tshark -f "udp port 5140" -i any -V -d "udp.port==5140,cflow"

Answer 9 · 2016-03-18T06:53:21.000Z

Thanks a lot !! The command works.
Several minutes later I can see the content of netflow on my terminal.
Although at the beginning, still can't find templates 256,263, since template may not cached.

But in the td-agent.log ,still No matching template for flow id 256, 263 all the time.

Bellowing is my status:

Answer 10 · 2016-03-18T07:23:20.000Z

But in the td-agent.log ,still No matching template for flow id 256, 263 all the time.

So, that probably means some exporters keep sending data flowsets without preceding templates.
If you look into the result of tshark, you can find which devices are doing this.

Answer 11 · 2016-03-18T23:33:24.000Z

I can capture the netflow V9 templates and data flowsets successfully via Tshark.
And Templates flowsets will be sent every 30 mins.

But why it keeps reporting warnings after td-agent runs more than 30 mins.
And I can't I write them out to file via Td-agent ?

Can you please help me ?

My td-agent.conf is as following:

<source>
  type netflow
  tag netflow.event
  bind 0.0.0.0
  port 5140
  cache_ttl 6000 # time_to_live
  versions [5, 9]
  definitions /etc/td-agent/netflow_def.yml
</source>

<match netflow.event.** >
   type copy
   <store>
        type file
        path /var/log/td-agent/netflow-data-test/netflow.data
   </store>
   <store>
        type stdout  # also no out put.
   </store>
</match>

Answer 12 · 2016-03-18T23:45:38.000Z

In the method of "def handle_v9_flowset_template(flowset, record)" (Line 200 )in parser_netflow.rb

I added 2 lines of log.info code( line 212 -- 214) as following.
Then build and reinstall the fluent-plugin-netflow.
But I can not find any output info in td-agent.log .

def handle_v9_flowset_template(flowset, record)
    ......
    ......
    @templates[key, @cache_ttl] = BinData::Struct.new(endian: :big, fields: fields)
    $log.info("cache_ttl is #{@cache_ttl}")
    $log.info("added template,flowset.source_id|template.template_id is #{key}")
    @templates.cleanup!

So I wonder if there's anything wrong with the
@templates[key, @cache_ttl] = BinData::Struct.new(endian: :big, fields: fields)
Can you give me some suggestions ?

Answer 13 · 2016-03-19T05:20:15.000Z

I'm guessing that netflow_field_for(field.field_type, field.field_length, category) returns nil for some fields of your template flowset. All field should be parseable by netflow_field_for to register the template.

I'd recommend that you narrow down which field is problematic on your template. If netflow_field_for returns something for every field, then BinData::Struct.new could be wrong as you pointed out.

      def handle_v9_flowset_options_template(flowset, record)
        ...
            NETFLOW_V9_FIELD_CATEGORIES.each do |category|
              template["#{category}_fields"].each do |field|
                entry = netflow_field_for(field.field_type, field.field_length, category)

                throw :field unless entry  # template won't be registered if this line happens

                fields += entry
              end
            end

BTW, I've also noticed we have a few things to fix in this netflow v9 parser and am addressing one of them in #12 (I'm also preparing another fix on my branch)

Answer 14 · 2016-03-24T09:08:18.000Z

@zhaohangbo Could you try #14 patch for your case?

Answer 15 · 2016-05-03T13:47:21.000Z

I also have issues with v9, but I believe I found the cause (although I'm no expert at Ruby/YAML so the exact solution may require someone else's involvement).
I've added debug code (a.k.a. print) in the parser_netflow.rb file. It appears that the skip function called in the netflow_field_for method has an issue (line 312); this seems to be linked with the bindata package... maybe something has changed in that package that caused failure.

I've setup fluentd with the netflow plugin a few weeks ago to capture netflow traffic from Cisco equipment. The router and switches send information in v5 which is not causing issues.
However the firewalls (Cisco ASA) are sending information in v9 format and I realized a few days ago that they were being received but not imported (but packet capture proved that they were sent, and templates were sent every minute).
I've upgrade to 0.2.2 of the netflow plugin.

During a certain amount of time as define in your netflow config in your device you will receive the following types of messages:
2016-04-21 12:52:47 -0400 [warn]: No matching template for host="10.X.Y.Z" source_id=0 flowset_id=263
(for many flowset_id)

The v9 format sends a template every few minutes as defined in your netflow config on your device (we have it set to 1, which means one minute). Upon starting, and until the template definition are sent, the logs will be dropped. Once you receive template information the handle_v9_flowset_template method is called in parser_netflow.rb.

If you check in your /var/log/td-agent/td-agent.log file, you should likely see the following exception and warning messages:
[warn]: unexpected error on parsing data=""\x00" (shortened) error_class=ArgumentError error="BinData::Skip requires either :length or :to_abs_offset"
[warn]: Unsupported field type=148 length=4
(for many field types)

A temporary fix is to add all the different "Unsupported field" to the YAML file like:
85:

4
skip85
148:
4
skip148

There are many more fields in v9 than v5 (looking at the wireshark code for their netflow parser).
I'm trying to find add those fields to the YAML (although I haven't found the exact definition of the format)

tl/dr;

There is a bug in the call of the skip function of the bindata package on line 312 where the length appears not to be sent and is marked as mandatory by this package (return [:skip, nil, {length: length}]); I'd need you to investigate why that is (I'm an experienced developer, but new at Ruby)
A temporary fix is to update the netflow_fields.yaml file, but this means changes would be required for new v9 templates (and restart the service afterwards)
There are many v9 fields that exist (and may be helpful) defined in the wireshark code that I plan to import to enrich the v9 template once I can confirm their length or type of data (IPv4_Address, etc.)