Set fielddata=true on [beat.hostname]

Question

Set fielddata=true on [beat.hostname]

Closed this issue 6 years ago · 8 comments

First, thanks for sharing this!

I followed the instructions, but I get this error :

Fielddata is disabled on text fields by default. Set fielddata=true on [beat.hostname] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.

Also, the dashboard looks like this, with shards failing :

Any idea of what I missed?

Note that the only thing I changed is that I defined this as output:

else if [postfix] {
    elasticsearch {
      hosts => localhost
      manage_template => false
      index => "%{[@metadata][beat]}-postfix-%{[@metadata][version]}-%{+yyyy.MM}-%{[host][name]}"
    }
  }

Also no that I'm using nginx and mysql plugins.

Answer 1 · 2019-01-28T02:16:52.000Z

Glad someone's using this!

Taking at a look at the error I wonder if maybe file beat has changed how they're handling host names. From the config(s) we're using beat.hostname but it sounds like maybe that field has changed or is different than the code in this repo is expecting.

To troubleshoot, I'd suggest deleting all resources created by this repo (anything with a name that includes "[Filebeat Postfix]"), check your Filebeat field names and configuration and then edit the files you're using from the repo to use the newer Filebeat field name(s) if they're different.

I might also check any customizations made to logstash or filebeat. Going through the Elastic forums it does look like other folks have had a similar issue in the past but the resolution doesn't seem clear: https://discuss.elastic.co/t/kibana-dashboard-error-set-fielddata-true-on-fieldname/82812/12

The changes you've made should be totally fine; I don't think this error is related to those.

Answer 2 · 2019-01-28T02:22:52.000Z

Maybe also helpful:

One common suggestion is to make sure you've gone through the full Filebeat setup including loading templates: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-template.html

Manually from the Filebeat :

# I think this is the path, not sure
cd /usr/share/filebeat

# Default template install
./filebeat setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

# Default dashboard setup
filebeat setup --dashboards

Answer 3 · 2019-01-28T09:38:00.000Z

Thanks a lot for the quick answer.
I did a few changes and it is now working properly.

I think the main issue were that my logstash config files were not in a proper order.

Also, looks like my index "%{[@metadata][beat]}-postfix-%{[@metadata][version]}-%{+yyyy.MM}-%{[host][name]}" were problematic. "%{[@metadata][beat]}-%{[@metadata][version]}-%{+yyyy.MM}-%{[host][name]}-postfix" fixed an issue.

The only not working visualization is "Message total" which say: Could not locate that index-pattern-field (id: postfix_queueid)

Any idea for this last issue ?

Answer 4 · 2019-01-28T09:42:15.000Z

Found it, just had to refresh the indexes...
Awesome dashboard by the way 👍

Answer 5 · 2019-01-30T17:55:04.000Z

@nicolasbadia thanks for following up with the fix! Glad it's working.

Answer 6 · 2019-05-20T11:32:42.000Z

Hello @nicolasbadia / @ActionScripted can you share you're configuration ?
I'm in the same issue...

Could not locate that index-pattern-field (id: postfix_queueid)
beat.hostname is empty

Answer 7 · 2020-07-02T06:21:02.000Z

Pls help me

Answer 8 · 2020-07-07T14:08:43.000Z

@phuongkhpt have you tried making the changes suggested in the red areas for fieldata=true? What version(s) of the Elastic components are you using?