Dashboard Visualization Erros

[sdresen]

Describe the bug
Fresh install using Docker. Visualizations in dashboards showing errors and not presenting data.

To Reproduce
Steps to reproduce the behavior:

1. Fresh install of Ubuntu 20.04
2. Fresh install of Docker
3. Fresh install of MaxMind
4. pfElk docker install script executed without errors
5. pfElk configuration followed in order without errors
   
   
   
   

Screenshots
If applicable, add screenshots to help explain your problem.

Operating System (please complete the following information):
Linux 5.4.0-58-generic x86_64
NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Docker version 20.10.1, build 831ebea
docker-compose version 1.25.0, build unknown

Elasticsearch, Logstash, Kibana (please complete the following information):

• Version of ELK: ELK_VERSION=7.10.0

**Service logs
es01.log
es02.log
es03.log
kibana.log
logstash.log

Additional context
Screenshots

** Last Note of Interest**
I am using pfSense and yet the Observer.Name field shows "OPNSense" in the Discover view where you see the specific data enrichment fields (Log Enrichment screenshot).

[a3ilson]

@sdresen - Thanks for providing a detailed request.

Let's break this down into two segments:

1. observer.name
2. Dashboards not populating (i.e. errors)

1. Observer.Name

• This field is defined within the 02-types.conf file along with observer.product and observer.serial_number
  • These fields are user defined
  • Amend to meet your needs (e.g. allow for delineation among multiple instances)
    • This is similar to the 07-interfaces.conf where you can define your network interfaces and aliases (referenced here

2. Other errors

• Similar issues were noted with #223 and #224 on the main pfelk repo...Still trying to identify the culprit and the only commonality is pfSense (I'm running OPNsense).
• Based on your provided screenshot, it appears that your logs are being parsed but the dashboards are reporting issues
  • Can you confirm the presence of pfelk-settings and pfelk-mappings-ecs templates are installed within Management>>Index Management>>Component Templates?
  • Try deleting all indices and after a few minutes see if that fixes the issue
  • Please provide a sample of the original.log messages (with and without the GROK failures)

[sdresen]

Thank you for the quick response. I think deleting the indices did much of the trick to get the reporting working. I updated the observer.product and observer.serial_number fields. I did confirm that the pfelk-settings and pfelk-mappings-ecs are present in the Component Templates section (screen shot attached).

The Firewall and DHCP visualizations appear to be working correctly. I'm still seeing "obj is undefined" for the Unbound dashboard.

Lastly, can you provide a bit more direction on the original.log message you'd like. Not clear what I should send.

[a3ilson]

@sdresen - Whew...glad that resolved most of the issue. I suspect that you had logs being sent prior to configuring (adding) the various templates if so the remedy is to purge the current indices which appears to have resolved your issue.

As for the unbound portion, are you running unbound (I assume you are)? However, the initial screenshot did not depict any indices (i.e. no received logs).

[sdresen]

The unbound issue may be that there's no DNS traffic hitting the pfSense unbound server. Unbound is active and running but I use PiHole's internally. So, could be simply that there's no traffic resolution hitting it.

[a3ilson]

That would be it! So looks like your up/running (e.g. everything good?).

[sdresen]

Yes, all good, many thanks!

[a3ilson]

Resolved