ZachChristensen28/TA-pihole_dns

"blocked" not extracting in Data Model

Closed this issue · 4 comments

Data Model preview does not show query_type:
image

Current regex will only extract "cached" and "reply" responses.
"gravity blocked" does not get captured in current regex.
This seems to work: https://regex101.com/r/vSBWp1/2/

[pihole_response]
#REGEX  = [[pihole_default]]([^vd][^o]\S+)\s+(\S+)\s+is\s+(.+)
REGEX = [[pihole_default]]\s(\S+|gravity blocked)\s+(\S+)\s+is\s+(\S+)
FORMAT = message_type::Response name::$1 query::$2 answer::$3

refer to #14 for correct pihole_default regex

Just updated my HF and restarted it. Will report back the results

@ZachChristensen28
Here is the specific report that gets no results:

| tstats prestats=t count from datamodel=Network_Resolution.DNS where sourcetype=pihole DNS.message_type=response DNS.reply_code="Query Refused" by DNS.src, DNS.query
| rename DNS.* as *
| `pihole_enrich_hostname(src)`
| eval Host=coalesce(Hostname,src)
| chart limit=0 count over query by Host
| addtotals
| sort 10 -Total
| fields - Total
| untable query src count
| xyseries src query count
| addtotals
| sort 10 -Total
| fields - Total
| untable src query count
| xyseries query src count

breaks at message_type and reply_code

Pasting same comment for each related issue:

Please check if log-queries=extra in /etc/dnsmasq.d/01-pihole.conf. If not, this could be the root cause of the issue. Anytime Pi-hole updates, it overwrites this file with updates. So if you have set this setting in the past, it would be erased during an update.

To fix, add log-queries=extra in /etc/dnsmasq.d/01-pihole.conf or follow the instructions at the top of the file and create a new file in the same directory with this change. Note: creating your own file to override this setting may cause issues during an update. removing the file and then re-adding it after the update will fix the issue.

Make sure to run pihole restardns after updating settings so the changes take affect.

Looks like that is the problem...

It reverted back to the default setting.

Changed it to log-queries=extra, commented out my new regex, enabled the default ones and ran a real-time search in Splunk and saw the logs change to the "extra" format.

I am rebuilding the Data Model Acceleration now and will report back

[edit] DMA is working and all Pihole reports now populate.

As a suggestion, can you append to the README.md to note that updates to Pihole will/can revert the log-queries=extra back to default?

I didn't realize this would happen after the update and other most likely will not either.

I am working on a tutorial video for r/pihole and r/splunk that I want to publish on Youtube and I will not that in the troubleshooting steps.