endgameinc/eqllib

Source file parse doesn't ignore '.'

CptOfEvilMinions opened this issue · 3 comments

CptOfEvilMinions commented

I created a custom source file to parse BRO logs. By default BRO has key names containing dots like id.orig_h or id.resp_h. When I do the following destination_address = 'id.orig_h' and run eqllib it ignores this mapping. However, if I manually change id.orig_h to dest_addr in the JSON log file and change my source file statement to destination_address = 'dest_addr' it works.

bro-source.toml

name = "Bro events"
strict = true
domain = "bro-domain"
filter_query = true

[timestamp]
field = "ts"
format = "%Y-%m-%d %H:%M:%S.%f"

[fields.mapping]
ts = "ts"
uid = "uid"
destination_address = 'id.orig_h'

[events.bro_conn]
filter = "conn_state"

[events.bro_conn.mapping]
proto = 'proto'
conn_state = 'conn_state'
local_orig = 'local_orig'
local_resp = 'local_resp'

bro-domain.toml

name = "bro-domain"
fields = [
  # Common Fields
  "ts",
  "uid",
  "destination_address"
]

[events.bro_conn]
fields = [
  "proto",
  "conn_state",
  "local_orig",
  "local_resp",
  "missed_bytes"
]
rw-access commented

hey @CptOfEvilMinions I think we've talked about this indirectly in other places but I haven't responded here. With EQL a . indicates a nested field, but I think with the bro logs the native fields were named "ip.orig_h" and "id.resp_h", which caused the problems.

This is an interesting scenario, because we currently require all field names to match a-zA-Z][a-zA-Z0-9_]*
https://github.com/endgameinc/eql/blob/aa55970fd57996aed7519a8eda94c3fe472d15c2/eql/etc/eql.ebnf#L231

Since . already means something in EQL, there are a few ways we could do this:

  1. One option is to escape all characters that don't match that regex. id\.orig_h.
  2. Another option is to use the string syntax and do something like this ["id.orig_h"].

Then your EQL queries would look like one of these

network where id\.orig_h == "192.168.1.1"
network where ["id.orig_h"] == "192.168.1.1"
network where .["id.orig_h"] == "192.168.1.1"

Also since your blog, it should be a lot easier to make your own schema, and EQL will autodetect it from your JSON file if you use the new interactive shell

Any preferences for the syntax?