logstash-plugins/logstash-patterns-core

COMBINEDAPACHELOG leading to grokparsefailures due to Email as Username

bne86 opened this issue · 0 comments

bne86 commented

Hi,
i have issues with our ELK stack analyzing some httpd logs. For users that are authenticated with their email address (see sample data) the filter does not work. Some info on the environment:

  • Version: all 5.2
  • Operating System: Docker containers running on CentOS 7
  • Config File:
   mutate { replace => { type => "apache_access" } }
   grok {
      match => [
      "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}",
      "message" , "%{COMBINEDAPACHELOG}",
      "message" , "%{COMMONAPACHELOG}+%{GREEDYDATA:extra_fields}"
      ]
      overwrite => [ "message" ]
   }
   geoip {
      source => "clientip"
      target => "geoip"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => ["response", "integer"]
      convert => ["bytes", "integer"]
      convert => ["responsetime", "float"]
      convert => [ "[geoip][coordinates]", "float"]
   }
   useragent {
      source => "agent"
   }
  • Sample Data:
    Working sample data:
    123.123.123.123 - bvonstvieth [08/Feb/2017:14:32:15 +0100] "PROPFIND /remote.php/webdav/ HTTP/1.1" 207 408 "-" "Mozilla/5.0 (Macintosh) mirall/2.2.4 (build 3709)"
    Not working data leading to _grokparsefailure:
    123.123.123.123 - b.von.st.vieth@example.com [08/Feb/2017:13:50:14 +0100] "PROPFIND /remote.php/webdav/ HTTP/1.1" 207 389 "-" "Mozilla/5.0 (Macintosh) mirall/2.2.4 (build 3709)"
  • Steps to Reproduce:
    Put logstash on /var/log/httpd and see what happens.

Taking a look at logstash-patterns-core and copying it to my grok filter makes it working, but shouldn't it be already in place?
match => [ "message" , "%{IPORHOST:clientip} %{HTTPDUSER:ident} %{HTTPDUSER:auth} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-)" ]