caddy-logs parser not working, but apache2-logs parser works
Closed this issue · 4 comments
jzemla commented
Hello,
Environment:
crowdsec v1.2.1 (docker)
caddy v2.4.6 (docker xcaddy build includes: caddy-l4 , format-encoder , realip , caddy2-proxyprotocol , caddy-crowdsec-bouncer/http@main , caddy-crowdsec-bouncer/layer4@main)
I'm having trouble getting this to parse my caddy access.log. I am using the suggested config from the example, but crowdsec is unable to parse the file. I apologize in advance for being a github/devops newbie -- if there is something I missed or can provide more insight into, please let me know!
Caddy - config.json:
"logging": {
"logs": {
"default": {
"level": "DEBUG",
"writer": {
"output": "stderr"
}
},
"access": {
"level": "DEBUG",
"writer": {
"output": "file",
"filename": "/var/log/caddy/access.log"
},
"encoder": {
"format": "formatted",
"template": "{common_log} \"{request>headers>Referer>[0]}\" \"{request>headers>User-Agent>[0]}\""
},
"include": [
"http.log.access.access"
]
}
}
},
Failed grok parse via caddy-logs:
# crowdsec -dsn file:///var/log/caddy/access.log -type caddy -no-api -trace
---snip---
TRAC[15-12-2021 11:58:22] INPUT '176.53.221.38 - - [15/Dec/2021:17:39:52 +0000] "GET / HTTP/1.1" 308 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36"'
TRAC[15-12-2021 11:58:22] node stage : s00-raw, current stage : s00-raw
TRAC[15-12-2021 11:58:22] Processing node 0/12 -> sparkling-waterfall node-name=sparkling-waterfall stage=s00-raw
TRAC[15-12-2021 11:58:22] Event entering node id=sparkling-waterfall name=crowdsecurity/docker-logs stage=s00-raw
DEBU[15-12-2021 11:58:22] Event leaving node : ko (failed filter) id=sparkling-waterfall name=crowdsecurity/docker-logs stage=s00-raw
TRAC[15-12-2021 11:58:22] node (sparkling-waterfall) ret : false node-name=sparkling-waterfall stage=s00-raw
TRAC[15-12-2021 11:58:22] Processing node 1/12 -> dawn-feather node-name=dawn-feather stage=s00-raw
TRAC[15-12-2021 11:58:22] Event entering node id=dawn-feather name=crowdsecurity/syslog-logs stage=s00-raw
DEBU[15-12-2021 11:58:22] Event leaving node : ko (failed filter) id=dawn-feather name=crowdsecurity/syslog-logs stage=s00-raw
TRAC[15-12-2021 11:58:22] node (dawn-feather) ret : false node-name=dawn-feather stage=s00-raw
TRAC[15-12-2021 11:58:22] Processing node 2/12 -> little-hill node-name=little-hill stage=s00-raw
TRAC[15-12-2021 11:58:22] Event entering node id=little-hill name=crowdsecurity/non-syslog stage=s00-raw
TRAC[15-12-2021 11:58:22] ! No grok pattern : 0x0 id=little-hill name=crowdsecurity/non-syslog stage=s00-raw
TRAC[15-12-2021 11:58:22] State after nodes : true id=little-hill name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:58:22] + Processing 4 statics id=little-hill name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:58:22] .Parsed[message] = '176.53.221.38 - - [15/Dec/2021:17:39:52 +0000] "GET / HTTP/1.1" 308 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36"' id=little-hill name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:58:22] .Parsed[program] = 'caddy' id=little-hill name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:58:22] .Meta[datasource_path] = '/var/log/caddy/test.log' id=little-hill name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:58:22] .Meta[datasource_type] = 'file' id=little-hill name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:58:22] Event leaving node : ok id=little-hill name=crowdsecurity/non-syslog stage=s00-raw
TRAC[15-12-2021 11:58:22] node is successful, check strategy
DEBU[15-12-2021 11:58:22] move Event from stage s00-raw to s01-parse id=little-hill name=crowdsecurity/non-syslog stage=s00-raw
TRAC[15-12-2021 11:58:22] Node successful, continue id=little-hill name=crowdsecurity/non-syslog stage=s00-raw
TRAC[15-12-2021 11:58:22] node (little-hill) ret : true node-name=little-hill stage=s00-raw
DEBU[15-12-2021 11:58:22] node successful, stop end stage s00-raw node-name=little-hill stage=s00-raw
TRAC[15-12-2021 11:58:22] node stage : s01-parse, current stage : s01-parse
TRAC[15-12-2021 11:58:22] Processing node 3/12 -> spring-water node-name=spring-water stage=s01-parse
TRAC[15-12-2021 11:58:22] Event entering node id=spring-water name=crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:58:22] Event leaving node : ko (failed filter) id=spring-water name=crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] node (spring-water) ret : false node-name=spring-water stage=s01-parse
TRAC[15-12-2021 11:58:22] Processing node 4/12 -> patient-pond node-name=patient-pond stage=s01-parse
TRAC[15-12-2021 11:58:22] Event entering node id=patient-pond name=crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] ! No grok pattern : 0x0 id=patient-pond name=crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] Event entering node id=solitary-snow name=child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] Node has not filter, enter id=solitary-snow name=child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] ! No grok pattern : 0x0 id=solitary-snow name=child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] Event entering node id=ancient-shape name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] Node has not filter, enter id=ancient-shape name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] Processing grok pattern : : 0xc0006a8f70 id=ancient-shape name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] extract path [common_log]
DEBU[15-12-2021 11:58:22] [common_log] doesn't exist
DEBU[15-12-2021 11:58:22] + Grok '%{NOT...' didn't return data on '' id=ancient-shape name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] State after nodes : false id=ancient-shape name=child-child-crowdsecurity/caddy-logs stage=s01-parse
DEBU[15-12-2021 11:58:22] Event leaving node : ko id=ancient-shape name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] sub-node (ancient-shape) ret : false (strategy:) id=solitary-snow name=child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] Event entering node id=black-dream name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] Node has not filter, enter id=black-dream name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] Processing grok pattern : : 0xc0004251f0 id=black-dream name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] extract path [request remote_addr]
DEBU[15-12-2021 11:58:22] [request remote_addr] doesn't exist
DEBU[15-12-2021 11:58:22] + Grok '%{IPO...' didn't return data on '' id=black-dream name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] State after nodes : false id=black-dream name=child-child-crowdsecurity/caddy-logs stage=s01-parse
DEBU[15-12-2021 11:58:22] Event leaving node : ko id=black-dream name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] sub-node (black-dream) ret : false (strategy:) id=solitary-snow name=child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] Event entering node id=wispy-dew name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] Node has not filter, enter id=wispy-dew name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] Processing grok pattern : : 0xc000425840 id=wispy-dew name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] extract path [request headers User-Agent]
DEBU[15-12-2021 11:58:22] [request headers User-Agent] doesn't exist
DEBU[15-12-2021 11:58:22] + Grok '\["%{...' didn't return data on '' id=wispy-dew name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] State after nodes : false id=wispy-dew name=child-child-crowdsecurity/caddy-logs stage=s01-parse
DEBU[15-12-2021 11:58:22] Event leaving node : ko id=wispy-dew name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] sub-node (wispy-dew) ret : false (strategy:) id=solitary-snow name=child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] State after nodes : false id=solitary-snow name=child-crowdsecurity/caddy-logs stage=s01-parse
DEBU[15-12-2021 11:58:22] Event leaving node : ko id=solitary-snow name=child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] sub-node (solitary-snow) ret : false (strategy:next_stage) id=patient-pond name=crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] State after nodes : false id=patient-pond name=crowdsecurity/caddy-logs stage=s01-parse
DEBU[15-12-2021 11:58:22] Event leaving node : ko id=patient-pond name=crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] node (patient-pond) ret : false node-name=patient-pond stage=s01-parse
TRAC[15-12-2021 11:58:22] Processing node 5/12 -> cold-cherry node-name=cold-cherry stage=s01-parse
TRAC[15-12-2021 11:58:22] Event entering node id=cold-cherry name=crowdsecurity/modsecurity stage=s01-parse
DEBU[15-12-2021 11:58:22] Event leaving node : ko (failed filter) id=cold-cherry name=crowdsecurity/modsecurity stage=s01-parse
TRAC[15-12-2021 11:58:22] node (cold-cherry) ret : false node-name=cold-cherry stage=s01-parse
TRAC[15-12-2021 11:58:22] Processing node 6/12 -> hidden-snowflake node-name=hidden-snowflake stage=s01-parse
TRAC[15-12-2021 11:58:22] Event entering node id=hidden-snowflake name=crowdsecurity/nginx-logs stage=s01-parse
DEBU[15-12-2021 11:58:22] Event leaving node : ko (failed filter) id=hidden-snowflake name=crowdsecurity/nginx-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] node (hidden-snowflake) ret : false node-name=hidden-snowflake stage=s01-parse
TRAC[15-12-2021 11:58:22] Processing node 7/12 -> young-waterfall node-name=young-waterfall stage=s01-parse
TRAC[15-12-2021 11:58:22] Event entering node id=young-waterfall name=crowdsecurity/sshd-logs stage=s01-parse
DEBU[15-12-2021 11:58:22] Event leaving node : ko (failed filter) id=young-waterfall name=crowdsecurity/sshd-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] node (young-waterfall) ret : false node-name=young-waterfall stage=s01-parse
DEBU[15-12-2021 11:58:22] Log didn't finish stage s01-parse
DEBU[15-12-2021 11:58:22] Discarding line {Type:0 ExpectMode:1 Whitelisted:false WhitelistReason: Stage:s01-parse Line:{Raw:176.53.221.38 - - [15/Dec/2021:17:39:52 +0000] "GET / HTTP/1.1" 308 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" Src:/var/log/caddy/test.log Time:2021-12-15 11:58:22.200949223 -0700 MST m=+4.603803966 Labels:map[type:caddy] Process:true Module:file} Parsed:map[message:176.53.221.38 - - [15/Dec/2021:17:39:52 +0000] "GET / HTTP/1.1" 308 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" program:caddy] Enriched:map[] Overflow:{Mapkey: BucketId: Whitelisted:false Reprocess:false Sources:map[] Alert:<nil> APIAlerts:[]} Time:2021-12-15 11:58:22.209799114 -0700 MST m=+4.612653893 StrTime: MarshaledTime: Process:false Meta:map[datasource_path:/var/log/caddy/test.log datasource_type:file]}
---snip---
I found that I can force crowdsec to use the apache2-logs parser by modifying /etc/crowdsec/parsers/s01-parse/apache2-logs.yaml to:
filter: "evt.Parsed.program startsWith 'caddy'"
...which then gets me this...
Successful grok parse via apache2-logs:
# crowdsec -dsn file:///var/log/caddy/access.log -type caddy -no-api -trace
---snip---
TRAC[15-12-2021 11:49:28] INPUT '176.53.221.38 - - [15/Dec/2021:17:39:52 +0000] "GET / HTTP/1.1" 308 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36"'
TRAC[15-12-2021 11:49:28] node stage : s00-raw, current stage : s00-raw
TRAC[15-12-2021 11:49:28] Processing node 0/11 -> lingering-dew node-name=lingering-dew stage=s00-raw
TRAC[15-12-2021 11:49:28] Event entering node id=lingering-dew name=crowdsecurity/docker-logs stage=s00-raw
DEBU[15-12-2021 11:49:28] Event leaving node : ko (failed filter) id=lingering-dew name=crowdsecurity/docker-logs stage=s00-raw
TRAC[15-12-2021 11:49:28] node (lingering-dew) ret : false node-name=lingering-dew stage=s00-raw
TRAC[15-12-2021 11:49:28] Processing node 1/11 -> silent-sea node-name=silent-sea stage=s00-raw
TRAC[15-12-2021 11:49:28] Event entering node id=silent-sea name=crowdsecurity/syslog-logs stage=s00-raw
DEBU[15-12-2021 11:49:28] Event leaving node : ko (failed filter) id=silent-sea name=crowdsecurity/syslog-logs stage=s00-raw
TRAC[15-12-2021 11:49:28] node (silent-sea) ret : false node-name=silent-sea stage=s00-raw
TRAC[15-12-2021 11:49:28] Processing node 2/11 -> polished-wood node-name=polished-wood stage=s00-raw
TRAC[15-12-2021 11:49:28] Event entering node id=polished-wood name=crowdsecurity/non-syslog stage=s00-raw
TRAC[15-12-2021 11:49:28] ! No grok pattern : 0x0 id=polished-wood name=crowdsecurity/non-syslog stage=s00-raw
TRAC[15-12-2021 11:49:28] State after nodes : true id=polished-wood name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:49:28] + Processing 4 statics id=polished-wood name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:49:28] .Parsed[message] = '176.53.221.38 - - [15/Dec/2021:17:39:52 +0000] "GET / HTTP/1.1" 308 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36"' id=polished-wood name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:49:28] .Parsed[program] = 'caddy' id=polished-wood name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:49:28] .Meta[datasource_path] = '/var/log/caddy/test.log' id=polished-wood name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:49:28] .Meta[datasource_type] = 'file' id=polished-wood name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:49:28] Event leaving node : ok id=polished-wood name=crowdsecurity/non-syslog stage=s00-raw
TRAC[15-12-2021 11:49:28] node is successful, check strategy
DEBU[15-12-2021 11:49:28] move Event from stage s00-raw to s01-parse id=polished-wood name=crowdsecurity/non-syslog stage=s00-raw
TRAC[15-12-2021 11:49:28] Node successful, continue id=polished-wood name=crowdsecurity/non-syslog stage=s00-raw
TRAC[15-12-2021 11:49:28] node (polished-wood) ret : true node-name=polished-wood stage=s00-raw
DEBU[15-12-2021 11:49:28] node successful, stop end stage s00-raw node-name=polished-wood stage=s00-raw
TRAC[15-12-2021 11:49:28] node stage : s01-parse, current stage : s01-parse
TRAC[15-12-2021 11:49:28] Processing node 3/11 -> throbbing-field node-name=throbbing-field stage=s01-parse
TRAC[15-12-2021 11:49:28] Event entering node id=throbbing-field name=crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:28] ! No grok pattern : 0x0 id=throbbing-field name=crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:28] Event entering node id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:28] Node has not filter, enter id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:28] Processing grok pattern : : 0xc000678000 id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:28] line 45.90.62.143 - - [15/Dec/2021:17:39:54 +0000] "GET / HTTP/1.1" 308 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" oneshot=/var/log/caddy/test.log type="file:///var/log/caddy/test.log"
DEBU[15-12-2021 11:49:28] + Grok '(%{IP...' returned 13 entries to merge in Parsed id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:28] .Parsed['response'] = '308' id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:28] .Parsed['bytes'] = '0' id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:28] .Parsed['verb'] = 'GET' id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:28] .Parsed['timestamp'] = '15/Dec/2021:17:39:52 +0000' id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:28] .Parsed['http_user_agent'] = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36' id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] .Parsed['target_fqdn'] = '' id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] .Parsed['request'] = '/' id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] .Parsed['auth'] = '-' id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] .Parsed['httpversion'] = '1.1' id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] .Parsed['referrer'] = '-' id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] .Parsed['clientip'] = '176.53.221.38' id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] .Parsed['ident'] = '-' id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] .Parsed['rawrequest'] = '' id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:29] event against holder 0/28 cfg=red-butterfly file=/etc/crowdsec/scenarios/f5-big-ip-cve-2020-5902.yaml name=crowdsecurity/f5-big-ip-cve-2020-5902
DEBU[15-12-2021 11:49:29] Event leaving node : ko (filter mismatch) cfg=red-butterfly file=/etc/crowdsec/scenarios/f5-big-ip-cve-2020-5902.yaml name=crowdsecurity/f5-big-ip-cve-2020-5902
TRAC[15-12-2021 11:49:29] event against holder 1/28 cfg=lingering-shadow file=/etc/crowdsec/scenarios/http-generic-bf.yaml name=crowdsecurity/http-generic-bf
DEBU[15-12-2021 11:49:29] .Meta[log_type] = 'http_access-log' id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] setting target StrTime to 15/Dec/2021:17:39:52 +0000
DEBU[15-12-2021 11:49:29] evt.StrTime = '15/Dec/2021:17:39:52 +0000' id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] .Meta[service] = 'http' id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] .Meta[source_ip] = '176.53.221.38' id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] .Meta[http_status] = '308' id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] .Meta[http_path] = '/' id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:29] State after nodes : true id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:29] ! No node statics id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] Event leaving node : ok id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:29] node is successful, check strategy
DEBU[15-12-2021 11:49:29] move Event from stage s01-parse to s02-enrich id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:29] Node successful, continue id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:29] sub-node (weathered-fire) ret : true (strategy:next_stage) id=throbbing-field name=crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] child is success, OnSuccess=next_stage, skip id=throbbing-field name=crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:29] State after nodes : true id=throbbing-field name=crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:29] ! No node statics id=throbbing-field name=crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] Event leaving node : ok id=throbbing-field name=crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:29] node is successful, check strategy
DEBU[15-12-2021 11:49:29] node reached the last stage : s02-enrich id=throbbing-field name=crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:29] Node successful, continue id=throbbing-field name=crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:29] node (throbbing-field) ret : true node-name=throbbing-field stage=s01-parse
DEBU[15-12-2021 11:49:29] node successful, stop end stage s01-parse node-name=throbbing-field stage=s01-parse
---snip---
Did I configure something incorrectly?
jzemla commented
So, I am openly calling myself out on knowing just enough to be dangerous and not having a full understanding of what I'm doing. Hello, world!
In short, this is resolved by changing:
"encoder": {
"format": "formatted",
"template": "{common_log} \"{request>headers>Referer>[0]}\" \"{request>headers>User-Agent>[0]}\""
},
to:
"encoder": {
"format": "json"
},
...and thus benefitting from caddy's structured log files which is the purpose of this bouncer to begin with.
hslatman commented
@jzemla: great that you found out yourself!
I should probably update the example for the logs in config.json. Back when I included it for the first time I had to output Caddy logs in the Apache format for CrowdSec to parse it. It seems support for the Caddy structured format was added to CrowdSec not too long ago, so it's nice that it now works out of the box 😄.
I've always considered the example for the logs as a kind of extra. It's not required to ingest the Caddy logs into CrowdSec to make the bouncer work, but it's a good thing to do, nonetheless.
Have opened #10 to track this. Your example will help me test this. Thanks!
jdeath commented
Sorry to open an old issue. Could anyone get the caddy-logs parser to work in 2.6.2? I also tried using a grok debugger to find what changed, but couldn't get it to work with both console and json log formats. I was able to get it to work by downloading caddy with the transform plugin and outputting in the common_log format and use the apache2 collection.
log {
format transform "{common_log}"
}
Also, I had to change the apache2-logs.yaml file to look for the logs coming from caddy instead of apache (I use homeassistant, so needed to use the plugin name)
filter: "evt.Parsed.program startsWith 'addon_c80c7555_caddy-2'"
Direct caddy logs would be nicer, but this method works. The bouncer works fine!
rserbitar commented
So, what is the correct config file currently? And how can I test that?
Edit: I also succeeded with transformation to apache2 like jdeath did. Would be ncie if native caddy format could be suported again . . .