Unifi OS 3 aren't being parsed
GNU-Plus-Windows-User opened this issue · 18 comments
Describe the bug
Unifi OS 3 and newer logs are not being parsed correctly, resulting in detection scenarios such as port scanning not working correctly.
To Reproduce
- Install the unifi collection
cscli collections install crowdsecurity/unifiand reload crowdsec - Setup a syslog endpoint via acquis.yaml using the following yaml:
source: syslog
listen_addr: 0.0.0.0
listen_port: 514
labels:
type: unifi
- Configure a Unifi OS 3 console or newer to log to the syslog endpoint
- check crowdsec logs and see
time="03-11-2023 04:48:35" level=error msg="could not parse message: version must be 1" client=0.0.0.0 type=syslog - check
cscli metricsand see no logs are being parsed
╭────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────╮
│ Source │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │
├────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┤
│ syslog:0.0.0.0 │ 295 │ - │ 295 │ - │
╰────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────╯
Expected behavior
Logs should be parsed
Screenshots
N/A
Additional context
This issue was originally reported within the CrowdSec Discord
check crowdsec logs and see time="03-11-2023 04:48:35" level=error msg="could not parse message: version must be 1" client=0.0.0.0 type=syslog
so the error is happening within syslog acquisition itself
It not even hitting the parsers at all..... so what format is it if its not RFC3164 or RFC5424
Can you post some example lines?
@LaurenceJJones I'm not sure what to look for, so let me know if you are missing some specific logs.
I didn't remove the MAC address from the last log line, that's how it was sent.
Feb 8 18:19:32 Unifi-Dream-Machine [LAN_LOCAL-RET-2147483647] DESCR="no rule description" IN=br0 OUT= MAC=fake-mac-address SRC=0.0.0.0 DST=0.0.0.0 LEN=40 TOS=00 PREC=0x00 TTL=128 ID=31307 DF PROTO=TCP SPT=54649 DPT=443 SEQ=578136041 ACK=657436146 WINDOW=8195 ACK URGP=0 UID=125 GID=132 MARK=1a0000
Feb 8 18:19:31 Unifi-Dream-Machine [WAN_LOCAL-D-2147483647] DESCR="[WAN_LOCAL]Drop All Other Traf" IN=eth4 OUT= MAC=fake-mac-address SRC=0.0.0.0 DST=0.0.0.0 LEN=40 TOS=00 PREC=0x00 TTL=239 ID=13706 PROTO=TCP SPT=45584 DPT=29552 SEQ=2451790175 ACK=0 WINDOW=1024 SYN URGP=0 MARK=1a0000
Feb 8 18:19:30 Unifi-Dream-Machine [LAN_IN-D-20038] DESCR="Default Implicit Deny" IN=br0 OUT=eth4 MAC=fake-mac-address SRC=0.0.0.0 DST=0.0.0.0 LEN=243 TOS=00 PREC=0x00 TTL=63 ID=3558 DF PROTO=UDP SPT=6537 DPT=6537 LEN=223 MARK=1a0000
Feb 8 18:23:33 Unifi-Dream-Machine [PREROUTING-DNAT-13] DESCR="PortForward DNAT [Reverse Proxy 44" IN=br5 OUT= MAC=fake-mac-address SRC=0.0.0.0 DST=0.0.0.0 LEN=60 TOS=00 PREC=0x00 TTL=64 ID=42142 DF PROTO=TCP SPT=50118 DPT=443 SEQ=746590349 ACK=0 WINDOW=64240 SYN URGP=0 MARK=1a0000
Feb 8 18:23:33 Unifi-Dream-Machine [POSTROUTING-MASQUERADE-14] DESCR="PortForward MASQUERADE [Rev" IN= OUT=br5 MAC= SRC=0.0.0.0 DST=0.0.0.0 LEN=60 TOS=00 PREC=0x00 TTL=63 ID=42142 DF PROTO=TCP SPT=50118 DPT=443 SEQ=746590349 ACK=0 WINDOW=64240 SYN URGP=0 MARK=1a0000
Can you capture the raw syslog packet?
The issue at the moment is the syslog acquisition NOT the parser. If you used rsyslog to a file it would work fine.
Hey @LaurenceJJones,
SYSLOG on CrowdSec Node: https://drive.proton.me/urls/SCXVG17A2R#0wuY9TIDGhzc
Local TCPDump from UDM SE: https://drive.proton.me/urls/F0SZV6Z4W0#GtsV6AZflD13
Lemme know if you need it in a dif format.
@WhyAydan Thank you for providing these, I didn't have the time to run a packet capture.
Tbh, no idea if thats what Laurence needs but who knows lol
Hmmm it seems to be RFC compliant on my end and within @WhyAydan pcap also
<45>Feb 12 09:52:07 ToonDreamMachine ToonDreamMachine syslog-ng[3459965]: Syslog connection established; fd='28', server='AF_INET(10.72.1.222:514)', local='AF_INET(0.0.0.0:0)'
Still would like a pcap from @GNU-Plus-Windows-User just incase it something we are not seeing
I will do some more testing
Hey, if it helps I also get the same error that @GNU-Plus-Windows-User gets from crowdsec
Hey, if it helps I also get the same error that @GNU-Plus-Windows-User gets from crowdsec
Okay, then I try to see if I can reply the packet the syslog endpoint.
Also if you get chance can you put the acquisition into debug log level as it should log the reason why the first RFC parser fails
time="2024-02-12T13:12:46Z" level=debug msg="could not parse as RFC5424 (version must be 1) : <15>Bedroom HIDDEN,USW_FLEX_MINI-2.0.0.704: INFORM: Send notify [setparam] inform to [http://192.168.1.1:8080/inform] Time 1974750" client=192.168.1.230 type=syslog
time="2024-02-12T13:12:47Z" level=debug msg="could not parse as RFC3164 (timestamp is not valid)" client=192.168.1.230 type=syslog
time="2024-02-12T13:12:47Z" level=error msg="could not parse message: version must be 1" client=192.168.1.230 type=syslog
time="2024-02-12T13:12:47Z" level=debug msg="could not parse as RFC5424 (version must be 1) : <15>Bedroom HIDDEN,USW_FLEX_MINI-2.0.0.704: INFORM: Send normal inform to [http://192.168.1.1:8080/inform] Time 1974751" client=192.168.1.230 type=syslog
time="2024-02-12T13:13:06Z" level=debug msg="could not parse as RFC3164 (timestamp is not valid)" client=192.168.1.230 type=syslog
time="2024-02-12T13:13:06Z" level=error msg="could not parse message: version must be 1" client=192.168.1.230 type=syslog
time="2024-02-12T13:13:06Z" level=debug msg="could not parse as RFC5424 (version must be 1) : <15>Bedroom HIDDEN,USW_FLEX_MINI-2.0.0.704: INFORM: Send normal inform to [http://192.168.1.1:8080/inform] Time 1974771" client=192.168.1.230 type=syslog
time="2024-02-12T13:13:24Z" level=debug msg="could not parse as RFC3164 (timestamp is not valid)" client=192.168.1.230 type=syslog
time="2024-02-12T13:13:24Z" level=error msg="could not parse message: version must be 1" client=192.168.1.230 type=syslog
time="2024-02-12T13:13:24Z" level=debug msg="could not parse as RFC5424 (version must be 1) : <15>Bedroom HIDDEN,USW_FLEX_MINI-2.0.0.704: INFORM: Send normal inform to [http://192.168.1.1:8080/inform] Time 1974788" client=192.168.1.230 type=syslog
time="2024-02-12T13:13:44Z" level=debug msg="could not parse as RFC3164 (timestamp is not valid)" client=192.168.1.230 type=syslog
time="2024-02-12T13:13:44Z" level=error msg="could not parse message: version must be 1" client=192.168.1.230 type=syslog
time="2024-02-12T13:13:44Z" level=debug msg="could not parse as RFC5424 (version must be 1) : <15>Bedroom HIDDEN,USW_FLEX_MINI-2.0.0.704: INFORM: Send normal inform to [http://192.168.1.1:8080/inform] Time 1974808" client=192.168.1.230 type=syslog
Any update on this issue?
would love to help to get this up and running given the new SIEM type logging also available in the new network release
I spent 3 days working on this before I came across someone telling me that it wasn't working properly. I appreciate the community and have learned a lot. How could we help get this working?
There is 2 issues which one is confirmed, the other im still in the air about:
- Syslog acquisition does not allow relayed packets (confirmed as per linked issue above)
- If you don't use CrowdSec syslog acquisition but instead write the packet to a file using
rsysloginstead does it work?
Keep in mind that currently the syslog parser in s00 is only RFC3164 compliant please ensure if you are testing the latter that you set the syslog option in unifi to use this RFC version. If there isnt an option I guess we need to update a newer RFC version one
I don't see where you could select RFC3164 on the udmpro. I have limited knowledge in this area, so I apologize. I do have it setup via rsyslog which pulls the information to the syslog file.
I even tried to pull just the IPS Alerts to a file and set it up so that crowdsec reads it, but it won't parse them. (I'm sure with my limited knowledge I'm missing something, but I'm trying.)
Here are what those files look like.
2024-11-02T17:15:47-04:00 UDMPRO CEF: 1|Ubiquiti|UniFi Network|8.5.6|security_detections-67265e439be3511159c45cae|IPS Alert 2: Misc Attack. Signature ET DROP Dshield Block Listed Source group 1. From: 193.163.125.105:50655, to: 10.180.8.66:80, protocol: TCP|Medium|signature_type=ET signature_id=2402000
2024-11-02T17:15:47-04:00 UDMPRO CEF: 1|Ubiquiti|UniFi Network|8.5.6|security_detections-67265e439be3511159c45cae|IPS Alert 2: Misc Attack. Signature ET DROP Dshield Block Listed Source group 1. From: 193.163.125.105:50655, to: 10.180.8.66:80, protocol: TCP|Medium|signature_type=ET signature_id=2402000
2024-11-02T17:20:47-04:00 UDMPRO CEF: 1|Ubiquiti|UniFi Network|8.5.6|security_detections-67265f6f9be3511159c45cf9|IPS Alert 2: Misc Attack. Signature ET DROP Dshield Block Listed Source group 1. From: 92.255.85.30:54406, to: 10.180.8.66:443, protocol: TCP|Medium|signature_type=ET signature_id=2402000
2024-11-02T17:20:47-04:00 UDMPRO CEF: 1|Ubiquiti|UniFi Network|8.5.6|security_detections-67265f6f9be3511159c45cf9|IPS Alert 2: Misc Attack. Signature ET DROP Dshield Block Listed Source group 1. From: 92.255.85.30:54406, to: 10.180.8.66:443, protocol: TCP|Medium|signature_type=ET signature_id=2402000
2024-11-02T17:25:48-04:00 UDMPRO CEF: 1|Ubiquiti|UniFi Network|8.5.6|security_detections-6726609c9be3511159c45d78|IPS Alert 2: Attempted Information Leak. Signature ET WEB_SPECIFIC_APPS Microhard Systems 3G/4G Cellular Ethernet and Serial Gateway - Default Credentials. From: 193.68.89.23:50010, to: 10.180.8.66:80, protocol: TCP|Medium|signature_type=ET signature_id=2025855
Yeah that is something called CEF, we dont support that at all yet, im trying to find a definition of the CEF format and it seems it whatever you make it to be. So not too useful for us to create a parser unless we create one per vendor.
I appreciate you looking at it and look forward to see what you come up with.