Based on a Splunk perspective. Below resources show that ingesting your logs is essential from proper analysis of the Indicators Of Compromise Never waste a good crisis...ingest all the logs!
Remote pre-auth arbitrary command execution due to logic vuln i.e. reliable execution possible.
https://support.citrix.com/article/CTX267027 https://www.fireeye.com/blog/threat-research/2020/01/vigilante-deploying-mitigation-for-citrix-netscaler-vulnerability-while-maintaining-backdoor.html https://www.trustedsec.com/blog/netscaler-remote-code-execution-forensics/ http://deyda.net/index.php/en/2020/01/15/checklist-for-citrix-adc-cve-2019-19781/ https://github.com/x1sec/CVE-2019-19781/blob/master/CVE-2019-19781-DFIR.md https://nvd.nist.gov/vuln/detail/CVE-2019-19781 https://www.reddit.com/r/blueteamsec/comments/en4m7j/multiple_exploits_for_cve201919781_citrix/ Dutch NCSC: https://english.ncsc.nl/latest/news/2020/january/20/install-patches-for-citrix-adc-en-citrix-gateway-servers https://www.ncsc.nl/actueel/advisory?id=NCSC%2D2019%2D0979 Flowchart from NCSC: https://english.ncsc.nl/binaries/ncsc-en/documents/publications/2020/januari/21/flowchart-citrix-vulnerability/Flowchart+Citrix+vulnerability.pdf Most orgs, sites, co's regurgitate the same... :-( just like me ;-)
- Collect Diagnostic Bundle Files (Collector file / dump) on the Netscalers and ingested the dump in Splunk: https://docs.citrix.com/en-us/citrix-adc/13/system/basic-operations/how-to-collect-technical-support-bundle.html
- Review HTTP log files -------------------------------------------------------------------:heavy_check_mark:used Splunk
/var/log/httpaccess.log | /var/log/httperror.log | /etc/httpd.conf | /var/log/websocketd.log from dumpfiles and our syslog data from Netscalers. Check Webserver log entries indicating successful exploitation...
- Check all modified from the 10th of Jan 2020 until now ----------------------------------:heavy_check_mark:used Splunk
Check filesystem paths of known malware
- Review all template files that are non standard------------------------------------------:heavy_check_mark:used Splunk
selection: c-uri-path: - '/../vpns/' - '/vpns/cfg/smb.conf' - '/vpns/portal/scripts/.pl' and all other variants you can think of. Look for malicious terms and or unexpected modifications in the Netscaler dirs.
- Check cronjobs for all users-------------------------------------------------------------:heavy_check_mark:used Splunk
/shell/crontab-l.out
- Check all running processes--------------------------------------------------------------:heavy_check_mark:used Splunk
/shell/top-b.out + others
- Check bash history-----------------------------------------------------------------------:heavy_check_mark:used Splunk
/var/log/bash.log from dumpfile & index=yourindex host=yournetscalerhosts (bash.log & sh.log) Basically check for all post-exploitation in shell history...
- Review listening services and tcp/udp connections----------------------------------------:heavy_check_mark:used Splunk
/shell/sockstat.out and other methods. Check ports used by know malware.
- Checked our ISP/ASN number for vulnerable systems in scope------------------------------------:heavy_check_mark:used Splunk
index=yourshodanindex asn=yourASN# CVE-2019-19781 vulns.CVE-2019-19781.verified=true
- Check firewall /ids/ips logging-----------------------------------------------------------:heavy_check_mark:used Splunk
No comment 😅
-
Check in Splunk (Based on Sigma rule-->https://github.com/Neo23x0/sigma/blob/master/rules/web/web_citrix_cve_2019_19781_exploit.yml#L17 detection: selection: c-uri-path: - '/../vpns/' - '/vpns/cfg/smb.conf' - '/vpns/portal/scripts/.pl' condition: selection fields:
- client_ip
- vhost
- url
- response
More possible IOC's:
Got hit with a new backdoor on the Citrix #netscaler CVE-2019-19781 honeypot last night. Its a DDoS bot that comms over...IRC Watch out for conns to 50.71.90.246:4545 and files named /tmp/.perl
Please add IOC's or other things to check if I missed something. #Sh*trix
#Honeypot logs to Splunk https://github.com/x1sec/citrix-honeypot
Results / data is written to the ./log directory. They are: hits.log - Scanning attempts and exploitation attempts with all data (e.g. headers, post body) all.log - All HTTP requests that are observed hitting the server logins.log - Attempted logins to the web interface tlsErrors.log - Often internet scanners will send invalid data to port 443. HTTPS errors are logged here.
#Updates
-
Citrix released patch on 23/01/2020: https://www.citrix.com/blogs/2020/01/23/fixes-now-available-for-citrix-adc-citrix-gateway-versions-12-1-and-13-0/
-
Install Splunk Universal Forwarder after you freshly reinstalled the Netscalers. (FreeBSD install) Docs: https://docs.splunk.com/Documentation/Forwarder/8.0.1/Forwarder/Installanixuniversalforwarder#Install_the_universal_forwarder_on_FreeBSD Binary: https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=freebsd&version=8.0.1&product=universalforwarder&filename=splunkforwarder-8.0.1-6db836e2fb9e-freebsd-11.1-amd64.txz&wget=true
-
How to Configure Citrix NetScaler to produce syslog data: https://docs.splunk.com/Documentation/AddOns/released/CitrixNetscaler/Setup https://docs.citrix.com/en-us/citrix-adc/12-1/system/audit-logging/configuring-audit-logging.html
-
Splunk Add-On for Citrix Netscaler: https://splunkbase.splunk.com/app/2770/