Shovel is a web application that offers a graphical user interface to explore Suricata EVE outputs. Its primary focus is to help Capture-the-Flag players analyse network flows during stressful and time-limited attack-defense games such as FAUSTCTF, ENOWARS or ECSC. Shovel is developed in the context of ECSC Team France training.
You might also want to have a look at these other awesome traffic analyser tools:
- https://github.com/secgroup/flower (first commit in 2018)
- https://github.com/eciavatta/caronte (first commit in 2020)
- https://github.com/OpenAttackDefenseTools/tulip (fork from flower in May 2022)
Compared to these traffic analyser tools, Shovel only relies on Suricata while making opinionated choices for the frontend. This has a few nice implications:
- dissection of all application protocols supported by Suricata (HTTP2, modbus, SMB, DNS, etc),
- flows payloads and dissections are stored inside SQLite databases for fast queries,
- ingest can be a folder of pcaps for non-root CTF, or a live capture (less delay),
- tags are defined using Suricata rules (regex, libmagic match, HTTP header, etc),
- no heavy build tools needed, Shovel is easy to tweak.
Moreover, Shovel is batteries-included with some Suricata alert rules.
┌────────────────────────┐
device │ Suricata with: │ eve.db ┌───────────────┐
or pcap │ - Eve SQLite plugin ├────────────►│ │
───────►│ - TCP payloads plugin │ payload.db │ Python webapp │
│ - UDP payloads plugin ├────────────►│ │
└────────────────────────┘ └────▲──────────┘
.env │
──────┘
Shovel is configured using environment variables.
Copy example.env to .env and update the optional configuration parameters.
You may update this file later and restart only the webapp.
Add the flag format in suricata/rules/suricata.rules if needed.
If you modify this file after starting Suricata, you may reload rules using
pkill -USR2 suricata.
Suricata supports multiple capture methods.
Please use a live capture with AF_PACKET when possible,
or libpcap if you can't mirror the traffic (archives replay or rootless CTF).
Place pcap files in a folder such as input_pcaps/.
If you are continuously adding new pcap, add --pcap-file-continuous to
Suricata command line.
Then you may start the compose using:
docker compose up -dIf you don't want to use Docker, you may manually launch Suricata and the web application using the two following commands:
./suricata/entrypoint.sh -r input_pcaps
(cd webapp && uvicorn --host 127.0.0.1 main:app)Tip
If the CTF event does not already provide PCAP files, then you may adapt the following command for a GNU/Linux system (22 is SSH):
ssh root@10.20.9.6 tcpdump -i game -n -w - 'tcp port not 22' | tcpdump -n -r - -G 30 -w input_pcaps/trace-%Y-%m-%d_%H-%M-%S.pcapFor a Microsoft Windows system, you may adapt the following command (3389 is RDP):
.\tshark.exe -b duration:60 -w \\share\captures\trace -f "tcp port not 3389"Warning
Please note that restarting Suricata will cause all network capture files to be loaded again. It might add some delay before observing new flows.
This mode requires access to a network device with the game traffic.
Here this device is named tun5.
Edit docker-compose.yml and comment option A and uncomment option B under
suricata container definitions.
Then, you may start the compose using:
docker compose up -dIf you don't want to use Docker, you may manually launch Suricata and the web application using the two following commands:
sudo ./suricata/entrypoint.sh -i tun5
(cd webapp && uvicorn --host 127.0.0.1 main:app)Warning
Please note that stopping Suricata will stop network capture.
You may run sudo tcpdump -n -i tun5 -G 30 -w trace-%Y-%m-%d_%H-%M-%S.pcap for
archiving purposes.
flow_id is derived from timestamp (ms scale) and current flow parameters (such
as source and destination ports and addresses). See source code:
https://github.com/OISF/suricata/blob/suricata-6.0.13/src/flow.h#L680.
You can edit suricata rules in suricata/rules/suricata.rules, then reload the rules
using:
pkill -USR2 suricata