Dsiem ES proxy

This tool replaces Logstash in Dsiem architecture by:

Implementing the scripted upsert functionality in 80_siem.conf, which is meant to avoid inserting duplicate entries for the same alarm into Elasticsearch, and to avoid updating status and tag fields if they've been updated through other means (e.g. Dsiem UI).
Upserting the default siem_alarms index template to Elasticsearch when there isn't one already exist.

An example of Dsiem architecture that completely replaces Logstash with Vector is shown in the diagram below. There is a docker compose deployment for it in here. The vector config files in particular have comments that further explain the setup.

flowchart TB

v3 -->|suricata alerts|es
v3 -->|normalized events|es
v3 -->|normalized events|df
ep -->|alarms|es

db -->|alarm_events|es
db -->|alarms|ep

s1 -->|eve.json|v3
s2 -->|syslog|v3

df -->|events through NATS|db

subgraph "Vector (Central)"
 v3[vector]
end

subgraph Source logs
 s1[suricata]
 s2[random_netdevice]
end

subgraph Elasticsearch cluster
 es[nodes]
end

subgraph Dsiem frontend
 ep[dsiem-es-proxy]
 df[frontend]
end

subgraph Dsiem backend
 db[backend + vector]
end

How to test the example

Clone this repo:

git clone https://github.com/mmta/dsiem-esproxy.git

Start the docker compose environment (replace eth0 with appropriate network interface):
```
cd examples/elasticsearch
export PROMISC_INTERFACE=eth0
docker compose up
```

From another terminal in the same host, install Kibana dashboard:

cd examples/elasticsearch
./kbndashboard-import.sh localhost ./kibana/dashboard-siem.json

From another terminal in the same host, ping an external IP, and make sure the traffic goes through that network interface (e.g. eth0).
```
ping 1.1.1.1
```
Open Kibana dashboard from http://localhost:5601, and verify that ping related alarms are created in the Dsiem dashboard.

Surrealdb version

There is another example that also replace Elasticsearch with Surrealdb in here.

Activate the compose environment:

cd examples/surrealdb
export PROMISC_INTERFACE=eth0
docker compose up

From another terminal in the same host, ping an external IP, and make sure the traffic goes through that network interface (e.g. eth0).
```
ping 1.1.1.1
```
After that, visit https://surrealist.app/ and use it to connect to:
- Endpoint: http://localhost:8000
- Namespace: default
- Database: dsiem
- Authentication: Anonymous

Misc Info

Docker image is available from packages. Refer to the compose example on how to use it, and the CLI --help parameter to see all the valid arguments.

Use or review the Dockerfile on how to build the project.

docker build . -t ghcr.io/mmta/dsiem-esproxy
docker run --rm ghcr.io/mmta/dsiem-esproxy --help