syslog2nats

Receives syslog messages and publishs them to NATS

syslog2nats is based on

syslog2nats consists of:

syslog server - common part for all syslogsidecar based processes
NATS specific plugins

Syslog server

Supported RFCs:

RFC3164 message consists of following symbolic parts:

priority
facility
severity
timestamp
hostname
tag
content

RFC5424

RFC5424 message consists of following symbolic parts:

priority
facility
severity
timestamp
hostname
version
app_name
proc_id
msg_id
structured_data
message

Non-RFC parts

syslogsidecar adds rfc of produced message:

Part name: "rfc"
Values: "RFC3164"|"RFC5424"

Badly formatted messages

syslogsidecar creates only one part for badly formatted message - former syslog message:

Part name: "data"

Severities

Valid severity levels and names are:

0 emerg
1 alert
2 crit
3 err
4 warning
5 notice
6 info
7 debug

syslogsidecar filters messages by level according to value in configuration, e.g. for:

{
  "SEVERITYLEVEL": 4,
  ...........
}

all messages with severity above 4 will be discarded.

Configuration

Configuration of syslog server part of syslogsidecar is saved in the file syslogreceiver.json:

{
    "SEVERITYLEVEL": 4,
    "ADDRTCP": "127.0.0.1:5141",
    "ADDRUDP": "127.0.0.1:5141",
    "UDSPATH": "",
    "ADDRTCPTLS": "127.0.0.1:5143",
    "CLIENT_CERT_PATH": "",
    "CLIENT_KEY_PATH ": "",
    "ROOT_CA_PATH": ""
}

Plugins

NATS plugins to syslogsidecar:

connector
producer
consumer (used for the tests)

Connector

Configuration file: connector.json

{
    "CONN_URL": "nats://127.0.0.1:4222"
}

The rest of connection options are default.

Connector creates sharable nats.Conn for:

periodic validation of connectivity with memphis
using by producer (production) and consumer (e2e test)

More about connector and underlying software - sputnik

Producer

Configuration file: syslogproducer.json

{
    "STREAM": "syslog"
}

Producer uses received from connector nats.Conn. It created JETSTREAM with name from configuration, the rest of stream options are default.

syslog messages are produced to jetstream as Header with empty payload:

    .................................
    msg := &nats.Msg{
		Subject: name,
		Header:  make(nats.Header),
	}

	putToheader := func(name string, value string) error {
		msg.Header.Add(name, value)
		return nil
	}

	syslogsidecar.Unpack(inmsg, putToheader)
    .................................

Build and run under vscode

go clean -cache -testcache
go build ./cmd/syslog2nats/
./syslog2nats --cf ./cmd/syslog2nats/conf/

e2e test

Simultaneuosly:

send 1000000 syslog messages
- TCP/IP
- RFC5424
receive
produce
consume
compare

Build and run under vscode:

go clean -cache -testcache
go build ./cmd/syslog-e2e/
./syslog-e2e

nats server runs as as part of syslog-e2e process.

Embedding configuration files

syslog-e2e uses embedded configuration files:

import (
	"embed"
  .........
)

//go:embed conf
var embconf embed.FS

func main() {
  ............................
  ............................
	cleanUp, _ := sidecar.UseEmbeddedConfiguration(&embconf)
	defer cleanUp()
	sidecar.Start(syslog2nats.NewConnector())
}

For this case content of conf subfolder embedded within process. No needs for "--cf" flag in command line.

Any value in configuration file may be overridden using environment variables

g41797/syslog2nats