input from nginx error_log show unsupported version

[progamer71]

I use nginx-1.12.0 version and flowgger-master on macOS 10.12.4

in the nginx configuration file nginx.conf, I instruct nginx to send error log to local syslog server

error_log   syslog:server=127.0.0.1 debug;

in flowgger.toml

[input]

### Syslog over UDP
type = "udp"
listen = "0.0.0.0:514"

[output]

### Debug output (stdout)
type = "stdout"

grep -rnw 'src' -e 'Unsupported version'
src/flowgger/decoder/rfc5424_decoder.rs:79:        return Err("Unsupported version");
nano +79 src/flowgger/decoder/rfc5424_decoder.rs

// I add a println! to this function to show line value
fn parse_pri_version(line: &str) -> Result<Pri, &'static str> {
    println!("line:{}", line); // <----show line value
    if !line.starts_with('<') {
        return Err("The priority should be inside brackets");
    }
    let mut parts = line[1..].splitn(2, '>');
    let pri_encoded: u8 =
        try!(try!(parts.next().ok_or("Empty priority")).parse().or(Err("Invalid priority")));
    let version = try!(parts.next().ok_or("Missing version"));
    if version != "1" {
        return Err("Unsupported version");
    }
    Ok(Pri {
           facility: pri_encoded >> 3,
           severity: pri_encoded & 7,
       })
}

after I recompile and restart flowgger, start nginx and make some error to generate a error log

target/release/flowgger
Flowgger 0.2.6
<184> Apr
Unsupported version

Do you have a plan to support input nginx log (both error_log and access_log)?

[jedisct1]

Hi,

And thanks for using Flowgger!

I would recommend using a structured format such as ltsv, which works really well with Nginx, instead of syslog messages, whose format vary according to the syslog daemon and its configuration.

What does a line of log look like?

[jedisct1]

The system syslog daemon on macOS still uses the very old RFC3164 format, that got obsolete with RFC5424.

The old format has limitations. Timestamps cannot be reliably parsed, and payloads are limited to strings. Key/value pairs are not supported.

Flowgger doesn't support the old format, only the RFC5424 one, which is supported by common logging daemons such as rsyslogd.

Still, even RFC5424 is terrible. Slow, complicated and limited. Use LTSV.

[progamer71]

Thanks you for quick response.

This is the example of error.log

2016/10/02 17:05:36 [emerg] 1395#0: open() "./log/error.log" failed (2: No such file or directory)
2016/10/02 17:05:53 [emerg] 1399#0: open() "./logs/nginx.pid" failed (2: No such file or directory)
2016/10/02 17:06:46 [alert] 1407#0: setrlimit(RLIMIT_NOFILE, 100000) failed (1: Operation not permitted)
2016/10/02 17:06:46 [alert] 1408#0: setrlimit(RLIMIT_NOFILE, 100000) failed (1: Operation not permitted)
2016/10/02 17:06:46 [alert] 1409#0: setrlimit(RLIMIT_NOFILE, 100000) failed (1: Operation not permitted)
2016/10/02 17:06:46 [alert] 1410#0: setrlimit(RLIMIT_NOFILE, 100000) failed (1: Operation not permitted)
2016/10/04 14:55:35 [emerg] 4414#0: open() "./conf/nginx.conf" failed (2: No such file or directory)

after search for a while, i found that nginx generate 2 log files
1 access_log: the format can be customized in configuration file
2 error_log: the format is hard coded in src/core/ngx_log.c
YYYY/MM/DD HH:MM:SS [LEVEL] PID#TID: *CID MESSAGE
(reference http://stackoverflow.com/questions/16711573/nginx-error-log-format-documentation)

My use case is to create a centralized logging system from many nginx servers.
The connection need TLS and compression.
So flowgger seem to fit my use case.

Right now my solution is
nginx.conf

error_log   logs/error.log;
...
access_log   logs/access.log;

flowgger.toml

[input]
### Standard input
type = "stdin"

[output]
### TLS output
type = "tls"
connect = [ "172.16.205.128:6514", "172.16.205.129:6514" ]
timeout = 3600
tls_threads = 1
tls_cert = "flowgger.pem"
tls_key = "flowgger.pem"
tls_ca_file = "flowgger.pem"
# tls_compatibility_level = "intermediate"
# tls_verify_peer = false
tls_compression = true
# tls_ciphers = "EECDH+AES128:EECDH+CHACHA20:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3$
# tls_async = false
# tls_recovery_delay_init = 1
# tls_recovery_delay_max = 10000
# tls_recovery_probe_time = 30000

run command

tail -f -n 0 logs/error.log | flowgger flowgger.toml &
tail -f -n 0 logs/access.log | flowgger flowgger.toml &

it is not the best solution but good enough for my use case

Thanks you for your great work