winston-logstash will not reconnect with logstash when using --config.reload.automatic

[hassan379]

Hi
Thank you for this awesome repository
but I have noticed that winston-logstash will not reconnect with logstash or even throw an error when using --config.reload.automatic .
for example when I execute this command
logstash -f test.conf --config.reload.automatic
everything works well without any problem but when I try to update test.conf, winston will not print any message in console if winston.transports.Console is used or send any payload to logstash
If I tried to stop logstash and then re-execute the command above, everything will back to its normal state
this is how i call winston

const logger = winston.createLogger({
    format: winston.format.combine(
        winston.format.timestamp({
            format: 'YYYY-MM-DD HH:mm:ss',
        }),
        winston.format.printf((info) => `${info.timestamp} ${info.level}: ${info.message}${info.splat !== undefined ? `${info.splat}` : ' '}`),
    ),
    transports: [
        new logstashTransport({
            port: 28777,
            max_connect_retries: -1,
            node_name: 'DESKTOP',
            host: '127.0.0.1'}),
        new winston.transports.Console({
            format: winston.format.combine(
                winston.format.timestamp({
                    format: 'YYYY-MM-DD HH:mm:ss',
                }),
                winston.format.printf((info) => colorizer.colorize(info.level, `[${info.timestamp}] [${info.level}]: ${info.message}${info.splat !== undefined ? `${info.splat}` : ' '}`)),
            ),
        }),
        new winstonDailyRotateFile({
            filename: 'src/logs/%DATE%.log',
            datePattern: 'YYYY-MM-DD',
        })
    ],
});

do you have any idea why ?
thanks in advance

[jaakkos]

Hi!

I don't have any direct advice, but I'll try to reproduce the issue and figure out the problem. I'll get back when I have some thing to share. 👍

[jaakkos]

I started to write test cases for the config reload when I noticed an issue with the reload when there was output with TCP configuration. It doesn't reload the configuration because it can't start the TCP server because the port is still in use. Have you noticed a similar case? my draft pr #76

logstash-logstash-1 | [2023-03-18T08:20:23,286][ERROR][logstash.outputs.tcp ][main] Could not start tcp server: Address in use {:host=>"0.0.0.0", :port=>9999}

[hassan379]

No
I can not see your error log in my logstash console
I am using logstash 8.6.2 and elasticsearch 8.6.2 on Windows
and this is my .conf file

    input {
        tcp { port => 28777 }
    }
    output {
      elasticsearch {
        hosts => ["http://localhost:9200/"]
        user => "elastic"
        password => "PASSWORD"
        index => "dummy_index"
      }
      stdout { }
    }

    filter {
        json {
          source => "message"
        }
        json {
          source => "message"
          remove_field => [ "message","event","timestamp" ]
        }
    }

[jaakkos]

My current test setup uses TCP output, which leads the reload case to this known bug https://discuss.elastic.co/t/tcp-output-plugin-blocking-port-across-configuration-reload/163732 . So I need to go around it to write the test case for config reloading.

[jaakkos]

While I failed to write a failing test due to the bug, I created a sandbox setup where I reproduced the scenario. On the reload, the Logstash closes the connection without any error, which leads to a case where the code is not retrying. Next, I need to figure out if adding the retry logic might break the backward compatibility; if not, it's a straightforward change.

[jaakkos]

A working solution is now for reconnecting to logstash after it closes the connection in draft #76. I still need to figure out the best way to write some tests and check if the change is backwards compatible.

[jaakkos]

I just released a new version which addressed this issue. If you still have issues with the reconnect don't hesitate to re-open the issue. 😄

[nitzan-blink]

@jaakkos I think this issue really describes two issues:

1. reconnect logstash after one sided disconnection <-- this one you already handled
2. all other transports are getting paralyzed, when using max_connect_retries: -1

the second issue is still relevant, easily reproducible, and very dangerous in production.
while winston-logstash tries to reconnect, all other transports do not work at all.
imagine logstash going down for 5 minutes - NOTHING is getting logged anywhere (in the first post there are TWO more transports - console and file), until it either returns, or the retry limit reached (in which case nothing further will log, unless you manually remove and add transports, in an error handler).
why is there such a strong coupling between the transports?