The logger is logging "status response" from Elastic / OpenSearch

[projx]

Hi

I'm doing some testing with this and discovered that the response from the Search server is being logged itself:

{ "_index": "test1", "_type": "_doc", "_id": "1QPANVVKTN6", "_score": 2, "@timestamp": "2022-08-02T15:42:25.449999872Z", "_source": { "agent.ephemeral_id": "c7629bf6-b7db-406e-a065-188f66f96d38", "agent.type": "opensearch-logger", "agent.version": "1.2.1", "args": [ "POST", "http://127.0.0.1:4080/api/_bulk", "200", "0.004238128662109375" ], "ecs.version": "1.4.0", "host.hostname": "mba.local", "host.id": "mba.local", "host.ip": "10.10.50.200", "host.name": "mba.local", "log.level": "INFO", "log.logger": "opensearch", "log.origin.file.line": 235, "log.origin.file.name": "base.py", "log.origin.function": "log_request_success", "log.origin.module": "base", "log.original": "POST http://127.0.0.1:4080/api/_bulk [status:200 request:0.004s]", "log.process.name": "MainProcess", "log.process.pid": 36416, "log.thread.id": 4298327424, "log.thread.name": "MainThread", "message": "POST http://127.0.0.1:4080/api/_bulk [status:200 request:0.004s]", "stack_info": "" } }
If you see the Message, you'll see this is the status return from Elastic and Open Search.

"message": "POST http://127.0.0.1:4080/api/_bulk [status:200 request:0.004s]",

[vduseev]

Hello!
Thank you for opening this issue.

The root cause of this situation is in the implementation of the opensearch-py library which we rely on to talk OpenSearch instance. This library logs its own POST/GET/etc requests when global log level or its own logger level are set to INFO.

You should be able to turn this behavior off not just for opensearch-py library but for any other library that uses logging module internally by setting up a global configuation for the logging module in your application.

opensearch-py, in particular, creates a logger with name opensearch, so we need to disable it in order to stop these logs from being collected.

import logging.config

LOGGING = {
    "disable_existing_loggers": True,
    "handlers": {
        "opensearch": {
            "level": "INFO",
            "class": "opensearch_logger.OpenSearchHandler",
            "index_name": "my-logs",
            "extra_fields": {"App": "test", "Environment": "dev"},
            "hosts": [{"host": "localhost", "port": 9200}],
            "http_auth": ("admin", "admin"),
            "http_compress": True,
            "use_ssl": True,
            "verify_certs": False,
            "ssl_assert_hostname": False,
            "ssl_show_warn": False,
        },
    },
    "loggers": {
        "opensearch": { "level": "ERROR", "propogate": False, "handlers": [] },
        "root": {
            "handlers": ["opensearch"],
            "level": "INFO",
        },
    },
}

logging.config.dictConfig(LOGGING)

Notice usage of "disable_existing_loggers": True, and "opensearch": { "level": "ERROR", "propogate": False, "handlers": [] }.

Let me know if that helps!

[projx]

Awesome thank you - you showed me the problem, I was not passing a name when calling logging.getLogger()... I added that and the problem went away.

[vduseev]

Great! Glad to hear that 👏