Usage with tqdm

[dr-costas]

Is there any way of using loguru.logger with tqdm?

I've tried many solutions but nothing printed the progress bar of tqdm in one line. In all solutions that I tried, each update of tqdm progress bar was a new line. For example:

  0%|           | 0/100 [00:00<?, ?it/s]
  1%|1         | 1/100 [00:00<00:09,  9.98it/s]
  2%|2         | 2/100 [00:00<00:09,  9.97it/s]
  3%|3         | 3/100 [00:00<00:09,  9.97it/s]

[Delgan]

Hi.

Did you try to replace the default handler with tqdm.write()? It seems to work fine for displaying the progress bar:

from tqdm import tqdm
from loguru import logger
import time

logger.remove()
logger.add(tqdm.write, end="")

logger.info("Initializing")

for x in tqdm(range(100)):
    logger.info("Iterating #{}", x)
    time.sleep(0.1)

[Delgan]

@dr-costas Does this code snippet suits your needs?

[Delgan]

I'm closing this issue as I added the code snippet to the documentation:

• Interoperability with tqdm iterations.

[vmvz]

Hi.

Did you try to replace the default handler with tqdm.write()? It seems to work fine for displaying the progress bar:

from tqdm import tqdm
from loguru import logger
import time

logger.remove()
logger.add(tqdm.write, end="")

logger.info("Initializing")

for x in tqdm(range(100)):
    logger.info("Iterating #{}", x)
    time.sleep(0.1)

Error:
TypeError: add() got an unexpected keyword argument 'end'

[Delgan]

@vmvz Yeah, sorry, this has been changed in v0.4.0 (the .add() method no longer automatically pass extra arguments to sink, because this was prone to error).

Instead, on should use this:

logger.add(lambda msg: tqdm.write(msg, end=""))

I updated the documentation consequently.

[vmvz]

@vmvz Yeah, sorry, this has been changed in v0.4.0 (the .add() method no longer automatically pass extra arguments to sink, because this was prone to error).

Instead, on should use this:

logger.add(lambda msg: tqdm.write(msg, end=""))

I updated the documentation consequently.

Code:

from tqdm import tqdm
from loguru import logger
import time

logger.remove()
logger.add(lambda msg: tqdm.write(msg, end=""))

logger.info("Initializing")

for x in tqdm(range(100)):
    logger.info("Iterating #{}", x)
    time.sleep(0.1)

Run result:

[Delgan]

@vmvz What is your Python version? What are tqdm, colorama and loguru (you can dump the output of pip freeze)?

I tested this snippet on both Linux and Windows, it's working fine on my computer (Python 3.8.1, tqdm 4.43.0, colorama 0.4.3, loguru 0.4.1).

Are you piping the output of your program? Are you using a non-conventional terminal maybe?

Also, you can try to see if the bug appears without using loguru:

from tqdm import tqdm
import time

for x in tqdm(range(100)):
    tqdm.write("Iterating #{}".format(x))
    time.sleep(0.1)

[vmvz]

@vmvz What is your Python version? What are tqdm, colorama and loguru (you can dump the output of pip freeze)?

I tested this snippet on both Linux and Windows, it's working fine on my computer (Python 3.8.1, tqdm 4.43.0, colorama 0.4.3, loguru 0.4.1).

Are you piping the output of your program? Are you using a non-conventional terminal maybe?

Also, you can try to see if the bug appears without using loguru:

from tqdm import tqdm
import time

for x in tqdm(range(100)):
    tqdm.write("Iterating #{}".format(x))
    time.sleep(0.1)

System: MacOS 10.14.6
Python: 3.7.6
tqdm==4.43.0
loguru==0.4.1
colorama (not installed)

[Delgan]

@vmvz Do you agree this is a problem with tqdm.write() and not loguru itself? It would be interesting to know what the developers of tqdm think about it, may worth opening a issue on their bug tracker: https://github.com/tqdm/tqdm/issues

[hollowgalaxy]

@vmvz worked for me with versions
python==3.7.6
tqdm==4.42.1 and 4.43.0
loguru==0.4.1
Ubuntu 18.04

@Delgan is there a way to run the above example without the remove? The code snippet above has the consequence of removing any other logger.add prior to the snippet.

[Delgan]

@hollowgalaxy I used logger.remove() to avoid duplicate logs sent to sys.stderr. You can solely remove the default handler (which automatically logs to sys.stderr) with logger.remove(0) (0 is the identifier of the handler).

[cgebbe]

Thanks for the workaround of logger.add(tqdm.write)! Unfortunately, this disables the coloring of the loguru log messages :/. Any ideas on how to keep this coloring?

[Delgan]

@cgebbe You can specify colorize=True while adding the sink to force colorization of messages.

[StringKe]

I encountered a similar problem, try this in the new version。

Just set it once in your code entry (__main__)

from loguru import logger
from tqdm import tqdm

logger_format = "<green>{time:YYYY-MM-DD HH:mm:ss.SSS}</green> [<level>{level: ^12}</level>] <level>{message}</level>"
logger.configure(handlers=[dict(sink=lambda msg: tqdm.write(msg, end=''), format=logger_format, colorize=True)])

[maoding]

Thanks for the advice. I have an additional question.

Does it make sense to split sinks into tqdm and non-tqdm related sinks and use bind to decide which one to use?

Usecase: Assume we have multiple modules and some use tqdm and others not.

import time

from loguru import logger
from tqdm import tqdm
import sys

logger.remove()
logger.add(
    lambda msg: tqdm.write(msg, end=""),
    colorize=True,
    filter=lambda record: "special" in record["extra"],
)
logger.add(sys.stderr, filter=lambda record: "special" not in record["extra"])

# Let's say log in module A without tqdm
logger.info("Initializing")

# Logs in module B with tqdm
for x in tqdm(range(10)):
    logger.bind(special=True).info("Iterating #{}", x)
    time.sleep(1)

Of course there is always the disadvantage using bind(special=True if you are in tqdm context. If you forget this the status bar will be a mess again.

Is there any advantage using this approach, e.g. better performance for logs in Module A?
Or is this simply boilerplate?

[Delgan]

@maoding I would not advise using two sinks, you can do it with one unique sink:

def sink(msg):
    if "special" in record["extra"]:
        tqdm.write(msg, end="")
    else:
        sys.stderr.write(msg)

logger.add(sink)

Regarding performances, I'm not sure. It depends if tqdm.write() adds a lot of overhead compared to the dict look-up using a custom sink.