Teradata/PyTd

Expensive logging calls during `convertValue` degrade large query performance

tetraptych opened this issue · 0 comments

The method convertValue of DefaultDataTypeConverter contains the following line (L230-L231):

        logger.trace(
            "Converting \"%s\" to (%s, %s).", value, dataType, typeCode)

This results in many calls to logger.isEnabledFor, even when configureLogging=False in the application configuration.

For context: I am fetching ~80,000 rows with 30 columns each for a total of 2,400,000 type conversions. Even if each of these logging calls is very fast the time becomes significant given the volume of data. cProfile shows that ~36% of the time in convertValue was spent in util.py:41(trace).

Maybe a fix similar to the one used for issue #37 could work?