Exception while displaying constants section of a query

[seqizz]

Hi,

First of all, thanks for this nice tool,

After clicking the details for a query (page uri is database/my_db_name/query/-118899751760609078/qual/1584481209/), I got following error on the powa-web's log:

    Traceback (most recent call last):
      File "/usr/lib/python3/dist-packages/tornado/web.py", line 1467, in _execute
        result = method(*self.path_args, **self.path_kwargs)
      File "/usr/lib/python3/dist-packages/powa/dashboards.py", line 78, in get
        data = {"data": [self.process(val, **url_params) for val in values]}
      File "/usr/lib/python3/dist-packages/powa/dashboards.py", line 78, in <listcomp>
        data = {"data": [self.process(val, **url_params) for val in values]}
      File "/usr/lib/python3/dist-packages/sqlalchemy/engine/result.py", line 715, in __iter__
        row = self.fetchone()
      File "/usr/lib/python3/dist-packages/sqlalchemy/engine/result.py", line 1028, in fetchone
        self.cursor, self.context)
      File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1344, in _handle_dbapi_exce
ption
        util.reraise(*exc_info)
      File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 186, in reraise
        raise value
      File "/usr/lib/python3/dist-packages/sqlalchemy/engine/result.py", line 1019, in fetchone
        row = self._fetchone_impl()
      File "/usr/lib/python3/dist-packages/sqlalchemy/engine/result.py", line 900, in _fetchone_impl
        return self.cursor.fetchone()
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcc in position 80: invalid continuation byte

The system is debian stretch with following versions:
python 3.5.3
tornado 4.4.3-1
psycopg2 2.7.6.1
sqlalchemy 1.0.15
PostgreSQL 11.3

[rjuju]

Thanks a lot!

I'd need some more information to understand what's happening here. What is the original database encoding, what is pg_qualstats version and do you know which datasource on the qual page is failing?

[seqizz]

Sorry for late reply:

• show server_encoding; returns UTF8, yet detailed settings are:

/etc/postgresql/11/main/postgresql.conf:lc_messages = 'en_DK.UTF-8' # locale for system error message
/etc/postgresql/11/main/postgresql.conf:lc_monetary = 'en_DK.UTF-8' # locale for monetary formatting
/etc/postgresql/11/main/postgresql.conf:lc_numeric = 'en_US.UTF-8' # locale for number formatting
/etc/postgresql/11/main/postgresql.conf:lc_time = 'en_DK.UTF-8'

• qualstats version is 1.0.8
• And for the last question, if I understand correctly, it's the query view > "predicates" section > last 2 columns.

[rjuju]

I actually missed that pg_qualstats is truncating constats to 80B, which is precisely where your error happens:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcc in position 80: invalid continuation byte

Since 0xCC is the first byte of a multi-byte value, you were unlucky enough to hit a bug in pg_qualstats, as we're truncating the constant to 80B without taking care of multi-byte characters. I'll fix that shortly.

In the meantime, I'm afraid that some of the values stored are definitely lost, so you can either delete the corresponding rows, or wait for the retention to purge them.

[rjuju]

The issue should be fixed in pg_qualstats with commit powa-team/pg_qualstats@ee7255d.

Thanks a lot for the report!

[seqizz]

Cool, thanks for the quick fix. Closing then, will test and re-open if something cries.