piskvorky/sqlitedict

UnicodeDecodeError

Closed this issue · 6 comments

dineshbvadhia commented

I've been using sqlitedict for years, first with py2.7 and now py3.x, without any problems. However, using sqlitedict within async code the error below occurs when opening. Serialization with the default pickle causes the same error.

with SqliteDict(filename, flag="r", journal_mode="OFF", encode=msgpack_serialize, decode=msgpack_deserialize) as f:

<class 'UnicodeDecodeError'> 'utf-8' codec can't decode byte 0x97 in position 1916: invalid start byte

To confirm it is a sqlitedict problem, the data was saved and opened as json dump and json load and it works fine.

Any thoughts?

piskvorky commented

Hi, do you have some repeatable minimal example?

dineshbvadhia commented

The minimal example works with and without the async code. The above error persists in the larger application.

Is it possible to knock-out the multi-threaded support to see if this narrows the problem area ?

piskvorky commented

A minimal failing example will be a great start, so we can replicate and confirm the issue (and later, its resolution).

dineshbvadhia commented

The minimal example is not failing :(

piskvorky commented

Then it's not a minimal failing example :)

Until we have a reproducible error, with a proper full stack trace, this will be hard to crack.

dineshbvadhia commented

SqliteDict doesn't work with the async framework.

A single-threaded kv store built on top of sqlite3 works with the async framework.

The async framework folks are looking into it.

Don't get your knickers in the twist. When there is something to report, it will be.