Issue with some characters

[slorg1]

hi,

I isolated an issue when using for example:

u'& &'

In an object. Then the key-value pair is missing in the hash (when I look in redis). By missing, I mean that when the object is persisted, that key-value pair (where the value is & & does not get written). As a result, the (partially) written object cannot be read.

I tested manually and redis has no issues with it. If I input the value directly in redis, loading the object that was previously crashing now loads.

It seems to be an issue only on write. I am not sure where the issue comes from, though. There is no error or anything written. It seems like the key-value just gets "skipped".

This is the configuration of the column:
rom.Text(required=True, index=True, keygen=rom.FULL_TEXT, prefix=True,)

It does seem also that these characters & get dropped by the indexer. Is it possible that it gets skipped because each character of the string got skipped by the indexer?

The error only manifests on read: then the object is incomplete and cannot be loaded.

Please let me know if you have any thoughts and/or if you can help.

Thank you in advance.

[slorg1]

Hi,

Actually, I might have found the code in question:

model.Model._apply_changes L 301:

            if ca._keygen and not delete and nval is not None and (ca._index or ca._prefix or ca._suffix):
                generated = ca._keygen(attr, new)
                if not generated:
                    # No index entries, we'll clean out old ones as necessary
                    continue

This will skip any indexed value considered to be empty. Since util.FULL_TEXT wipes away punctuation: it returns the empty string.
Here the if checks a wide "falsy" value. It might make sense to differentiate empty from None here.

As a result, L372 and 373 never run which does not send that pair to the writer while it is in redis_data. This explains why it takes to subsequently read the object to see the error: data and redis_data are accidentally different here.

Thank you.

[josiahcarlson]

You have found the issue. The fix is simply the replacement of continue with pass.

[josiahcarlson]

Thank you for finding and diagnosing the issue :)

[slorg1]

Thank you for the fix!