String encoding breaks in several edge cases

Question

String encoding breaks in several edge cases

ryancdotorg opened this issue 5 years ago · 2 comments

The root cause seems to be the string encoder assuming that any character \ud800 is the first half of a well formed UTF-16 surrogate pair. That assumption fails in the following cases:

Code points U+E000 to U+FFFF
Unpaired surrogates

JavaScript strings are not necessarily well formed UTF-16. The code needs to process characters in the range \ud800 to \udbff by checking whether they are followed by a character in the range \udc00 to \udfff, and if not, encoding U+FFFD instead. Anything \udc00 to \udfff by itself should also be encoded as U+FFFD.

For example, CBOR.encode("\uff08\u9999\u6e2f\uff09") gives 6bf3928699e6b8aff3929080 rather than the expected 6cefbc88e9a699e6b8afefbc89.

Answer 1 · 2020-09-03T21:00:06.000Z

This is resolved at https://github.com/aaronhuggins/cbor-redux.

Answer 2 · 2021-12-12T04:24:15.000Z

This is resolved at CBOR-es in JavaScript (ES Module) with TextEncoder/TextDecoder
https://github.com/code4fukui/CBOR-es/blob/master/CBOR.js