buriy/python-readability

Issue with utf8 and HTML entities

uuencode opened this issue · 2 comments

uuencode commented

I am using the latest pip version - readability-lxml 0.8.1 and I found a curious issue. When there are both non-ascii utf-8 chars and HTML entities the output is not properly utf-8 encoded.

<!DOCTYPE html>
<html><head>
<title>title</title>
<meta charset="utf-8" />
</head><body>
This is déjà vu &hellip;
</body></html>

doc.summary() returns This is déjà vu …

&hellip; being converted beforehand to …
doc.summary().encode('raw_unicode_escape').decode('utf-8') returns This is déjà vu \u2026

It is very common to have both non-ascii utf-8 and HTML entities together.
As the output is HTML anyway, leaving entities unprocessed could be a solution.

Thank you.

uuencode commented

Found another issue that mentioned Document(response.content) should be used instead of Document(response.text) and that fixed it.

#163

A good idea to update the readme.

  • ❤️1
buriy commented

Thanks! Updated readme!