Make charset detection in `.text()` optional
Closed this issue · 2 comments
The fetch spec demands the UTF8 decode algorithm to be applied to incoming bodies. This basically strips a BOM & then proceeds to decode the content as UTF8, without any charset detection. Chrome & Firefox both implement this. (Test case: fetch('http://kreata.ee/iso-8859-15.php').then(r => r.text()).then(t => console.log(t));)
In contrast with the spec, this library currently uses charset detection on any call to text(). This introduces an inconsistency between server-side & client-side fetch. It also has a performance cost.
To remedy this, I am proposing to remove charset detection entirely. While supporting decoding with an optional parameter (as in text({ decode: true })) is tempting, it would not address the issue of server / client inconsistencies. Where needed, charset detection should be implemented on top of the .arrayBuffer() or .blob() interface, so that it can be used consistently in both node & browsers.