Codec encoding errors in OutputSplitter

[cBog]

I was trying to extract from https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2 on Ubuntu.

I got an error in the OutputSplitter:

INFO: Using 64 extract processes.
Process ForkProcess-1:
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "{redacted_path}/lib/python3.6/site-packages/wikiextractor/WikiExtractor.py", line 494, in reduce_process
    output.write(ordering_buffer.pop(next_ordinal))
  File "{redacted_path}/lib/python3.6/site-packages/wikiextractor/WikiExtractor.py", line 175, in write
    self.file.write(data)
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2013' in position 2809: ordinal not in range(256)

As I understand, the default file system codec can vary between OSs. I can force it to use unicode by setting LC_CTYPE=C.UTF-8, but the following change I found also works:

diff --git a/wikiextractor/WikiExtractor.py b/wikiextractor/WikiExtractor.py
index ff63783..f6aea70 100755
--- a/wikiextractor/WikiExtractor.py
+++ b/wikiextractor/WikiExtractor.py
@@ -181,7 +181,7 @@ class OutputSplitter():
         if self.compress:
             return bz2.BZ2File(filename + '.bz2', 'w')
         else:
-            return open(filename, 'w')
+            return open(filename, 'w', encoding='utf-8')


 # ----------------------------------------------------------------------

I don't have particularly high confidence in my understanding of this. Is there a reason not to make that change to prevent such issues for others?