Syntax highlighting changes have broken all examples in the HTML spec

[domenic]

See e.g. the examples in https://html.spec.whatwg.org/multipage/webappapis.html#realms-settings-objects-global-objects

All the <pre> blocks are now empty :(.

On CI, I am getting the following sorts of errors:

Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 652, in __init__
    self.handle()
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 328, in handle_one_request
    method()
  File "/whatwg/build/highlighter/server.py", line 25, in do_GET
    html,css = highlight(data, lang=lang, output="html", unescape=True)
  File "/whatwg/build/highlighter/highlighter/highlight.py", line 25, in highlight
    html = highlightEl(html, lang)
  File "/whatwg/build/highlighter/highlighter/highlight.py", line 76, in highlightEl
    coloredText = highlightWithPygments(text, lang)
  File "/whatwg/build/highlighter/highlighter/highlight.py", line 159, in highlightWithPygments
    from .pygments import pygments
  File "/whatwg/build/highlighter/highlighter/pygments/pygments/__init__.py", line 30, in <module>
    from pygments.util import StringIO, BytesIO
ImportError: No module named pygments.util

I can also reproduce these locally by doing ./build.sh -d. So, something about the Docker environment.

I think I'll try rolling back for now?

/cc @sideshowbarker

[domenic]

Separately we may want to make the error case not output empty <pre>s, but instead the original text content.

[domenic]

Reopening to track progress on re-enabling this...

[domenic]

I have a fix for this: if I insert

ENV PYTHONPATH="/whatwg/build/highlighter/highlighter/pygments:${PYTHONPATH}"

into the dockerfiles, it seems to work. Should we just do that? Seems like something is misconfigured though. /cc @tabatkins

[tabatkins]

Sigh, yeah, this is a bug I reported on Pygments. Pygments itself is written incorrectly, attempting to import its own submodules globally rather than with local imports. See tabatkins/highlighter#5. I wasn't aware that this was more widely problematic, tho. I'll try and patch some more of their includes.

[tabatkins]

Ah, I see what the problem is. @sideshowbarker MikeSmith had Pygments globally installed, so his environment still worked properly when it did some global import. Your CI doesn't, so the badly-done imports just straight-up fail.

[sideshowbarker]

Separately we may want to make the error case not output empty <pre>s, but instead the original text content.

Yeah I thought I already had it doing that actually. But clearly what I implemented for that isn’t working

[sideshowbarker]

@sideshowbarker MikeSmith had Pygments globally installed, so his environment still worked properly when it did some global import

Ah yeah. I’ve removed it now

[sideshowbarker]

I have a fix for this: if I insert

ENV PYTHONPATH="/whatwg/build/highlighter/highlighter/pygments:${PYTHONPATH}"

into the dockerfiles, it seems to work. Should we just do that?

Yeah we should. And we should do the equivalent for the build.sh file, so that it will also work when the build is run locally

[tabatkins]

Ugh, these includes are pervasive, and they used the bad practice of actually putting significant code in their __init__.py files, so that's more rewriting I have to do. This'll take a little bit, definitely do the hotfix for now.

[domenic]

I'm not super-happy about the hotfix because it would permanently modify users' systems (by changing their PYTHONPATH). On CI that's not a problem, but if we're wanting to modify build.sh so people can build locally, it is.I am dumb, that's not how exporting variables works.

I tried inserting

sys.path.insert(0, os.path.join(os.path.dirname(os.path.realpath(__file__)), "highlighter", "pygments"))

into server.py, which supposedly should have the same effect as the above, but it doesn't...

[tabatkins]

The better hotfix is to give Pygments what it wants for now - a globally-installed version. Just pip install pygments in your Travis setup. We know that this works.