Transition whatwg.org to new hosting
Closed this issue · 40 comments
To improve our uptime and allow for more admins, we are moving the hosting of whatwg.org from DreamHost to DigitalOcean. These are the domains:
- blog.whatwg.org
- books.idea.whatwg.org
- books.spec.whatwg.org (redirect)
-
build.whatwg.org(will be left alone in this issue) - c.whatwg.org (redirect)
- compat.spec.whatwg.org
- console.spec.whatwg.org
- developer.whatwg.org (redirect)
- developers.whatwg.org (redirect)
- dom.spec.whatwg.org
- domparsing.spec.whatwg.org (redirect)
- encoding.spec.whatwg.org
- fetch.spec.whatwg.org
- figures.idea.whatwg.org
- figures.spec.whatwg.org (redirect)
- forums.whatwg.org
- fullscreen.spec.whatwg.org
- help.whatwg.org (redirect)
- html-differences.whatwg.org
- html.spec.whatwg.org
- idea.whatwg.org
- images.whatwg.org
- infra.spec.whatwg.org
- lists.whatwg.org
- javascript.spec.whatwg.org (redirect)
- mediasession.spec.whatwg.org (redirect)
- mimesniff.spec.whatwg.org
- n.whatwg.org
- notifications.spec.whatwg.org
- quirks.spec.whatwg.org
- resources.whatwg.org
- spec.whatwg.org
- specs.whatwg.org (redirect)
- storage.spec.whatwg.org
- streams.spec.whatwg.org (check content-type of https://streams.spec.whatwg.org/demos/resources/web-animations.min.js.map)
- svn.whatwg.org
- url.spec.whatwg.org (needs audit of extension-less files)
- validator.whatwg.org (redirect)
- webvtt.spec.whatwg.org (redirect)
- whatwg.org and www.whatwg.org (needs audit of extension-less files)
- wiki.whatwg.org (but: #27)
- xhr.spec.whatwg.org
- xn--7ca.whatwg.org (redirect)
Separate from the hosting question, these are static sites that could be hosted inside https://github.com/whatwg/whatwg.org/:
- help.whatwg.org
- images.whatwg.org
- n.whatwg.org
- resources.whatwg.org (already in https://github.com/whatwg/resources.whatwg.org, leave alone?)
I'd like @sideshowbarker's opinion on everything help.whatwg.org.
I'd be okay with merging whatwg/resources.whatwg.org into whatwg/whatwg.org at some point, though leaving it alone for now seems fine too? Seems somewhat more complicated if we want to keep the history intact.
For the two others I created whatwg/whatwg.org#52.
As a next step I think we should try to move everything that's not *.spec.whatwg (so we will move spec.whatwg), blog.whatwg, or wiki.whatwg. Then we could move blog.whatwg and wiki.whatwg and maybe html.whatwg (it seems like we'll continue to have a need to execute scripts so we might want to make a longer term plan first), and last all the specs (that are all intertwined I think).
Yeah, let's leave resources.whatwg.org alone, because it's used by HTML it might get more traffic, so might not want it deployed in the same place. Will you put images.whatwg.org and n.whatwg.org into the repo though? We can set up the deploy for that before DNS is changed.
Oops, that's whatwg/whatwg.org#52
help.whatwg.org can just be redirected
resources.whatwg.org is a tricky one, because it's used by all specs. If it's not on the same IP and using the same cert as the specs, then load time will be less awesome than it could be. Maybe we should try hosting everything static on a single machine, and just make it beefier?
I think we should host all static things on the same server instance. But note that any performance benefits won't come until Let's Encrypt actually offers multi-domain wildcard certificates.
We could also request a single cert for all the domains on a single machine.
The downside is that if the DNS of any one of those domains changes, I think certbot renew would start failing.
The current setup for spec.whatwg.org (nested inside whatwg.org source-wise) leads to fun:
https://whatwg.org/specs/web-forms/current-work/
https://spec.whatwg.org/web-forms/current-work/
These both work, but the latter looks like an accident. I suggest that we just let those URLs go 404, and preserve only the original URLs.
Yes, we should do that.
I'm extracting out spec.whatwg.org now.
Handing html-differences in whatwg/html-differences#5
I've added lists.whatwg.org to the list, it's currently broken by HSTS, and it would be nice to restore it somehow.
@foolip I suspect we can't get ownership of that until we have DNS control since it's highly likely a name that's tightly coupled to DreamHost's managed mailing list service. (When we take that over we should make sure that whatwg@lists.whatwg.org still works. Same for whatwg@whatwg.org.)
Right, so the challenge is how to get the archives from DreamHost and, presumably, to keep updating them if we're not shutting down all of the lists.
Well, the main list we'd end up keeping is also archived at https://lists.w3.org/Archives/Public/public-whatwg-archive/. If we can it might be better to just find a mapping from the old URL to those URLs and redirect everything there. The alternative would be to install the same mailing list software (unless we want to deal with local redirects).
That would work, if https://lists.w3.org/Archives/Public/public-whatwg-archive/ contains everything that our own archives do. Still probably a lot of work.
It should be complete, but yes.
@domenic, is any action needed for build.whatwg.org? At the very least we shouldn't mess up its DNS in eventual DNS handover.
It might be a good idea for me to move that all from my personal AWS to one of the droplets. It'd also be good to switch to LetsEncrypt.
If you're up for it, I'd appreciate the help doing so; the readme at https://github.com/domenic/wattsi-server should have most of the info. Then you could switch the DNS to point to the droplet and I could shut down my server completely.
Eventually we probably want to move that to a whatwg/build.whatwg.org repository.
I've added it to the list so that it's not forgotten.
What's left blocking HTML is now print.pdf. Should try to make https://github.com/izh1979/makepdf part of the Travis build. Thanks @izh1979 for putting it together!
So I'm thinking maybe we could just build/install https://github.com/izh1979/makepdf on Travis for each commit?
The other alternative would be to make it part of build.whatwg.org, but then we'd have to worry about malicious input, and not just processing what gets merged into the spec.
The more we can outsource to Travis the better I think.
Yes, putting it in the Docker stuff for CI build is my plan. It sounds like I should work on that sooner rather than later.
Yeah, otherwise we'll have to either run PHP on the machine serving HTML, or on some other machine that accepts the PDF and deploys it.
Getting the pdfbuild repo working on Travis isn't trivial, but is probably possible in the end.
Or you can rewrite the script in some other language. ;-) It's not big, nor complex.
And it's easier when you have working variant. ;-)
@izh1979 @foolip where would said script be located? Or is this related to whatwg/html-build#120 (comment) & whatwg/html-build#122
Lastly, what is http://sgr-a.ru/h/whatwgpdf.php ?
P.S. @annevk 👍 for PDF'ing the standards.
Thanks in advance!
@snuggs the script used for the HTML Standard is hosted at https://github.com/izh1979/makepdf and that .php page runs it.
Well, this is amusing, while trying to figure out how to do this, I came across https://www.princexml.com/forum/topic/2666/using-better-compression-and-pdf-size-optimization which is @izh1979 asking about compressing PDFs :)
Yeah, I've generating PDFs for WHATWG HTML spec for about of 3 years. ;-)
All that remains here now is lists.whatwg.org, which is tied together with the DNS. Somebody needs to learn how email actually works to do this job, maybe we should liaison with the IETF :)
An update on lists.whatwg.org. https://packages.debian.org/sid/mailman3-suite looked very promising, but is only in Debian unstable, and installation didn't actually work out of the box on a fresh Debian unstable, neither an (upgraded) DigitalOcean droplet or in a debian:unstable Docker image. (I think it failed to create/start a postgres db, and is probably easy-ish to fix.)
Because Debian unstable also doesn't get security updates in the same way, it's probably best to follow http://docs.mailman3.org/en/latest/prodsetup.html or similar, and 213da82 is what I tried so far. With some more tinkering it's probably possible to make that work.
I haven't tried https://github.com/maxking/docker-mailman yet because it seems somewhat implausible to me that the total complexity will be lower, one still has to have a database and web server separately, and we don't want archives, so it'd pretty much just give us mailman3-core and the django stuff.
On setting up an email server (not just mailman), something very discouraging at https://www.digitalocean.com/community/tutorials/how-to-set-up-a-postfix-e-mail-server-with-dovecot:
Postfix has several hundred configuration parameters. If you want to administer a mail server that reliably delivers business requirements to a sizable organization, you should make yourself intimate with all of them (man 5 postconf). This tutorial will not be enough, on its own, to make you a competent professional email admin.
Oh, wow, thanks for looking into this further. Can you explain a bit more about the docker-mailman not working? It seems to have the database inside the container.
I haven't tried setting it up the Docker image. From my reading of https://hub.docker.com/r/alexbarcelo/mailman3/ last time, given the POSTGRES_HOST configuration I surmised that the database would be external, but maybe it'll use an internal one if not.