Better/more flexible table of contents for serial documents

[ktbartholomew]

There are a number of problems with the way we currently generate the global table of contents for multi-page documents:

1. It requires the creation of a "magic" file named _toc.rst
2. It causes spurious 404 errors between the presenter and content service because we've coded the possible existence of _toc.rst as a special case.
3. From Sphinx's perspective, _toc.rst is a single static file at the top level of a document, when in practice it is a set of links that accompanies every page in the document.

This issue is the engineering counterpart for how to solve the problem described in rackerlabs/docs-core-infra-user-guide#408.

[smashwilson]

I'm hoping to introduce a mechanism to help this as part of deconst/content-service#51: when a bulk upload is performed, if a metadata/toc.html file is present, it'll be used as a toc for all envelopes uploaded as part of that bundle.

I'm thinking that we should have separate local_toc and group_toc attributes for the TOCs generated as part of each page and uploaded with the batch, respectively, and keep toc around at least temporarily as it is for backwards compatibility.

It'd be better to have the preparer generate the toc.html file automatically rather than relying on the content repository to have a _toc.rst file, too.

[ktbartholomew]

I think it would be more in keeping with the content envelope philosophy to generate the global toc as a property of every envelope submitted within the document:

{
  "toc": "lotsofHTML",
  "body": "moarHTML",
}

This was abandoned when I first looked at this problem because it wasn't immediately clear how to make Sphinx behave this way.

I'm having bad feelings about (my) previous decision to fetch the ToC separately. It's too much of a special case and leads to unnecessary complexity in the presenter. (e.g: "ignore a 404 error iff it was a toc request)

If we are still going to fetch additional documents, I would like the purpose of those documents to be arbitrary, and not specific to the purpose of a ToC. The repository metadata might include an array of addenda, where each item is a specific content ID that should be loaded in addition to the main content envelope for the request.

[smashwilson]

Oh, yeah, we could just do the TOC injection as part of the envelope preparation - I like that.

Getting rid of that extra fetch is 👍

I'd still say to call the element global_toc because, at least in Sphinx, we have a TOC for the entire repository (_toc.rst) and one for the local page (the toc element we have now). This would let you use either in a template. Unless you'd say there's no reason to ever use the local page TOC when you have a global one?

[smashwilson]

And calling it toc would make this a purely Sphinx preparer concern - hah.

[ktbartholomew]

Well, which to use and whether to have both global and local seems like a presentation concern. If the information is there, why not have both?

The problem with a truly global TOC that exists as a single envelope is that it either

1. Needs root-relative links or a way of identifying where it exists in the presented site (see deconst/deconst-docs#105)
2. Needs to be generated on a per-page basis to produce relative links that actually work

[ktbartholomew]

My proposal for "addenda" could make this not a Sphinx concern, if we still end up storing the TOC as a separate envelope. I'm imaging an envelope like this:

{
  "meta": {
    "addenda": {
      "global_toc": "https://github.com/org/repo/toc/",
      "other_cool_thing": "https://github.com/org/repo/cool-thing/"
    }
  }
}

...that is then referenced in templates like this:

<div id="global-toc">
{{ deconst.content.envelope.meta.addenda.global_toc.body }}
</div>

[smashwilson]

Oh, like a general way to embed some envelopes in other ones by content ID? Yeah, that'd work.

👎 on stashing it in "meta" though - meta is for arbitrary source-to-template communication, while this would need to be processed by the presented (to do the additional document fetches). Let's make it a top-level envelope attribute.

[ktbartholomew]

👍 on 👎 on stashing in meta. (algebraically, that would be a net 0, but it actually lands us at a -2...anyways) I'm that person who puts all my junk in one big pile and "knows" exactly where everything is in the pile. You're the person who shops at Container Store.

[smashwilson]

I've made a start on implementing the preparer side of this in #64.

• To ensure the TOC URIs are accurate, I'm using those {{ to('...') }} content replacement directives I'd put in the presenter a while ago. The presenter will generate absolute URIs that reflect the actual content mappings at page render time, so they'll be guaranteed to stay up to date. That lets us generate one TOC for each content repository and avoids the brittleness of trying to get the relative URIs right everywhere.
• I'm taking your "addenda" suggestion to solve the TOC location problem. It's more general solution that could be useful elsewhere, and it's an excuse to use the word "addenda", which is just cool-sounding. 👍

[smashwilson]

This has shipped 🙌 Tables of contents are now fetched as "addenda" documents.