docs-tools holds all common build system components for MongoDB
documenting projects (e.g. the MongoDB Manual, MongoDB Ecosystem
Documentation, etc.) These tools include: Sphinx extensions, the
meta-build system, themes, deployment and orchestration scripts, and
scripts that generate common restructured text elements.
The goal of docs-tools is to totally remove all programs from the
documentation source trees to facilitate maintenance of these
components in the context of a multi-branch and multi-repository
documentation project.
This document explains the components and operations of the build system, and serves as a guide for anyone who wants to maintain a MongoDB Documentation-like project.
fabric is a deployment orchestration system
that provides simple and convenient Python interfaces for shell
scripting and task running. docs-tools uses fabric tasks for two
purposes:
- to deploy build products to testing and production environments.
- to bundle groups of related operations in a consistent and simple interface.
Historically, these operations were a mixture of multi-step makefile targets, shell scripts, and Python programs. Combining all of these operations into a single interface increases portability, consistency, and overall reliability of the build systems.
Makecloth is a
meta-build tool that generates build specifications
(e.g. Makefiles and ninja.build files.) Makecloth makes it
possible to take advantage of the dependency resolution and
concurrency features of a robust and stable buildsystem while mostly
writing Python rather than a build tool specific specification.
In the docs-tools implementation, Makecloth scripts generate
buildscripts based on project-specific data supplied with the source
data, so that many projects can use common logic. See the build data
for the MongoDB Manual
for an example.
rstcloth use the model of
makecloth but generates reStructuredText: In some situations, it
makes sense to generate rst programatically rather than maintains
that content manually.
For example:
- to make tables, it often makes more sense to generate the table based on more generic data than to build the table directly in restructured text, given the flexibility of the table syntax.
- to keep multiple related pieces of content in sync, it often makes sense to generate when two pieces of content derive from the same source but need different reStructuredText forms.
- to enforce common formatting requirements centrally for forms that appear throughout the documentation.
This approach allows minimally invasive generation of output for multiple formats and reduces the amount of custom template logic.
This directory contains conventional Sphinx extensions, required for producing and referencing MongoDB documentation projects. This includes the extensions for the MongoDB Domain to cross reference MongoDB objects (database commands, shell methods, database output, etc.) as well as custom admonition and directive types.
These are standard Sphinx themes customized for MongoDB documentation projects, centralized here for ease of use across projects.
dtf is a consistency testing framework for documentation projects to provide ways of enforcing conventions in documentation. The content of these scripts just include the test logic, and not the testing running architecture, or data used to define testable situations.
The scripts in the bin directory include common components and
logic shared between bootstrapping operations, fabric scripts, and
meta-build scripts.
bootstrap.py
is an example bootstrapping script that intializes a docs-tools
based repository.
See the mongodb manual repository
for an example of an integrated repository using
docs-tools. In particular, note the build_conf.yaml
<https://github.com/mongodb/docs/blob/master/config/build_conf.yaml>
file: this is the only strict requirement for docs-tools.
Copy the bootstrap.py
script in the top level of your repository and create a
docs_meta.py file in the bin/ directory of your
repository. You can now run the following command to bootstrap your
build system:
python bootstrap.py
To begin using the meta-build system, specify a generated makefile in
the build.system.files list in the docs_meta.py file. This
will correspond to one of the scripts in the makecloth/ directory
of docs-tools. Then in your makefile, include the following
bootstrapping content:
output = build
tools = $(output)/docs-tools
.PHONY:$(output)/makefile.meta
-include $(output)/makefile.meta
build/makefile.meta:$(tools)/makecloth/meta.py
@mkdir -p $(output)
@python $< $@
Modify the tools variable if you downloaded docs-tools to an
alternate location. You may also include the following bootstrapping
helper target:
bootstrap fabfile build/docs-tools:
@python bootstrap.py
@echo "[bootstrap]: configured build environment."
Some of the makecloth files require a corresponding yaml file in
the directory specified in the build.paths.builddata field of
docs_meta.yaml.
To generate the run make without any arguments.
There are several ways to add functionality to the build system:
- add additional tasks to the
fabfilemodules according to standard fabric development practices. - add additional rstcloth generated files. These scripts typically take inputs via arguments on the command line or a file passed in on the command line. For integration, add corresponding makecloth scripts to generate build instructions.
- add additional makecloth files for additional processing and build
products. Add generated makecloth files to the
build.system.filesindocs_meta.yamlto generate a new makefile.
- improve documentation of each makecloth's purpose and use.
- expand documentation of makecloth data forms and schema.
- improve symlink handling throughout makecloth (for windows compatibility.)
- stabilize makecloth and rstcloth and move them out of the tree.
- build helpers for working with development virtualenvs.
- make build system fully Python 3 compatible.