Split up infrastructure page

[h-vetinari]

I'm looking at our infrastructure page, and it's pretty long and unwieldy.

I think it should be split up, possibly into several different pages. At the very least, I'd like to split it into stuff that's recipe-relevant (compilers, sysroot, CDTs), and stuff that regular maintainers need much less often (website, metadata, CI setup, etc.).

Other possible things that might deserve their own pages:

• bot commands
• staged-recipes
• "special package handling" (i.e. repodata patches, admin requests, feedstock-outputs)
• everything regro-related

Thoughts @conda-forge/core?

CC @zklaus for recent efforts on these pages.

[jaimergp]

Last year, I experimented with how Diátaxis would apply to our docs in https://cf-infra-docs.netlify.app/docs. Some of the docs written for that experiment are being migrated over to our new website. @zklaus is almost finished, I think, with that effort.

I fully support the splitting. Here are some unordered thoughts that might make sense along with these efforts:

• I think it would be beneficial to have the "life cycle" published as a standalone documentation entry because it gives a big picture of the whole infra. And then more details can be filled in in specific pages.
• The bot-commands could be autogenerated from the conda-forge-webservices code similar to what we have done with the conda-forge.yml JSON schema.
• If there's code out there emitting documentation, we should not rewrite those in the website. Either make it programmatically renderable in the website, or just point to the external documentation from the place(s) that make(s) most sense.

Also I think we need to distinguish between the infra that supports running a single build job, and the infra that supports orchestrating build jobs and automation. The line gets blurry at times, but I think it's worth trying it.

[jakirkham]

It is worth noting that we link to the admin bot requests all over the place. So if there is some way to forward the old links to whatever new page they live on, that would be quite helpful

[h-vetinari]

Last year, I experimented with how Diátaxis would apply to our docs in https://cf-infra-docs.netlify.app/docs. Some of the docs written for that experiment are being migrated over to our new website. @zklaus is almost finished, I think, with that effort.

That looks very nice! I think we should not only take over the content, but also start structuring things in a similar way.

I think it would be beneficial to have the "life cycle" published as a standalone documentation entry because it gives a big picture of the whole infra. And then more details can be filled in in specific pages.

Sounds great - i.e. a big overview that branches off into separate little topic pages. I think in terms of diataxis quadrants that would be "Explanation" AFAICT, and very useful at that. We're lacking in that category a bit IMO (e.g. people struggle a lot with understanding the different roles of build: & host: environments; I have been wanting to write something up about that for a long time).

[h-vetinari]

Hm, this turned out to be a way deeper rabbit hole than I wanted to dive into 😅

First, a proposal only for Reference:Infrastructure (topic of this issue):

[...]
├ Reference
| ├ Infrastructure
| | ├ Staged recipes            # including teams
| | ├ Compilers
| | ├ Other core packages       # sysroots, CDTs, msys2, ...
| | ├ CI Setup                  # docker images, conda-forge-ci-setup
| | ├ CI providers
| | ├ Global pinning
| | ├ Feedstock evolution       # conda-smithy
| | ├ Feedstock metadata        # graphs, feedstocks feedstock-outputs
| | ├ Metadata corrections      # repodata-patches, admin-requests
| | ├ Bot repos                 # all things regro
| | ├ Website
| | └ ...
| ├ Bot commands
| └ ...
└ ...

In order to orient myself, I however had to look at the bigger picture, so here's a suggested page structure based on the existing docs and the @jaimergp's experiment:

Overall structure proposal

	vs. docs	vs. Infra Exp.	Comment
Welcome	== here	== Welcome
For users	== here	n/a	could surely be improved...
└ ...			... but not in scope here
For maintainers	== here	n/a
├ Tutorials	partially here	== Tutorials
| ├ Generate a conda recipe			Using greyskull or whatever
| ├ Write your first recipe			Segue from first tutorial, make some changes
| ├ Submit your first recipe		~= placeholder	Segue again, process for submission
| └ ...
├ How-to guides	distributed all over	== How-to guides
| ├ Easy
| | ├ Read a recipe			Component overview, refer to in-depth explanation
| | ├ Rerender feedstock
| | ├ Test packages
| | ├ Deal with numpy	roughly here
| | ├ Enable osx-arm64	roughly here		Same for arches
| | ├ Talk to the bots			refer to bot command reference page
| | └ ...
| ├ Medium
| | ├ Cross-compile	~= here
| | ├ Enable CUDA	~= here
| | ├ Multi-output recipes	~= here
| | ├ LTS branches	~= here
| | └ ...
| ├ Emergencies
| | ├ Patch repodata
| | └ Mark a package broken	~= here
├ Understanding c-f	not much	~= Fundamentals	"Explanation" part of diataxis
| ├ Life-cycle of a pkg		== here
| | └ ...
| ├ How a pkg gets built
| | ├ Compilation concepts			shared/static/etc.
| | ├ Env. roles	short comment		e.g. build: vs. host:
| | ├ Compilers	partially here		e.g. role of activation
| | ├ Global pinning	~= here
| | ├ Run exports	~= here
| | ├ Jinja functions
| | └ ...
| ├ How our bots work		see here
| | ├ Migrations	~= here
| | ├ regro infra
| | └ ...
| ├ Security	~= here	see placeholder
| | ├ Output validation
| | └ ...
| ├ Maintainer roles	roughly here & here
| └ ...
├ Reference	all over	== Reference
| ├ Infrastructure		~same content split somewhat differently
| | ├ Staged recipes		roughly here	incl. teams
| | ├ Compilers	roughly here
| | ├ Other core packages			sysroots, CDTs, msys2, ...
| | ├ CI Setup			docker images, conda-forge-ci-setup
| | ├ CI providers		roughly here
| | ├ Global pinning	~= here		+explain migrators
| | ├ Feedstock evolution			conda-smithy
| | ├ Feedstock metadata			graphs, feedstocks, feedstock-outputs
| | ├ Metadata corrections			repodata-patches, admin-requests
| | ├ Bot repos			all things regro
| | ├ Website		== here
| | └ ...
| ├ Bot commands	~= here	== here
| ├ Feedstock settings	mostly here	== here	conda-forge.yml, CBC, migrators
| ├ Glossary	== here	== here
| ├ Current pins		== here	Auto-generated / subset?
| ├ Ongoing migrations			refer to status page
| └ ...
├ Miniforge		== here
└ FAQ	~= here		separate because there's no clean split into how-to & reference...?

[zklaus]

I really like this! One of the great things about Diátaxis is that it's explicitly build to not force you to do everything in one fell swoop. So moving in this direction with a plan as posted by @h-vetinari as a general guideline to follow sounds great to me.

With #2158, my transfer of content into the infrastructure page will be concluded, so I am all for moving forward in the proposed direction.

[jakirkham]

With #2158, my transfer of content into the infrastructure page will be concluded, so I am all for moving forward in the proposed direction.

This was merged earlier today. So it sounds like this issue is wide open

[h-vetinari]

I opened a more broadly-scoped issue about the overall structure proposed above in #2164. I'll bring it up in the core call this week to see if people are on board with the overall direction. Once we have a Reference section, it would be easier to move things, I think.