Improve RTD build process

Question

Improve RTD build process

oscarmcm opened this issue a year ago · 11 comments

Originally posted by @benjaoming in #119 (comment)

It does pass the build, but I believe this is not what we want in the end, this comes from the build log:

Processing /home/docs/checkouts/readthedocs.org/user_builds/django-nyt/checkouts/latest
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
WARNING: django-nyt 1.3 does not provide the extra 'docs'
Collecting django<4.3,>=2.2 (from django-nyt==1.3)
  Downloading Django-4.2.1-py3-none-any.whl (8.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.0/8.0 MB 186.8 MB/s eta 0:00:00
Collecting asgiref<4,>=3.6.0 (from django<4.3,>=2.2->django-nyt==1.3)
  Downloading asgiref-3.6.0-py3-none-any.whl (23 kB)
Collecting sqlparse>=0.3.1 (from django<4.3,>=2.2->django-nyt==1.3)
  Downloading sqlparse-0.4.4-py3-none-any.whl (41 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.2/41.2 kB 189.4 MB/s eta 0:00:00
Building wheels for collected packages: django-nyt
  Building wheel for django-nyt (pyproject.toml): started
  Building wheel for django-nyt (pyproject.toml): finished with status 'done'
  Created wheel for django-nyt: filename=django_nyt-1.3-py3-none-any.whl size=37182 sha256=aba75df76ec565f0d6a839f5765b41c06b3d4471f5bd5c95718397e11a33050b
  Stored in directory: /tmp/pip-ephem-wheel-cache-sy3lphk2/wheels/d6/35/b9/a2fd0913eb69cebf9dde1d242c2cb14ddfcc8153b48459626d
Successfully built django-nyt
Installing collected packages: sqlparse, asgiref, django, django-nyt
Successfully installed asgiref-3.6.0 django-4.2.1 django-nyt-1.3 sqlparse-0.4.4

As we don't have a docs mode in our pyproject.toml configuration (pip install django-nyt[docs], sorry here)

So then, what I believe is the right process, it to add more install steps with the dependencies listed here

Something like this I believe will do the trick:

version: 2
build:
  os: ubuntu-20.04
  tools:
    python: "3.9"
sphinx:
   configuration: docs/conf.py
python:
   install:
    - method: pip
      path: .
    - method: pip
      path: sphinx
    - method: pip
      path: channels
    - method: pip
      path: sphinx_rtd_theme

Answer 1 · 2023-05-04T16:57:37.000Z

@benjaoming I'll appreciate any feedback here

Answer 2 · 2023-05-04T17:14:57.000Z

Ohhhh I see!!

This:

    - method: pip
      path: sphinx

will be transformed to python sphinx/setup.py install...

Does RDT support install packages without the need to use a requirement file?

Answer 3 · 2023-05-04T22:07:46.000Z

Ahhhh seems that this is valid

version: 2

build:
  os: ubuntu-22.04
  tools:
    python: "3.11"
  commands:
    - pip install pelican
    - pelican --settings docs/pelicanconf.py --output _readthedocs/html/ docs/

So maybe we can place many commands with pip install per each package.

Answer 4 · 2023-05-07T12:23:49.000Z

@oscarmcm thanks for looking into all this!

I think there's a point in having the requirements outside of the RTD configurations in a requirements.txt or similar so that they can be installed when writing docs locally.

Answer 5 · 2023-05-08T17:00:02.000Z

Yeah, an I'm trying to get rid of the requirements.txt in the root dir, like what we have for Django-Wiki because that's the real point on having an environment for the docs, and thus this environment already has the required packages

So we have 3 options here:

Maybe the most easier solution is use the .txt file, but place it under the docs folder.
Add more commands to the build section in the read the docs config, as this is only need for the read the docs environment.
Create a section for install Django-NYT with the docs support pip install <package>[docs].

@benjaoming what do you think?

Answer 6 · 2023-05-08T21:06:58.000Z

I think the 3rd option works well, that's probably the best pattern in general - it supports local docs installation and RTD as well 👍

Answer 7 · 2023-05-09T18:06:20.000Z

My last question here: doesn't it feels weird ship to end user the possibility to install something unrelated to the app? why the end users (more than the maintainers or contributors) will then be able to do pip install django-nyt[docs]?

From my point of view, the ability to install package features should remain for that, and that's the reason why do we have different environments.

Thanks for any feedback!

Answer 8 · 2023-05-09T20:45:43.000Z

@oscarmcm end-users can also be Linux distributions. Someone creating a .deb or .rpm can want to build with debugging, development or docs. So it's okay, even though not very relevant :) Byte-wise, I guess it's a very tiny bit of code :)

Answer 9 · 2023-05-09T20:56:55.000Z

@oscarmcm it's very comparable to all the hatch and test dependencies. I think it belongs to pyproject.toml if it fits 👍

Answer 10 · 2023-05-09T22:29:38.000Z

@benjaoming thanks for all the feedback here, appreciate it a lot 🙏 .

Answer 11 · 2023-05-10T15:31:11.000Z

Now the docs build correctly! https://readthedocs.org/projects/django-nyt/builds/20588399/

Processing /home/docs/checkouts/readthedocs.org/user_builds/django-nyt/checkouts/latest
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting django<4.3,>=2.2 (from django-nyt==1.3)
  Downloading Django-4.2.1-py3-none-any.whl (8.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.0/8.0 MB 202.2 MB/s eta 0:00:00
Collecting channels==4.0.0 (from django-nyt==1.3)
  Downloading channels-4.0.0-py3-none-any.whl (28 kB)
Collecting sphinx-rtd-theme==1.2.0 (from django-nyt==1.3)
  Downloading sphinx_rtd_theme-1.2.0-py2.py3-none-any.whl (2.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.8/2.8 MB 233.0 MB/s eta 0:00:00
Collecting sphinx==6.2.1 (from django-nyt==1.3)
  Downloading sphinx-6.2.1-py3-none-any.whl (3.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.0/3.0 MB 111.6 MB/s eta 0:00:00
Collecting asgiref<4,>=3.5.0 (from channels==4.0.0->django-nyt==1.3)
  Downloading asgiref-3.6.0-py3-none-any.whl (23 kB)
Collecting sphinxcontrib-applehelp (from sphinx==6.2.1->django-nyt==1.3)
  Downloading sphinxcontrib_applehelp-1.0.4-py3-none-any.whl (120 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 120.6/120.6 kB 261.9 MB/s eta 0:00:00
Collecting sphinxcontrib-devhelp (from sphinx==6.2.1->django-nyt==1.3)
  Downloading sphinxcontrib_devhelp-1.0.2-py2.py3-none-any.whl (84 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 84.7/84.7 kB 260.0 MB/s eta 0:00:00
Collecting sphinxcontrib-jsmath (from sphinx==6.2.1->django-nyt==1.3)
  Downloading sphinxcontrib_jsmath-1.0.1-py2.py3-none-any.whl (5.1 kB)
Collecting sphinxcontrib-htmlhelp>=2.0.0 (from sphinx==6.2.1->django-nyt==1.3)
  Downloading sphinxcontrib_htmlhelp-2.0.1-py3-none-any.whl (99 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 99.8/99.8 kB 258.4 MB/s eta 0:00:00
Requirement already satisfied: sphinxcontrib-serializinghtml>=1.1.5 in /home/docs/checkouts/readthedocs.org/user_builds/django-nyt/envs/latest/lib/python3.11/site-packages (from sphinx==6.2.1->django-nyt==1.3) (1.1.5)
Collecting sphinxcontrib-qthelp (from sphinx==6.2.1->django-nyt==1.3)
  Downloading sphinxcontrib_qthelp-1.0.3-py2.py3-none-any.whl (90 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 90.6/90.6 kB 186.7 MB/s eta 0:00:00
Requirement already satisfied: Jinja2>=3.0 in /home/docs/checkouts/readthedocs.org/user_builds/django-nyt/envs/latest/lib/python3.11/site-packages (from sphinx==6.2.1->django-nyt==1.3) (3.0.3)
Collecting Jinja2>=3.0 (from sphinx==6.2.1->django-nyt==1.3)
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.1/133.1 kB 282.2 MB/s eta 0:00:00
Requirement already satisfied: Pygments>=2.13 in /home/docs/checkouts/readthedocs.org/user_builds/django-nyt/envs/latest/lib/python3.11/site-packages (from sphinx==6.2.1->django-nyt==1.3) (2.15.1)
Collecting docutils<0.20,>=0.18.1 (from sphinx==6.2.1->django-nyt==1.3)
  Downloading docutils-0.19-py3-none-any.whl (570 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 570.5/570.5 kB 300.5 MB/s eta 0:00:00
Requirement already satisfied: snowballstemmer>=2.0 in /home/docs/checkouts/readthedocs.org/user_builds/django-nyt/envs/latest/lib/python3.11/site-packages (from sphinx==6.2.1->django-nyt==1.3) (2.2.0)
Requirement already satisfied: babel>=2.9 in /home/docs/checkouts/readthedocs.org/user_builds/django-nyt/envs/latest/lib/python3.11/site-packages (from sphinx==6.2.1->django-nyt==1.3) (2.12.1)
Requirement already satisfied: alabaster<0.8,>=0.7 in /home/docs/checkouts/readthedocs.org/user_builds/django-nyt/envs/latest/lib/python3.11/site-packages (from sphinx==6.2.1->django-nyt==1.3) (0.7.13)
Requirement already satisfied: imagesize>=1.3 in /home/docs/checkouts/readthedocs.org/user_builds/django-nyt/envs/latest/lib/python3.11/site-packages (from sphinx==6.2.1->django-nyt==1.3) (1.4.1)
Requirement already satisfied: requests>=2.25.0 in /home/docs/checkouts/readthedocs.org/user_builds/django-nyt/envs/latest/lib/python3.11/site-packages (from sphinx==6.2.1->django-nyt==1.3) (2.30.0)
Requirement already satisfied: packaging>=21.0 in /home/docs/checkouts/readthedocs.org/user_builds/django-nyt/envs/latest/lib/python3.11/site-packages (from sphinx==6.2.1->django-nyt==1.3) (23.1)
  Downloading docutils-0.18.1-py2.py3-none-any.whl (570 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 570.0/570.0 kB 292.9 MB/s eta 0:00:00
Collecting sphinxcontrib-jquery!=3.0.0,>=2.0.0 (from sphinx-rtd-theme==1.2.0->django-nyt==1.3)
  Downloading sphinxcontrib_jquery-4.1-py2.py3-none-any.whl (121 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.1/121.1 kB 289.4 MB/s eta 0:00:00
Collecting sqlparse>=0.3.1 (from django<4.3,>=2.2->django-nyt==1.3)
  Downloading sqlparse-0.4.4-py3-none-any.whl (41 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.2/41.2 kB 221.4 MB/s eta 0:00:00
Requirement already satisfied: MarkupSafe>=2.0 in /home/docs/checkouts/readthedocs.org/user_builds/django-nyt/envs/latest/lib/python3.11/site-packages (from Jinja2>=3.0->sphinx==6.2.1->django-nyt==1.3) (2.1.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/docs/checkouts/readthedocs.org/user_builds/django-nyt/envs/latest/lib/python3.11/site-packages (from requests>=2.25.0->sphinx==6.2.1->django-nyt==1.3) (3.1.0)
Requirement already satisfied: idna<4,>=2.5 in /home/docs/checkouts/readthedocs.org/user_builds/django-nyt/envs/latest/lib/python3.11/site-packages (from requests>=2.25.0->sphinx==6.2.1->django-nyt==1.3) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/docs/checkouts/readthedocs.org/user_builds/django-nyt/envs/latest/lib/python3.11/site-packages (from requests>=2.25.0->sphinx==6.2.1->django-nyt==1.3) (2.0.2)
Requirement already satisfied: certifi>=2017.4.17 in /home/docs/checkouts/readthedocs.org/user_builds/django-nyt/envs/latest/lib/python3.11/site-packages (from requests>=2.25.0->sphinx==6.2.1->django-nyt==1.3) (2023.5.7)
Building wheels for collected packages: django-nyt
  Building wheel for django-nyt (pyproject.toml): started
  Building wheel for django-nyt (pyproject.toml): finished with status 'done'
  Created wheel for django-nyt: filename=django_nyt-1.3-py3-none-any.whl size=37263 sha256=9e3729c8505cf15a0a8d08d617a1e58827dc09399b9590feac1dc7a28c3fa140
  Stored in directory: /tmp/pip-ephem-wheel-cache-sog359sl/wheels/67/ac/9c/460638e510dd22b444b106130a018a0b24b84744c525d2edf7
Successfully built django-nyt
Installing collected packages: sqlparse, sphinxcontrib-qthelp, sphinxcontrib-jsmath, sphinxcontrib-htmlhelp, sphinxcontrib-devhelp, sphinxcontrib-applehelp, Jinja2, docutils, asgiref, sphinx, django, sphinxcontrib-jquery, django-nyt, channels, sphinx-rtd-theme
  Attempting uninstall: Jinja2
    Found existing installation: Jinja2 3.0.3
    Uninstalling Jinja2-3.0.3:
      Successfully uninstalled Jinja2-3.0.3
  Attempting uninstall: docutils
    Found existing installation: docutils 0.17.1
    Uninstalling docutils-0.17.1:
      Successfully uninstalled docutils-0.17.1
  Attempting uninstall: sphinx
    Found existing installation: Sphinx 1.8.6
    Uninstalling Sphinx-1.8.6:
      Successfully uninstalled Sphinx-1.8.6
  Attempting uninstall: sphinx-rtd-theme
    Found existing installation: sphinx-rtd-theme 0.4.3
    Uninstalling sphinx-rtd-theme-0.4.3:
      Successfully uninstalled sphinx-rtd-theme-0.4.3
Successfully installed Jinja2-3.1.2 asgiref-3.6.0 channels-4.0.0 django-4.2.1 django-nyt-1.3 docutils-0.18.1 sphinx-6.2.1 sphinx-rtd-theme-1.2.0 sphinxcontrib-applehelp-1.0.4 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-2.0.1 sphinxcontrib-jquery-4.1 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sqlparse-0.4.4