How to specify packages in pyproject.toml - build fails
andy-maier opened this issue · 4 comments
andy-maier commented
I have a project with two Python packages. It uses the flat layout and has a pyproject.toml file (see below) that uses setuptools as the backend, and its setup.py file has just a call to setuptools.setup() without any arguments.
When I invoke "build", it fails because it cannot identify the Python packages:
$ python -m build
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
- setuptools-scm>=8.0
- setuptools>=61.0
- wheel>=0.43.0
* Getting build dependencies for sdist...
error: Multiple top-level packages discovered in a flat-layout: ['try', 'done', 'perf', 'attic', 'design', 'images', 'pywbem', 'issues', 'packaging', 'tmp_issues', 'test_pylint', 'pywbem_mock'].
To avoid accidental inclusion of unwanted files or directories,
setuptools will not proceed with this build.
If you are trying to create a single distribution with multiple packages
on purpose, you should not rely on automatic discovery.
Instead, consider the following options:
1. set up custom discovery (`find` directive with `include` or `exclude`)
2. use a `src-layout`
3. explicitly set `py_modules` or `packages` with a list of names
To find more information, look for "package discovery" on setuptools docs.
ERROR Backend subprocess exited when trying to invoke get_requires_for_build_sdist
When I try to follow the option 3, it fails as well, saying that packages is not allowed (the same happens with py_modules):
$ python -m build
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
- setuptools-scm>=8.0
- setuptools>=61.0
- wheel>=0.43.0
* Getting build dependencies for sdist...
Traceback (most recent call last):
File "/Users/maiera/virtualenvs/pywbem312/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/Users/maiera/virtualenvs/pywbem312/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/maiera/virtualenvs/pywbem312/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 287, in get_requires_for_build_sdist
return hook(config_settings)
^^^^^^^^^^^^^^^^^^^^^
File "/private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/build-env-vw8dmmgw/lib/python3.12/site-packages/setuptools/build_meta.py", line 328, in get_requires_for_build_sdist
return self._get_build_requires(config_settings, requirements=[])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/build-env-vw8dmmgw/lib/python3.12/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
self.run_setup()
File "/private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/build-env-vw8dmmgw/lib/python3.12/site-packages/setuptools/build_meta.py", line 311, in run_setup
exec(code, locals())
File "<string>", line 360, in <module>
File "/private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/build-env-vw8dmmgw/lib/python3.12/site-packages/setuptools/__init__.py", line 104, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/build-env-vw8dmmgw/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 158, in setup
dist.parse_config_files()
File "/private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/build-env-vw8dmmgw/lib/python3.12/site-packages/setuptools/dist.py", line 631, in parse_config_files
pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
File "/private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/build-env-vw8dmmgw/lib/python3.12/site-packages/setuptools/config/pyprojecttoml.py", line 68, in apply_configuration
config = read_configuration(filepath, True, ignore_option_errors, dist)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/build-env-vw8dmmgw/lib/python3.12/site-packages/setuptools/config/pyprojecttoml.py", line 129, in read_configuration
validate(subset, filepath)
File "/private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/build-env-vw8dmmgw/lib/python3.12/site-packages/setuptools/config/pyprojecttoml.py", line 57, in validate
raise ValueError(f"{error}\n{summary}") from None
ValueError: invalid pyproject.toml config: `project`.
configuration error: `project` must not contain {'packages'} properties
The pyproject.toml file with packages specified looks like this:
[build-system]
requires = [
"setuptools>=61.0",
"setuptools-scm>=8.0",
"wheel>=0.43.0",
]
build-backend = "setuptools.build_meta"
[project]
name = "pywbem"
packages = [
"pywbem",
"pywbem_mock"
]
description = "pywbem - A WBEM client"
authors = [
{name = "Tim Potter", email = "tpot@hp.com"}
]
maintainers = [
{name = "Andreas Maier", email = "maiera@de.ibm.com"},
{name = "Karl Schopmeyer", email = "k.schopmeyer@swbell.net"}
]
readme = "README.md"
license = {text = "LGPL version 2.1, or (at your option) any later version"}
keywords = ["cim", "wbem", "client"]
classifiers = [
"License :: OSI Approved :: GNU Lesser General Public License v2 or later (LGPLv2+)",
"Development Status :: 5 - Production/Stable",
"Intended Audience :: Developers",
"Intended Audience :: System Administrators",
"Topic :: Software Development :: Libraries :: Python Modules",
"Topic :: System :: Systems Administration",
"Environment :: Console",
"Operating System :: OS Independent",
"Programming Language :: Python :: 2",
"Programming Language :: Python :: 2.7",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
]
requires-python = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*"
dynamic = ["version"]
dependencies = [
# Direct dependencies for install (must be consistent with minimum-constraints-install.txt)
"mock>=2.0.0,<4.0.0; python_version == '2.7'",
"ply>=3.10",
# PyYAML 5.3 fixed narrow build error on Python 2.7
# PyYAML 5.3.1 addressed issue 38100 reported by safety
# PyYAML 5.2 addressed issue 38639 reported by safety
# PyYAML 6.0 removed support for Python 2.7, but 6.0b1 installs in installtest
"PyYAML>=5.3.1,<6.0; python_version == '2.7'",
"PyYAML>=5.3.1; python_version >= '3.6'",
# virtualenv 20.0.0 (required on py3.8+) requires six>=0.12.0
# tox 3.17 (used in dev-requirements.txt) requires six>=1.14.0
# six 1.16.0 removes the ImportWarning raised by Python 3.10
"six>=1.14.0; python_version <= '3.9'",
"six>=1.16.0; python_version >= '3.10'",
# requests 2.22.0 removed the pinning of urllib3 to <1.25.0, and urllib 1.25.9
# is required to address safety issues
# requests 2.25.0 tolerates urllib3 1.26.5 which is needed on Python 3.10 to
# remove ImportWarning in six
"requests>=2.25.0; python_version == '2.7'",
"requests>=2.25.0; python_version == '3.6'",
"requests>=2.31.0; python_version >= '3.7'",
# yamlloader 1.1.0 gets istalled by mistake on py27+34 on Ubuntu when using setup.py install. See issue #2745.
"yamlloader>=0.5.5,<1.0.0; python_version == '2.7'",
"yamlloader>=0.5.5; python_version >= '3.6'",
# Indirect dependencies for install that are needed for some reason (must be consistent with minimum-constraints-install.txt)
# setuptools 61.0.0 breaks "setup.py install", see https://github.com/pypa/setuptools/issues/3198
"setuptools!=61.0.0; python_version >= '3.7'",
# urllib3 1.24.1 addressed issue 37055 reported by safety
# urllib3 1.24.2 addressed issue 37071 reported by safety
# urllib3 1.25.9 addressed issue 38834 reported by safety
# urllib3 needs to be pinned to <1.25 for requests <2.22.0
# urllib3 1.26.5 vendors six 1.16.0 which is needed on Python 3.10 to remove ImportWarning
# urllib3 2.0 requires py>=3.7
"urllib3>=1.26.18,<2.0.0; python_version == '2.7'",
"urllib3>=1.26.18,<2.0.0; python_version == '3.6'",
"urllib3>=1.26.18; python_version >= '3.7'",
# certifi 2020.6.20 removed support for Python 2.7 but does not declare it
"certifi>=2019.11.28,<2020.6.20; python_version == '2.7'",
"certifi>=2023.07.22; python_version >= '3.6'",
"idna>=2.8",
]
[project.optional-dependencies]
test = ["TBD"]
[project.urls]
Homepage = "https://pywbem.github.io/pywbem/"
"Bug Tracker" = "https://github.com/pywbem/pywbem/issues"
Documentation = "https://pywbem.readthedocs.io/en/latest/"
Repository = "https://github.com/pywbem/pywbem"
Changelog = "https://pywbem.readthedocs.io/en/latest/changes.html"
[tool.setuptools_scm]
# Get the version from the Git tag, and write a version file:
version_file = "pywbem/_version.py"
This happened with:
macOS
Python 3.12
build 1.2.1
setuptools 69.5.1
wheel 0.43.0
My questions are:
- How can I specify the Python packages (without going to the src layout)?
- The fact that the config error with "packages" not allowed is surfaced as a long traceback is not very nice. Can you solve this in "build" by handling the exception?
abravalheri commented
You probably are looking for tool.setuptools.packages or tool.setuptools.py-modules. Ref: https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html#setuptools-specific-configuration (or https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#custom-discovery if you have a nested tree of packages and don't want to specify one by one).
andy-maier commented
Yes, that's what I was looking for. Thanks much! My question 1 is addressed with that.
On question 2: Any chance something can be done about the tracebacks?
layday commented
The error occurs in a subprocess and arrives as a subprocess.CalledProcessError (with its output already having been emitted), which is next to useless. Unfortunately, this is intrinsic to how build hooks are invoked. A configuration error class would have to be standardised and configuration errors captured by the hook caller before build would be able to hide the stack trace.
layday commented
If you feel like this is something worth pursuing, you can start a discussion at https://discuss.python.org/c/packaging/14. I'll close this now.