pyproject.toml is not properly configured to build

[aphearin]

Halotools v0.8 is ready for release. All CI tests pass, the docs build, and the wheel generated by this github action works fine. However, I am not able to generate a valid build to upload to pypi. I think the issue must have to do with some combination of setup.cfg, setup.py, and pyproject.toml, because these have been heavily modified very recently, and this will be the first release based on the new package structure. With lots of help from @pllim, in PR #1041 I tried to implement these files to mirror the corresponding ones in the astroquery library, which also still uses the helpers, but no significant differences are jumping out at me.

Build method 1

(following astropy instructions here)

$ pip install build --upgrade
$ python -m build --sdist .

• This build happens suspiciously fast. Halotools a few tens of thousands of lines of cython to compile, and I notice in the build message that none of the cython appears to be compiling. And on a probably related note, I see warnings such as this:

warning: no files found matching 'ez_setup.py'
warning: no files found matching '*.cpp' under directory 'halotools'
warning: no files found matching '*.h' under directory 'halotools'
no previously-included directories found matching 'build'
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'docs/api'
warning: no files found matching 'astropy_helpers/ez_setup.py'
warning: no files found matching '*.pyx' under directory 'astropy_helpers/astropy_helpers'
warning: no files found matching '*.h' under directory 'astropy_helpers/astropy_helpers'
no previously-included directories found matching 'astropy_helpers/build'
warning: no previously-included files matching '*.o' found anywhere in distribution

• The build purports to terminate successfully. However, when I expand the tarball and cd into dist, neither pyproject.toml nor setup.py are present.
• When I run pip install -e .[test], I get the following traceback:

Obtaining file:///Users/aphearin/work/repositories/python/halotools/dist/halotools-0.8
ERROR: file:///Users/aphearin/work/repositories/python/halotools/dist/halotools-0.8 does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.

• When I try to run pytest, I get hundreds of errors.

Build method 2

(following alternative astropy instructions here)

$ python setup.py build sdist --format=gztar

Doing things this way, I see that now at least the cython actually compiles. However, when I expand the tarball, cd into dist, again I notice that pyproject.toml is absent, although setup.py is present. However, when I run the test suite:
$ python setup.py test
I get the following traceback:

  File "/Users/aphearin/work/repositories/python/halotools/dist/halotools-0.8/setup.py", line 16, in <module>
    from astropy_helpers.setup_helpers import setup
ImportError: cannot import name 'setup' from 'astropy_helpers.setup_helpers' (/Users/aphearin/work/repositories/python/halotools/dist/halotools-0.8/.eggs/astropy_helpers-2.0.11-py3.9.egg/astropy_helpers/setup_helpers.py)

[pllim]

I think you need to add pyproject.toml to your manifest at https://github.com/astropy/halotools/blob/master/MANIFEST.in

[aphearin]

Yes, I definitely forgot that and surely that is needed. I haven't actually ever modified the MANIFEST.in file in halotools so I'll start paying closer attention to it now that the package has been restructured. Thanks for catching that.

Once I add that and I try again to do python -m build --sdist ., then pyproject.toml is indeed included in the build as it should be.

[aphearin]

Ok so pyproject.toml is now included in MANIFEST. Also, separately, this line has been included in setup.cfg (mirroring what is done in astropy here. Without this additional new line, I was getting an error message related to package discovery. I am now able to successfully build without error messages. Progress!

Build method 1

$ python -m build --sdist .

• I still notice that none of the cython compiles during this process.
• Nonetheless, I am able to successfully build and run pip install -e .[test] without any errors.
• When I run pytest, I still notice that none of the cython compiles, and I get a traceback with a ton of errors that concludes with:

 File "/Users/aphearin/work/repositories/python/halotools/dist/halotools-0.8/astropy_helpers/astropy_helpers/conftest.py", line 57, in pytest_unconfigure
    combined_cdata.write_file('.coverage.subprocess')
AAttributeError: 'CoverageData' object has no attribute 'write_file'

I tried running python setup.py test instead, since this is what I normally run when I do tests that are local to my machine. But I notice that setup.py is not included in MANIFEST.in so then there is no setup.py available within dist. Is this a problem?

Build method 2

$ python setup.py build sdist --format=gztar

Doing things this way, again the cython actually compiles, and now of course pyproject.toml is present when I build using the command above, expand the tarball, and cd into dist. Also, as before, setup.py is present when I build this way. And when I run the test suite:
$ python setup.py test

I get the following traceback:

/Users/aphearin/opt/miniconda3/envs/dstar/lib/python3.9/site-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: helpers-4.0.1 is an invalid version and will not be supported in a future release
  warnings.warn(
Traceback (most recent call last):
  File "/Users/aphearin/work/repositories/python/halotools/dist/halotools-0.8/setup.py", line 16, in <module>
    from astropy_helpers.setup_helpers import setup
ImportError: cannot import name 'setup' from 'astropy_helpers.setup_helpers' (/Users/aphearin/work/repositories/python/halotools/dist/halotools-0.8/.eggs/astropy_helpers-2.0.11-py3.9.egg/astropy_helpers/setup_helpers.py)

[pllim]

The time you would sink into fixing this might as well be used to get rid of astropy-helpers altogether? I was hoping you would have similar success as astroquery but maybe your Cython/C is giving you problems that astroquery does not have to deal with.

https://docs.astropy.org/projects/package-template/en/latest/ape17.html

[aphearin]

Well, yeah, my ignorance about what's going on under the hood in astropy-helpers reduces me to a noob for these aspects of halotools maintenance, so I see the sense in that suggestion. I manage a bunch of other libraries that are packaged with standard tooling without running into all this confusion. The only difference from those is all the cython used in halotools (I've since migrated to numba and JAX for performance-critical kernels). Anyway, I'd been hoping to postpone ripping off the bandaid until after getting this version out, but you're right that this is taking a lot of work as it is, so probably better to just get on with it.