Install sage_conf using flit instead of setuptools

[mkoeppe]

So far this is just an illustration of what can be done with flit (#29846) and what cannot:

• Both non-src and src layouts are supported (for the latter, see pypa/flit#260).
• Symlinks of directories or files to somewhere outside of the root are not supported.

Not intended for merging at this point.

Depends on #29846

Component: build

Branch/Commit: u/mkoeppe/install_sage_conf_using_flit @ 7ea5fab

Issue created by migration from https://trac.sagemath.org/ticket/29854

[mkoeppe]

Branch: u/mkoeppe/install_sage_conf_using_flit

[mkoeppe]

Last 10 new commits:

dacea8b	WIP: Install sage-env-config with sage_conf
1ee054c	Fixup
5f7b1cc	Merge branch 't/21559/change-src-bin-installation' into t/29850/install_sage_env_config_with_sage_conf
dfb4400	build/pkgs/pytoml: New
11b6003	build/pkgs/flit, flit_core: New
20e2d59	Add dependencies
721e8fd	build/bin/sage-pip-install: For PEP 517 packages, do not try to uninstall first
5e7155e	build/pkgs/pytoml/spkg-install.in: New
8f58336	Merge branch 't/29846/add_packages_flit_and_dependencies_flit_core__pytoml' into t/29850/install_sage_env_config_with_sage_conf
fd83992	build/pkgs/sage_conf: Install with flit instead of setuptools

[mkoeppe]

Commit: fd83992

[mkoeppe]

Work Issues: Rebase so it is not on top of #29850

[mkoeppe]

comment:3

This branch does not work because it is trying to use a symlink for sage_conf.py into the SAGE_ROOT/src directory.

[sage_conf-none] Processing /Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/sage_conf/src
[sage_conf-none]   Created temporary directory: /private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-req-build-w7fopzwn
[sage_conf-none]   Added file:///Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/sage_conf/src to build tracker '/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-req-tracker-jmerw27d'
[sage_conf-none]     Created temporary directory: /private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-modern-metadata-dxzef0bb
[sage_conf-none]     Preparing wheel metadata: started
[sage_conf-none]     Running command /Users/mkoeppe/s/sage/sage-rebasing/local/bin/python3 /Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/tmpb7arkmg5
[sage_conf-none]     Traceback (most recent call last):
[sage_conf-none]       File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
[sage_conf-none]         main()
[sage_conf-none]       File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
[sage_conf-none]         json_out['return_val'] = hook(**hook_input['kwargs'])
[sage_conf-none]       File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 133, in prepare_metadata_for_build_wheel
[sage_conf-none]         return hook(metadata_directory, config_settings)
[sage_conf-none]       File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit_core/buildapi.py", line 28, in prepare_metadata_for_build_wheel
[sage_conf-none]         module = Module(ini_info.module, os.getcwd())
[sage_conf-none]       File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit_core/common.py", line 55, in __init__
[sage_conf-none]         raise ValueError("No file/folder found for module {}".format(name))
[sage_conf-none]     ValueError: No file/folder found for module sage_conf
[sage_conf-none]     Preparing wheel metadata: finished with status 'error'
[sage_conf-none] ERROR: Command errored out with exit status 1: /Users/mkoeppe/s/sage/sage-rebasing/local/bin/python3 /Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/tmpb7arkmg5 Check the logs for full command output.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

e537d48

build/pkgs/sage_conf: Install with flit instead of setuptools

[sagetrac-git]

Changed commit from fd83992 to e537d48

[mkoeppe]

Changed work issues from Rebase so it is not on top of #29850 to none

[mkoeppe]

Dependencies: #29846

[mkoeppe]

comment:6

Rebased on top of only #29846 (flit)

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

27cf73b	build/pkgs/sage_conf: Switch to src layout where src is a symlink (also does not work)
4d20d4a	build/pkgs/sage_conf: Back to sage_conf.py[.in] as a file here, not a symlink (works)

[sagetrac-git]

Changed commit from e537d48 to 4d20d4a

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1 +1,3 @@
+So far this is just an illustration of what can be done with flit (#29846) and what cannot.
 
+Not intended for merging at this point.

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,3 +1,7 @@
-So far this is just an illustration of what can be done with flit (#29846) and what cannot.
+So far this is just an illustration of what can be done with flit (#29846) and what cannot:
+
+- Both non-src and src layouts are supported (for the latter, see https://github.com/takluyver/flit/pull/260).
+- Symlinks of directories or files to somewhere outside of the root are not supported.
 
 Not intended for merging at this point.
+

[mkoeppe]

comment:10

Both with this version and the version in 27cf73b, also:

flit build 
Fetching list of valid trove classifiers                                                                                                                                                                      I-flit.validate
Version number normalised: '9.2.beta0' -> '9.2b0' (see PEP 440)                                                                                                                                         W-flit_core.versionno
Found 4 files tracked in git                                                                                                                                                                                     I-flit.sdist
Traceback (most recent call last):
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/bin/flit", line 8, in <module>
    sys.exit(main())
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit/__init__.py", line 171, in main
    gen_setup_py=args.setup_py)
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit/build.py", line 46, in main
    sdist_file = sb.build(dist_dir, gen_setup_py=gen_setup_py)
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit/sdist.py", line 223, in build
    return Path(super().build(str(target_dir), gen_setup_py=gen_setup_py))
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit_core/sdist.py", line 181, in build
    files_to_add = self.apply_includes_excludes(self.select_files())
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit_core/sdist.py", line 155, in apply_includes_excludes
    .format(", ".join(missing_crucial)))
Exception: Crucial files were excluded from the sdist: src/sage_conf.py

(because sage_conf.py is generated and not tracked in git)

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

7ea5fab

build/pkgs/sage_conf/src/pyproject.toml: Add tool.flit.sdist section

[sagetrac-git]

Changed commit from 4d20d4a to 7ea5fab

[mkoeppe]

comment:12

This fixes the sdist.

Same change on top of 27cf73b creates a tar file with a packaged symlink:

 lrwxr-xr-x       0/0             0 sage_conf-9.2b0/src --> ../../../../src

and gives an error afterwards:

Built sdist: dist/sage_conf-9.2b0.tar.gz                                                                                                                                                                    I-flit_core.sdist
Traceback (most recent call last):
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/bin/flit", line 8, in <module>
    sys.exit(main())
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit/__init__.py", line 171, in main
    gen_setup_py=args.setup_py)
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit/build.py", line 51, in main
    with unpacked_tarball(sdist_file) as tmpdir:
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit/build.py", line 24, in unpacked_tarball
    tf.extractall(tmpdir)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/tarfile.py", line 2000, in extractall
    numeric_owner=numeric_owner)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/tarfile.py", line 2042, in extract
    numeric_owner=numeric_owner)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/tarfile.py", line 2104, in _extract_member
    os.makedirs(upperdirs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/tmp6jcs8l04/sage_conf-9.2b0/src'

[mkoeppe]

comment:13

A big part of the issues with symlinks comes, of course, from the behavior of pip, which copies the local source directory to a temporary location, breaking symlinks in the process. Various unresolved pip issues talk about finding a solution for local builds, most recently pypa/pip#7555 (earlier issues: pypa/pip#2195 and within, pypa/pip#7882).

[mkoeppe]

comment:15

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.