Upgrade to pyproject.toml and pypa/gh-action-pypi-publish

Question

Upgrade to pyproject.toml and pypa/gh-action-pypi-publish

simonw opened this issue a year ago · 24 comments

Answer 1 · 2024-01-16T19:30:24.000Z

Tried the in-development version of this to build datasette-test:

./LICENSE
./pyproject.toml
./tests
./tests/test_datasette_test.py
./README.md
./setup.py
./.gitignore
./datasette_test
./datasette_test/__init__.py
./.github
./.github/workflows
./.github/workflows/publish.yml
./.github/workflows/test.yml

[project]
name = "datasette-test"
version = "0.1"
description = "Utilities to help write tests for Datasette plugins and applications"
readme = "README.md"
requires-python = ">=3.8"
authors = [{name = "Simon Willison"}]
license = {text = "Apache-2.0"}
classifiers = [
    "License :: OSI Approved :: Apache Software License"
]
dependencies = [

]

[project.urls]
Homepage = "https://github.com/simonw/datasette-test"
Changelog = "https://github.com/simonw/datasette-test/releases"
Issues = "https://github.com/simonw/datasette-test/issues"
CI = "https://github.com/simonw/datasette-test/actions"


[project.optional-dependencies]
test = ["pytest"]

This worked:

pip install -e '.[test]'

The publish.yml looked like this:

name: Publish Python Package

on:
  release:
    types: [created]

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
    steps:
    - uses: actions/checkout@v4
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v5
      with:
        python-version: ${{ matrix.python-version }}
        cache: pip
        cache-dependency-path: pyproject.toml
    - name: Install dependencies
      run: |
        pip install '.[test]'
    - name: Run tests
      run: |
        pytest
  deploy:
    runs-on: ubuntu-latest
    needs: [test]
    environment: release
    permissions:
      id-token: write
    steps:
    - uses: actions/checkout@v4
    - name: Set up Python
      uses: actions/setup-python@v5
      with:
        python-version: "3.12"
        cache: pip
        cache-dependency-path: pyproject.toml
    - name: Install dependencies
      run: |
        pip install setuptools wheel build
    - name: Build
      run: |
        python -m build
    - name: Publish
      uses: pypa/gh-action-pypi-publish@release/v1

Compare with this one which I know works: https://github.com/datasette/datasette-build/blob/02e612b667aaa0bc07da24c81023fa6b97d6c270/.github/workflows/publish.yml

Diff here: https://gist.github.com/simonw/968f85a24fead6bf814622b5390d7124/revisions

I think it's going to work. I'm going to risk committing this here, then test it live.

Answer 2 · 2024-01-16T19:39:03.000Z

Annoying: https://github.com/simonw/python-lib/actions/runs/7546633197/job/20544794655

I think I can fix that by manually editing that workflow myself.

Answer 3 · 2024-01-16T19:44:36.000Z

Yeah I need to run this manually on my own laptop to help solve this:

Answer 4 · 2024-01-16T19:49:37.000Z

git clone git@github.com:simonw/python-lib-template-demo
git clone git@github.com:simonw/python-lib
mv python-lib-template-demo python-lib-template-demo-old
cat python-lib/input-for-demo.txt | cookiecutter python-lib
# Now python-lib-template-demo exists as well
mv python-lib-template-demo-old/.git python-lib-template-demo
cd python-lib-template-demo
git diff
# Looks good
git commit -a -m 'Updates'
git push

Answer 5 · 2024-01-16T19:54:56.000Z

That seemed to work - here's the new example output, which passes its tests: https://github.com/simonw/python-lib-template-demo/tree/8ca44689

And here's the full diff simonw/python-lib-template-demo@f417323...8ca4468

Answer 6 · 2024-01-16T19:56:31.000Z

Next test: use https://github.com/new?template_name=python-lib-template-repository&template_owner=simonw to create a datasette-test package and then ship it!

Answer 7 · 2024-01-16T19:58:14.000Z

OK, that created this: https://github.com/datasette/datasette-test/tree/8d5f8262dc3a88f3c6d97f0cef3b55264cabc695

Answer 8 · 2024-01-16T20:19:18.000Z

The test.yml workflow in that is completely wrong - full of setup.py references: https://github.com/datasette/datasette-test/blob/8d5f8262dc3a88f3c6d97f0cef3b55264cabc695/.github/workflows/test.yml

Answer 9 · 2024-01-16T20:25:48.000Z

I'm really confused. https://github.com/simonw/python-lib/blob/main/%7B%7Bcookiecutter.hyphenated%7D%7D/.github/workflows/test.yml has the correct stuff in it. Why did datasette-test get the old version?

Answer 10 · 2024-01-16T20:28:03.000Z

Oh! That's because I needed to update these specific files:

That's my workaround for the problem where you can't push an update to a workflow - I include copies of them in the template repository itself.

Answer 11 · 2024-01-16T20:31:01.000Z

Fixing that like so, after generating a fresh output in /tmp:

cd python-lib-template-repository
cp /tmp/datasette-test/.github/workflows/test.yml .github/workflows/test.yml
cp /tmp/datasette-test/.github/workflows/publish.yml .github/workflows/publish.yml

Answer 12 · 2024-01-16T20:32:09.000Z

I'm going to try generating a repo from that manually: https://github.com/new?template_name=python-lib-template-repository&template_owner=simonw

Answer 13 · 2024-01-16T20:32:59.000Z

Created: https://github.com/simonw/python-lib-issue-6

I'll copy the workflows from that into https://github.com/datasette/datasette-test/

Answer 14 · 2024-01-16T20:37:13.000Z

I'm going to ship 0.1 of datasette/datasette-test to see if the publish workflow works.

Answer 15 · 2024-01-16T20:38:26.000Z

Trying that with a Trusted Publisher:

Answer 16 · 2024-01-16T20:40:29.000Z

https://github.com/datasette/datasette-test/actions/runs/7547266249/job/20546831786 it worked!

https://pypi.org/project/datasette-test/0.1/ was released by the new template.

Answer 17 · 2024-01-16T20:42:29.000Z

There's one sharp edge left: the tests fail the first time the repository is cloned from the template: simonw/python-lib-issue-6@54887b2

https://github.com/simonw/python-lib-issue-6/actions/runs/7547200692/job/20546630262 failed:

Answer 18 · 2024-01-16T20:47:23.000Z

It doesn't like that there are no tests.

Answer 19 · 2024-01-16T20:52:25.000Z

One more test: https://github.com/simonw/python-lib-issues-6-demo-2

Tests now pass on initial commit: simonw/python-lib-issues-6-demo-2@73d216e

Answer 20 · 2024-01-16T20:52:57.000Z

The setup.py did show some warnings: https://github.com/simonw/python-lib-issues-6-demo-2/actions/runs/7547373090

Answer 21 · 2024-01-16T20:54:31.000Z

I'll use that to create a new test repo, check for no errors, then delete that repo.

Answer 22 · 2024-01-16T20:56:03.000Z

Yes, that worked - I added whitespace to the tests/*.py file too to make sure the tests passed.

Answer 23 · 2024-01-16T20:56:40.000Z

This is done!

One last thing: I'm going to update the documentation on https://github.com/simonw/python-lib to remind me how to update the template in the future.

Answer 24 · 2024-01-16T21:00:51.000Z

That documentation is here: https://github.com/simonw/python-lib/blob/main/README.md#notes-on-updating-this-cookiecutter-template