pyproject.toml (PEP 518) support

Question

pyproject.toml (PEP 518) support

lopopolo opened this issue 6 years ago · 29 comments

lopopolo commented 6 years ago

Hi, I was wondering if there were plans for pyproject.toml support.

see: peritus/bumpversion#192, PEP 518

Answer 1 · 2018-11-18T23:30:15.000Z

Just spent some time reading the PEP and I'm open to adding it to the project at some point.

Answer 2 · 2018-11-18T23:41:13.000Z

To clarify what you meant is bumpversion reading pyproject.toml for its configuration?

Answer 3 · 2018-11-19T00:08:54.000Z

Yes. Read config from pyproject.toml instead of setup.cfg or .bumpversion.cfg. :)

Answer 4 · 2019-03-02T18:00:46.000Z

Though its your call, I would suggest dropping support for .bumpversion.cfg. There are just too many separate config files. Most people can use either of setup.cfg or pyproject.toml.

Answer 5 · 2019-03-02T19:25:34.000Z

This is the case for Python projects.

However, pojects done in another language can also benefit from bumpversion, and there it does not make sense to have either setup.cfg or pyproject.toml.

Answer 6 · 2019-05-30T15:46:19.000Z

Did a bit of hacking on this, it's a little awkward to fit in with the current scheme of looking in multiple places, having to use separate parsers for different configs etc.. Found that TOML is a much better fit than INI in general, because it natively supports booleans, arrays, nested tables and so on, rather than having to hack in equivalents. For this reason, I would recommend supporting a .bumpversion.toml (we basically get it for free once the machinery is in there for parsing the pyproject.toml) and encouraging users to use that instead of INI format - that works regardless of the project language. INI can stick around for legacy purposes, of course, but if/when people move away from setup.cfg, consider throwing a PendingDeprecationWarning on INI usage. There's just much less code involved when working with TOML, and the interface is much more pythonic (INI is obviously in stdlib but it's older than today's notions of pythonicness).

That said, we need to be cautious about writing back to pyproject.toml because it may contain a lot of information about a lot of other tools; comments and ideally formatting should be persisted. There are a few different TOML libraries, but tomlkit seems to be the best option for that persistence. Unfortunately, its API is not exactly the load/dump we're used to from json.

Answer 7 · 2019-08-09T03:08:58.000Z

I agree with the suggestion of bumpversion.toml. Toml is much easier to understand than ini and less likely to have quoting and spacing issues. tomlkit is good but its formatting can be a bit wonky sometimes so I'd rather not mess with a file as important as pyproject.toml. So bumpversion.toml seems like a good solution.

Answer 8 · 2019-08-24T23:35:39.000Z

I like @clbarnes's suggestion -- support for TOML (in general) with .bumpversion.cfg being deprecated for .bumpversion.toml (eventually) and support for pyproject.toml.

Answer 9 · 2019-12-14T20:42:01.000Z

Has there been any update on this?

Answer 10 · 2019-12-14T22:01:08.000Z

Having thought about it, I don't think this is a good idea under bump2version's current mode of operation. It rewrites the config file every time, and when you have a lot of tools which depend on the same config, that's not necessarily desirable. TOML is much more flexible in layout than INI, and supports comments - you'll want to keep that if you have a big complex pyproject file. The tomlkit library can persist formatting and comments but is a bit awkward for everything else.

In an ideal world, a tool like bumpversion could store the current version in an external file, and only store config (which arguably the current version is not) in the config file. But that's possibly too major a departure.

Answer 11 · 2020-01-16T17:02:13.000Z

Here is the awesome-pyproject list. It would be helpful if and when bump2version supports pyproject.toml to notify the list maintainer or submitting a PR to update the list indicating that bump2version supports the file now. Just FYI. :)

Answer 12 · 2020-10-02T13:50:53.000Z

Any updates on this?

Answer 13 · 2020-10-02T15:19:59.000Z

Having thought about it, I don't think this is a good idea under bump2version's current mode of operation. It rewrites the config file every time, and when you have a lot of tools which depend on the same config, that's not necessarily desirable.

Why don't we use bump2version to update the .toml then? With this, it is not rewriting everything but only updating/string-replacing the version.

Answer 14 · 2020-10-02T15:32:07.000Z

To me, that strays into parsing HTML with regex territory. I wouldn't want to bet that no tool will ever use a key ambiguous with our query.

Answer 15 · 2020-10-02T17:01:58.000Z

Sorry for derailing the discussion with my comment. I was not talking about parsing. I see the danger in rewriting the toml by parsing and then pretty-printing everything just to write the version.. But actually I found already a PR for this: #90 (where it's also explained why this is bad).
Or alternatively when fixing issue #94, the version is not so much a config option but just represented in one file.

Answer 16 · 2021-03-03T20:59:05.000Z

To clarify, has pyproject.toml support been incorporated? If so, what's the syntax for writing your configuration settings in the pyproject.toml file? (bump2version docs still only explain .cfg file setup)

Answer 17 · 2021-08-03T20:55:06.000Z

@adam-grant-hendry I don't think that it is already merged.

As @clbarnes states:

That said, we need to be cautious about writing back to pyproject.toml because it may contain a lot of information about a lot of other tools; comments and ideally formatting should be persisted. There are a few different TOML libraries, but tomlkit seems to be the best option for that persistence. Unfortunately, its API is not exactly the load/dump we're used to from json.

In most cases, using pyproject.toml with a PEP517 build backend and other options might not be a good idea.

In addtion to that, PEP621 must be taken into account, which implies a PEP440 version identifier called project.version, which should be included. In this case, the current version must not be written to the configuration file again, but the pyproject.toml must be updated accordingly.

The basic problem here is, that configuration file is re-used, which must be abstracted.

Next what must be concerned is the configuration file format.

[bump2version:file:src/main.py]
search = __VERSION__ = {current_version}
replace = __VERSION__ = {new_version}

might become

[tool.bump2version.file."src/main.py"]
search = "__VERSION__ = {current_version}"
replace = "__VERSION__ = {new_version}"

or

[[tool.bump2version.file]]
file="src/main.py"
search = "__VERSION__ = {current_version}"
replace = "__VERSION__ = {new_version}"

The first one is more straight forward, but the last one offers the ability to do as follows:

[[tool.bump2version.file]]
glob="**/__init__.py"
search = "__VERSION__ = {current_version}"
replace = "__VERSION__ = {new_version}"

which is more intuitive.

I'll try create a refactoring of the configuration within the next days which might enable the on-going development and discussion.

Answer 18 · 2021-08-10T20:31:06.000Z

I'd advise against adding pyproject.toml support but moving to a bumpversion specific TOML file (.bumpversion.toml) instead for reasons, many of which, already mentioned in this thread:

pyproject.toml is an important file used by countless tools so it'd be safer to treat it as read-only to avoid style changes and possible content changes resulting from TOML parser/writer bugs. A shared configuration file that many tools read is great. A shared file that some tools also write to ... not so great. My understanding is that setup.cfg support already received some complaints regarding this.
bump2version's usefulness is in no way limited to Python projects so using a Python specific conf file seems like bad taste (IMO). Also, supporting only one configuration file has the advantage that there's only one possible place to look for config if you want to know if the tool is being used or not.
People will configure search-and-replace patterns for pyproject.toml. It feels risky to do a parser/writer passthrough (possibly doing minor style changes) (also changing version) and a following search-and-replace in the same file.

A bumpversion specific config can be made flatter and easier to read. It could look something like

commit = true
tag = true
current_version = "0.5.2"

[[files]]
path="src/main.py"
search = '__VERSION__ = "{current_version}"'
replace = '__VERSION__ = "{new_version}"'

[[files]]
path="pyproject.toml"
search = 'version = "{current_version}"'
replace = 'version = "{new_version}"'

Answer 19 · 2021-08-10T20:38:25.000Z

While I have nothing against a project specific toml file, much of the appeal of pyproject.toml support is to avoid having a separate config file for each tool in a project.

Perhaps a compromise would be to (by default) support a specifically named toml file, and then provide a command-line option to override. Ex: maybe by default it looks for bumpversion.toml, but there's also a --config option that one could use to do --config=pyproject.toml if they so wished.

Answer 20 · 2021-08-11T12:08:40.000Z

much of the appeal of pyproject.toml support is to avoid having a separate config file for each tool in a project.

IMO this appeal only applies to, as the name pyproject suggests, Python project specific tools. I would not find it appealing at all if suddenly my .gitignore, .editorconfig, .pre-commit-conf.yaml and .codecov.yaml were crammed into pyproject.toml as these files are being read by programming language agnostic tools. I think bump2version falls in the same category.

Answer 21 · 2021-08-11T12:24:22.000Z

Regarding the TOML file format, we could allow the files list to be a heterogeneous array where a string value is a shortcut for filepath, e.g. allow

files = [
    "src/main.py",
    "pyproject.toml",
    { path="foo.json", search = 'version: "{current_version}"', replace = 'version: "{new_version}"' },
]

as alternative to

[[files]]
path="src/main.py"

[[files]]
path="pyproject.toml"

[[files]]
path="foo.json"
search = 'version: "{current_version}"'
replace = 'version: "{new_version}"'

Answer 22 · 2021-08-11T17:43:01.000Z

much of the appeal of pyproject.toml support is to avoid having a separate config file for each tool in a project.

IMO this appeal only applies to, as the name pyproject suggests, Python project specific tools. I would not find it appealing at all if suddenly my .gitignore, .editorconfig, .pre-commit-conf.yaml and .codecov.yaml were crammed into pyproject.toml as these files are being read by programming language agnostic tools. I think bump2version falls in the same category.

I do see your point that bump2version is not a Python-specific tool, having said that, all of those examples you gave (except gitignore) I actually would prefer to configure in a pyproject.toml. pre-commit for example had an issue where others requested it: pre-commit/pre-commit#1165

If my project is a Python project then I'd like to configure all my tools in a single file. This is why I suggest having the option to support either pyproject.toml or whateveryouwanttocallthefile.toml if the project isn't Python-specific.

Answer 23 · 2021-08-11T19:51:28.000Z

I think we shouldn't mix things up.

Though mentioned otherwise, adding support for pyproject.toml doesn't mean that it should be the only or single source of truth. It just means, that python developers who wish to use pyproject.toml because of PEP518 as central configuration for their development are free to do so.

If you do not (wish to) use pyproject.toml or use bump2version for the development with other programming languages, you should still be free to do so.

Although, there might be a tendency to use a toml-based configurattion file over setup.cfg or .bump2version.cfg, it does not mean, that everything should be done in pyproject.toml.

So I think we should consider adding support for pyproject.toml just to be compliant to PEP518. What happens afterward, is a different story. This part of the road(map) is simply the job of the authors / maintainers like @c4urself .

Using an ini-based, toml-based, yaml-based or json-based configuration will result in python always in dict instances, so the impact in fact should be minimal, if it is done right.

So, if we just imagine, the support for pyproject.toml is here, which sequence should be used when loading the configuration? Should we use pyproject.toml in favor of all other files or should it be the fallback?

What I've seen so far is that implementations vary. Some prefer the usage of pyproject.toml over their own configuration, some prefer their own configuration.

My gut instincts tend to tell my the letter should be the preferred way.

In short, the following sequence of searching / reading configuration should be used:

Search for .bump2version.cfg
Search for setup.cfg
Search for .bump2version.toml
Search for pyproject.toml

In order to make things more easy for the user, the configuration files 1-3 should be written back. If a pyproject.toml is used. the configuration entries for modifying the version numbers should be added automatically unless the user configures it differently. This will help, keeping the PEP621 entries PEP440 compliant.

This should be easy to implement, if I understand the existing implementation correctly.

Answer 24 · 2021-08-12T10:56:16.000Z

maybe by default it looks for bumpversion.toml, but there's also a --config option that one could use to do --config=pyproject.toml if they so wished.

Just a minor point: pyproject.toml would need to be treated differently to bumpversion.toml because all of the bumpversion-specific config would be buried in a sub-table rather than at the root. So it wouldn't be enough to just include a --config= argument which is generic over bumpversion-specific files and pyproject files, unless that argument also allowed you to specify the table within the file to look for (e.g. --config=pyproject.toml:tool.bumpversion. You could hardcode special behaviour for pyproject files but that makes it a bit less transparent; although it would also allow you to hardcode additional special behaviour for other language-specific config files with space for arbitrary tools (e.g. rust's Cargo.toml) in the same entry point.

In short, the following sequence of searching / reading configuration should be used:

I'd suggest a minor tweak to the resolution order above: bumpversion-specific files should always be sought before generic files, because plenty of projects could have a setup.cfg without bumpversion config in it, and then a separate bumpversion config file. Additionally, TOML is strictly a better format than INI, so I'd be in favour of an order like bumpversion.toml -> bumpversion.ini -> pyproject.toml:tool.bumpversion -> setup.cfg.

I think we shouldn't mix things up.

Having had a couple of years to mull over this issue, I think I too am personally coming down on the side of a specific config file rather than writing back to the pyproject.toml, especially as all of bumpversion's config would be nested pretty deep into the pyproject, and deep nesting is not handled very ergonomically by TOML. Certainly that would be the easiest first pass, and it helps establish bumpversion as a project-agnostic tool (albeit one with a particularly strong relationship with python). But there seems to be a fair amount of appetite for building it into a shared config file and it may not be all that difficult.

Using an ini-based, toml-based, yaml-based or json-based configuration

I do have an opinion on this: moving from INI to TOML is strictly an upgrade which drastically improves the complexity and maintainability of the tool (once INI support is actually deprecated, of course). INI was a mistake (globally, that is; not one made specifically by bumpversion). That there will be a transition period where both forms is supported is IMO merely an unfortunate hangover from those dark days; not a design goal. I think that supporting multiple config formats as a feature would complicate the lives of contributors, maintainers, and users for very little benefit: sure, there are YAML enthusiasts out there and javascripters inexplicably love hand-writing JSON, but I don't think supporting them all really makes bumpversion any more attractive a solution. "There should be one-- and preferably only one --obvious way to do it."

Answer 25 · 2022-03-05T20:39:08.000Z

It seems now that poetry supports the command poetry version <rule> where one can bump the version listed in the pyproject.toml by major/minor/patch (see the poetry docs.).
In python3.8, one can use the builtin module importlib.metadata to query for package versions dynamically in source code (e.g. my_version = metadata.version('my_pkg'))

With these two updates, bump2version now seems obsolete:

I no longer have to add locations I want version strings updated in a .bumpversion.cfg or setup.cfg. I can do this programmatically with python and I can delete my cfg files
The "single source of truth" is in the pyproject.toml, which resolves the myriad of solutions offered by the PyPA
I can remove bump2version, eliminating an extra dependency in my project
In the advent of PEP 517 and 518, setuptools has deprecated setup.py install (see this) and appears to be transitioning from being a front + backend to simply a backend. Hence, with the growing popularity of pyproject.toml, it seems highly likely adding a pyproject.toml to one's project when packaging will soon become a requirement anyway. Hence, I can achieve everything with one toml file instead of mutliple toml plus config files.

Answer 26 · 2022-03-06T07:50:42.000Z

Thanks @adam-grant-hendry for pointing out all the recent development around version management.

It looks like it become more and more standardized by the day. I also recently discovered Poetry's version command and I like that. But I'm not sure I'll be able to completely get rid of bump2version yet.

For instance, how do you manage a set of version upgrade rules? In my project I have a special rule in my .bumpversion.cfg to keep the changelog.md in sync:

[bumpversion:file:./changelog.md]
search = {{gh}}`{current_version} (unreleased)
replace = {{gh}}`{new_version} (unreleased)

Source: https://github.com/kdeldycke/meta-package-manager/blob/2a170359269b27ebe2413089c47d0cc913e11b26/.bumpversion.cfg#L6-L8

Do you suggest to replace these rules with a custom Python script? Why not. After all, I ended up writing my own changelog version management code and a quite convoluted GitHub workflow, as nowadays, version management is a complex mix of version bump, tagging, building, packaging and publishing (on PyPi).

Answer 27 · 2022-03-07T02:28:28.000Z

@kdeldycke
poetry v1.2 supports plugins and several people have already made "bumpversion" plugins for custom script support:

Unfortunately (I say this because I have loved bump2version), I don't think it would be worth anyone's time to write a custom script for bump2version; they would soon pyproject.toml front-end equivalent anyway.

Had bump2version got on board and decided to adopt the pyproject.toml file, one could have put these setup.cfg/.bumpversion.cfg ini-file style options into the pyproject.toml. Many other 3rd-party packages already have done this, though I respect the bump2version developers' decision points against adopting pyproject.toml wholesale.

If any work is to be done, I recommend it be toward using a plugin or creating one that suites your needs.

Sadly, there's not much of a selling point for me to use bump2version any more:

plugins are easy to write
I don't have to rely on a 3rd party package
I can remove additional .cfg and .ini files, getting closer to a single source of truth

I might've kept bump2version had pyproject.toml been adopted, but it's too late now. Metadata reading being incorporated into the python 3.8+ stdlib too...this is a no brainer for me.

Answer 28 · 2023-03-07T21:50:43.000Z

https://github.com/monim67/poetry-bumpversion is a much neatier solution.

Answer 29 · 2023-06-08T11:48:05.000Z

FYI, bump2version is no longer maintained but a new project is resurrecting it and is called Bump My Version.

Good new, the last bump-my-version 0.4.0 supports pyproject.toml out of the box.

For more information, see: #268