Add ability to filter out extras from setup.py
Closed this issue · 5 comments
Hi. Thanks very much for making codemetapy
— it has been quite useful. We're using it for our project pyhf
but we have quite a few extras in our setup.py
to make it easy for us to setup different environments in CI and elsewhere. At the moment codemetapy
tries to include information on all of our extras, which is not necessarily the desired output.
For example, in a new Python virtual environment if you do
(codemeta-example) $ python -m pip install -q --upgrade pip setuptools wheel
(codemeta-example) $ python -m pip install -q codemetapy "pyhf==0.5.2"
(codemeta-example) $ python -m pip list
Package Version
------------------ -------
attrs 20.2.0
click 7.1.2
CodeMetaPy 0.3.2
importlib-metadata 2.0.0
jsonpatch 1.26
jsonpointer 2.0
jsonschema 3.2.0
nameparser 1.0.6
numpy 1.19.2
pip 20.2.3
pkg-resources 0.0.0
pyhf 0.5.2
pyrsistent 0.17.3
PyYAML 5.3.1
scipy 1.5.2
setuptools 50.3.0
six 1.15.0
tqdm 4.50.0
wheel 0.35.1
zipp 3.3.0
(codemeta-example) $ codemetapy pyhf > codemeta.json
Processing source #1 of 1
Obtaining python package metadata for: pyhf
Found metadata in /home/feickert/.venvs/codemeta-example/lib/python3.7/site-packages/pyhf-0.5.2.dist-info
WARNING: No translation for distutils key Metadata-Version
WARNING: No translation for distutils key Project-URL
WARNING: No translation for distutils key Project-URL
WARNING: No translation for distutils key Project-URL
WARNING: No translation for distutils key Platform
WARNING: No translation for distutils key Requires-Python
WARNING: No translation for distutils key Description-Content-Type
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
You end up with a very long codemeta.json
that has redundancies in it due to it not parsing ~=
correctly and trying to deal with all the extras
(codemeta-example) $ wc -l codemeta.json
1600 codemeta.json
For comparison, the codemeta.json
that I ended up creating after removing the entries from "softwareRequirements" that were from extras is
$ wc -l codemeta.json
127 codemeta.json
If it would be possible to add a --no-extras
flag or something along those lines of functionality that would be fantastic. Regardless, thank you for helping make software better!
Thanks for using codemetapy! I see the problem indeed. I'll prepare a fix for it, probably removing all these extras by default or at least ensuring there won't be any duplicates.
I just released v0.3.3 that should hopefully address your issues! Version information is now properly parsed (on a first-come basis, including all the extras by default). And there's a parameter to explicitly skip all the extras.
https://github.com/proycon/codemetapy/releases/tag/v0.3.3
Feel free to reopen if there's something not in order.
@proycon Thanks so much for the super speedy response and patch release! Wow! :D
However, I'm still seeing some extras leak in. pyhf
's requirements are just the following
scipy>=1.4.0
click>=6.0
tqdm
jsonschema>=3.2.0
jsonpatch
pyyaml
yet continuing with the example environment above
(codemeta-example) $ python -m pip install -q --upgrade codemetapy
(codemeta-example) $ python -m pip list
Package Version
------------------ -------
attrs 20.2.0
click 7.1.2
CodeMetaPy 0.3.3
importlib-metadata 2.0.0
jsonpatch 1.26
jsonpointer 2.0
jsonschema 3.2.0
nameparser 1.0.6
numpy 1.19.2
pip 20.2.3
pkg-resources 0.0.0
pyhf 0.5.2
pyrsistent 0.17.3
PyYAML 5.3.1
scipy 1.5.2
setuptools 50.3.0
six 1.15.0
tqdm 4.50.2
wheel 0.35.1
zipp 3.3.0
(codemeta-example) $ codemetapy --no-extras pyhf > codemeta.json
Processing source #1 of 1
Obtaining python package metadata for: pyhf
Found metadata in /home/feickert/.venvs/codemeta-example/lib/python3.7/site-packages/pyhf-0.5.2.dist-info
WARNING: No translation for distutils key Metadata-Version
WARNING: No translation for distutils key Project-URL
WARNING: No translation for distutils key Project-URL
WARNING: No translation for distutils key Project-URL
WARNING: No translation for distutils key Platform
WARNING: No translation for distutils key Requires-Python
WARNING: No translation for distutils key Description-Content-Type
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
WARNING: No translation for distutils key Provides-Extra
(codemeta-example) $ wc -l codemeta.json
154 codemeta.json
which is longer than it should be as it also includes iminuit
, jaxlib
, and jax
which are extras for pyhf
(codemeta-example) $ sed -n '96,135 p' codemeta.json
{
"@type": "SoftwareApplication",
"identifier": "iminuit",
"name": "iminuit",
"provider": {
"@id": "https://pypi.org",
"@type": "Organization",
"name": "The Python Package Index",
"url": "https://pypi.org"
},
"runtimePlatform": "Python 3",
"version": ">=1.4."
},
{
"@type": "SoftwareApplication",
"identifier": "jaxlib",
"name": "jaxlib",
"provider": {
"@id": "https://pypi.org",
"@type": "Organization",
"name": "The Python Package Index",
"url": "https://pypi.org"
},
"runtimePlatform": "Python 3",
"version": ">0.1.3"
},
{
"@type": "SoftwareApplication",
"identifier": "jax",
"name": "jax",
"provider": {
"@id": "https://pypi.org",
"@type": "Organization",
"name": "The Python Package Index",
"url": "https://pypi.org"
},
"runtimePlatform": "Python 3",
"version": ">0.1.5"
}
],
I was a bit too quick I see :) There was indeed a bug. It should be solved now in the new v0.3.4 release I just did. When running on pyhf with --no-extras I get:
$ wc -l codemeta.json
115 codemeta.json
which is a bit less than you initially reported after your manual pruning step. But it includes everything from your setup.cfg and none of the extras. Does it look right to you now?
Does it look right to you now?
Looking good! Thanks so much! 🚀
$ pip list | grep CodeMetaPy
CodeMetaPy 0.3.4
$ codemetapy --no-extras pyhf > codemeta.json
$ sed -n '20,96 p' codemeta.json
"softwareRequirements": [
{
"@type": "SoftwareApplication",
"identifier": "scipy",
"name": "scipy",
"provider": {
"@id": "https://pypi.org",
"@type": "Organization",
"name": "The Python Package Index",
"url": "https://pypi.org"
},
"runtimePlatform": "Python 3",
"version": ">=1.4.0"
},
{
"@type": "SoftwareApplication",
"identifier": "click",
"name": "click",
"provider": {
"@id": "https://pypi.org",
"@type": "Organization",
"name": "The Python Package Index",
"url": "https://pypi.org"
},
"runtimePlatform": "Python 3",
"version": ">=6.0"
},
{
"@type": "SoftwareApplication",
"identifier": "tqdm",
"name": "tqdm",
"provider": {
"@id": "https://pypi.org",
"@type": "Organization",
"name": "The Python Package Index",
"url": "https://pypi.org"
},
"runtimePlatform": "Python 3"
},
{
"@type": "SoftwareApplication",
"identifier": "jsonschema",
"name": "jsonschema",
"provider": {
"@id": "https://pypi.org",
"@type": "Organization",
"name": "The Python Package Index",
"url": "https://pypi.org"
},
"runtimePlatform": "Python 3",
"version": ">=3.2.0"
},
{
"@type": "SoftwareApplication",
"identifier": "jsonpatch",
"name": "jsonpatch",
"provider": {
"@id": "https://pypi.org",
"@type": "Organization",
"name": "The Python Package Index",
"url": "https://pypi.org"
},
"runtimePlatform": "Python 3"
},
{
"@type": "SoftwareApplication",
"identifier": "pyyaml",
"name": "pyyaml",
"provider": {
"@id": "https://pypi.org",
"@type": "Organization",
"name": "The Python Package Index",
"url": "https://pypi.org"
},
"runtimePlatform": "Python 3"
}
],