DeprecationWarning: pkg_resources is deprecated as an API
simonw opened this issue · 25 comments
Got this running tests against Python 3.11.
../../../.local/share/virtualenvs/datasette-big-local-6Yn-280V/lib/python3.11/site-packages/datasette/app.py:14: in <module>
import pkg_resources
../../../.local/share/virtualenvs/datasette-big-local-6Yn-280V/lib/python3.11/site-packages/pkg_resources/__init__.py:121: in <module>
warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)
E DeprecationWarning: pkg_resources is deprecated as an API
I ran with pytest -Werror --pdb -x
to get the debugger for that warning, but it turned out searching the code worked better. It's used in these two places:
datasette/datasette/plugins.py
Lines 43 to 50 in 5890a20
Line 1037 in 5890a20
I ran this prompt against ChatGPT with the Browsing alpha:
if pkg_resources.resource_isdir(plugin.__name__, "static"): static_path = pkg_resources.resource_filename( plugin.__name__, "static" ) if pkg_resources.resource_isdir(plugin.__name__, "templates"): templates_path = pkg_resources.resource_filename( plugin.__name__, "templates" )This code gives a deprecation warning in Python 3.11 - fix it
It looked up the fix for me:
And suggested:
import importlib.resources
# Replace pkg_resources.resource_isdir with importlib.resources.files().is_file()
if importlib.resources.files(plugin.__name__).joinpath("static").is_file():
static_path = importlib.resources.as_file(
importlib.resources.files(plugin.__name__).joinpath("static")
)
if importlib.resources.files(plugin.__name__).joinpath("templates").is_file():
templates_path = importlib.resources.as_file(
importlib.resources.files(plugin.__name__).joinpath("templates")
)
This looks wrong to me - I would expect something like is_directory()
not is_file()
for telling if static/
is a directory.
Another prompt:
How to fix this:
pkg_resources.get_distribution(package).version
Response:
import importlib.metadata
# Get the version number of the specified package
package_version = importlib.metadata.version(package)
That seems to work:
>>> import importlib.metadata
>>> importlib.metadata.version("datasette")
'0.64.2'
>>> importlib.metadata.version("pluggy")
'1.0.0'
>>> importlib.metadata.version("not-a-package")
...
importlib.metadata.PackageNotFoundError: No package metadata was found for not-a-package
This looks wrong to me - I would expect something like
is_directory()
notis_file()
for telling ifstatic/
is a directory.
I was right about that:
>>> importlib.resources.files('datasette_graphql')
PosixPath('/Users/simon/.local/share/virtualenvs/datasette-big-local-6Yn-280V/lib/python3.11/site-packages/datasette_graphql')
>>> importlib.resources.files('datasette_graphql').joinpath("static")
PosixPath('/Users/simon/.local/share/virtualenvs/datasette-big-local-6Yn-280V/lib/python3.11/site-packages/datasette_graphql/static')
>>> p = importlib.resources.files('datasette_graphql').joinpath("static")
>>> p
PosixPath('/Users/simon/.local/share/virtualenvs/datasette-big-local-6Yn-280V/lib/python3.11/site-packages/datasette_graphql/static')
>>> p.is_
p.is_absolute() p.is_char_device() p.is_fifo() p.is_mount() p.is_reserved() p.is_symlink()
p.is_block_device() p.is_dir() p.is_file() p.is_relative_to( p.is_socket()
>>> p.is_dir()
True
>>> p.is_file()
False
Frustrating that this warning doesn't show up in the Datasette test suite itself. It shows up in plugin test suites that run this test:
@pytest.mark.asyncio
async def test_plugin_is_installed():
datasette = Datasette(memory=True)
response = await datasette.client.get("/-/plugins.json")
assert response.status_code == 200
installed_plugins = {p["name"] for p in response.json()}
assert "datasette-chronicle" in installed_plugins
If you run that test inside Datasette core installed_plugins
is an empty set, which presumably is why the warning doesn't get triggered there.
Weird, I still can't get the warning to show even with this:
@pytest.mark.asyncio
async def test_plugin_is_installed():
datasette = Datasette(memory=True)
class DummyPlugin:
__name__ = "DummyPlugin"
@hookimpl
def actors_from_ids(self, datasette, actor_ids):
return {}
try:
pm.register(DummyPlugin(), name="DummyPlugin")
response = await datasette.client.get("/-/plugins.json")
assert response.status_code == 200
installed_plugins = {p["name"] for p in response.json()}
assert "DummyPlugin" in installed_plugins
finally:
pm.unregister(name="ReturnNothingPlugin")
Here's the exception it uses:
>>> importlib.metadata.version("datasette")
'1.0a6'
>>> importlib.metadata.version("datasette2")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/importlib/metadata/__init__.py", line 996, in version
return distribution(distribution_name).version
File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/importlib/metadata/__init__.py", line 969, in distribution
return Distribution.from_name(distribution_name)
File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/importlib/metadata/__init__.py", line 548, in from_name
raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: No package metadata was found for datasette2
Now I need to switch out pkg_resources
in plugins.py
:
datasette/datasette/plugins.py
Lines 33 to 74 in 852f501
The importlib.metadata.entry_points()
function is pretty interesting:
>>> import importlib.metadata
>>> from pprint import pprint
>>> pprint(importlib.metadata.entry_points())
{'babel.checkers': [EntryPoint(name='num_plurals', value='babel.messages.checkers:num_plurals', group='babel.checkers'),
EntryPoint(name='python_format', value='babel.messages.checkers:python_format', group='babel.checkers')],
'babel.extractors': [EntryPoint(name='jinja2', value='jinja2.ext:babel_extract[i18n]', group='babel.extractors'),
EntryPoint(name='ignore', value='babel.messages.extract:extract_nothing', group='babel.extractors'),
EntryPoint(name='javascript', value='babel.messages.extract:extract_javascript', group='babel.extractors'),
EntryPoint(name='python', value='babel.messages.extract:extract_python', group='babel.extractors')],
'console_scripts': [EntryPoint(name='datasette', value='datasette.cli:cli', group='console_scripts'),
EntryPoint(name='normalizer', value='charset_normalizer.cli.normalizer:cli_detect', group='console_scripts'),
EntryPoint(name='pypprint', value='pprintpp:console', group='console_scripts'),
EntryPoint(name='cog', value='cogapp:main', group='console_scripts'),
EntryPoint(name='icdiff', value='icdiff:start', group='console_scripts'),
EntryPoint(name='pycodestyle', value='pycodestyle:_main', group='console_scripts'),
EntryPoint(name='sphinx-autobuild', value='sphinx_autobuild.__main__:main', group='console_scripts'),
EntryPoint(name='sphinx-apidoc', value='sphinx.ext.apidoc:main', group='console_scripts'),
EntryPoint(name='sphinx-autogen', value='sphinx.ext.autosummary.generate:main', group='console_scripts'),
EntryPoint(name='sphinx-build', value='sphinx.cmd.build:main', group='console_scripts'),
EntryPoint(name='sphinx-quickstart', value='sphinx.cmd.quickstart:main', group='console_scripts'),
EntryPoint(name='sphinx-to-sqlite', value='sphinx_to_sqlite.cli:cli', group='console_scripts'),
EntryPoint(name='pybabel', value='babel.messages.frontend:main', group='console_scripts'),
EntryPoint(name='docutils', value='docutils.__main__:main', group='console_scripts'),
EntryPoint(name='isort', value='isort.main:main', group='console_scripts'),
EntryPoint(name='isort-identify-imports', value='isort.main:identify_imports_main', group='console_scripts'),
EntryPoint(name='hupper', value='hupper.cli:main', group='console_scripts'),
EntryPoint(name='sqlite-utils', value='sqlite_utils.cli:cli', group='console_scripts'),
EntryPoint(name='py.test', value='pytest:console_main', group='console_scripts'),
EntryPoint(name='pytest', value='pytest:console_main', group='console_scripts'),
EntryPoint(name='pyflakes', value='pyflakes.api:main', group='console_scripts'),
EntryPoint(name='livereload', value='livereload.cli:main', group='console_scripts'),
EntryPoint(name='uvicorn', value='uvicorn.main:main', group='console_scripts'),
EntryPoint(name='httpx', value='httpx:main', group='console_scripts'),
EntryPoint(name='flake8', value='flake8.main.cli:main', group='console_scripts'),
EntryPoint(name='blacken-docs', value='blacken_docs:main', group='console_scripts'),
EntryPoint(name='pip', value='pip._internal.cli.main:main', group='console_scripts'),
EntryPoint(name='pip3', value='pip._internal.cli.main:main', group='console_scripts'),
EntryPoint(name='pip3.10', value='pip._internal.cli.main:main', group='console_scripts'),
EntryPoint(name='wheel', value='wheel.cli:main', group='console_scripts'),
EntryPoint(name='pygmentize', value='pygments.cmdline:main', group='console_scripts'),
EntryPoint(name='black', value='black:patched_main', group='console_scripts'),
EntryPoint(name='blackd', value='blackd:patched_main [d]', group='console_scripts'),
EntryPoint(name='codespell', value='codespell_lib:_script_main', group='console_scripts'),
EntryPoint(name='tabulate', value='tabulate:_main', group='console_scripts')],
'datasette': [EntryPoint(name='debug_permissions', value='datasette_debug_permissions', group='datasette'),
EntryPoint(name='codespaces', value='datasette_codespaces', group='datasette'),
EntryPoint(name='vega', value='datasette_vega', group='datasette'),
EntryPoint(name='x_forwarded_host', value='datasette_x_forwarded_host', group='datasette'),
EntryPoint(name='json_html', value='datasette_json_html', group='datasette'),
EntryPoint(name='datasette_write_ui', value='datasette_write_ui', group='datasette'),
EntryPoint(name='pretty_json', value='datasette_pretty_json', group='datasette'),
EntryPoint(name='graphql', value='datasette_graphql', group='datasette')],
'distutils.commands': [EntryPoint(name='compile_catalog', value='babel.messages.frontend:compile_catalog', group='distutils.commands'),
EntryPoint(name='extract_messages', value='babel.messages.frontend:extract_messages', group='distutils.commands'),
EntryPoint(name='init_catalog', value='babel.messages.frontend:init_catalog', group='distutils.commands'),
EntryPoint(name='update_catalog', value='babel.messages.frontend:update_catalog', group='distutils.commands'),
EntryPoint(name='isort', value='isort.setuptools_commands:ISortCommand', group='distutils.commands'),
EntryPoint(name='alias', value='setuptools.command.alias:alias', group='distutils.commands'),
EntryPoint(name='bdist_egg', value='setuptools.command.bdist_egg:bdist_egg', group='distutils.commands'),
EntryPoint(name='bdist_rpm', value='setuptools.command.bdist_rpm:bdist_rpm', group='distutils.commands'),
EntryPoint(name='build', value='setuptools.command.build:build', group='distutils.commands'),
EntryPoint(name='build_clib', value='setuptools.command.build_clib:build_clib', group='distutils.commands'),
EntryPoint(name='build_ext', value='setuptools.command.build_ext:build_ext', group='distutils.commands'),
EntryPoint(name='build_py', value='setuptools.command.build_py:build_py', group='distutils.commands'),
EntryPoint(name='develop', value='setuptools.command.develop:develop', group='distutils.commands'),
EntryPoint(name='dist_info', value='setuptools.command.dist_info:dist_info', group='distutils.commands'),
EntryPoint(name='easy_install', value='setuptools.command.easy_install:easy_install', group='distutils.commands'),
EntryPoint(name='editable_wheel', value='setuptools.command.editable_wheel:editable_wheel', group='distutils.commands'),
EntryPoint(name='egg_info', value='setuptools.command.egg_info:egg_info', group='distutils.commands'),
EntryPoint(name='install', value='setuptools.command.install:install', group='distutils.commands'),
EntryPoint(name='install_egg_info', value='setuptools.command.install_egg_info:install_egg_info', group='distutils.commands'),
EntryPoint(name='install_lib', value='setuptools.command.install_lib:install_lib', group='distutils.commands'),
EntryPoint(name='install_scripts', value='setuptools.command.install_scripts:install_scripts', group='distutils.commands'),
EntryPoint(name='rotate', value='setuptools.command.rotate:rotate', group='distutils.commands'),
EntryPoint(name='saveopts', value='setuptools.command.saveopts:saveopts', group='distutils.commands'),
EntryPoint(name='sdist', value='setuptools.command.sdist:sdist', group='distutils.commands'),
EntryPoint(name='setopt', value='setuptools.command.setopt:setopt', group='distutils.commands'),
EntryPoint(name='test', value='setuptools.command.test:test', group='distutils.commands'),
EntryPoint(name='upload_docs', value='setuptools.command.upload_docs:upload_docs', group='distutils.commands'),
EntryPoint(name='bdist_wheel', value='wheel.bdist_wheel:bdist_wheel', group='distutils.commands')],
'distutils.setup_keywords': [EntryPoint(name='message_extractors', value='babel.messages.frontend:check_message_extractors', group='distutils.setup_keywords'),
EntryPoint(name='cffi_modules', value='cffi.setuptools_ext:cffi_modules', group='distutils.setup_keywords'),
EntryPoint(name='dependency_links', value='setuptools.dist:assert_string_list', group='distutils.setup_keywords'),
EntryPoint(name='eager_resources', value='setuptools.dist:assert_string_list', group='distutils.setup_keywords'),
EntryPoint(name='entry_points', value='setuptools.dist:check_entry_points', group='distutils.setup_keywords'),
EntryPoint(name='exclude_package_data', value='setuptools.dist:check_package_data', group='distutils.setup_keywords'),
EntryPoint(name='extras_require', value='setuptools.dist:check_extras', group='distutils.setup_keywords'),
EntryPoint(name='include_package_data', value='setuptools.dist:assert_bool', group='distutils.setup_keywords'),
EntryPoint(name='install_requires', value='setuptools.dist:check_requirements', group='distutils.setup_keywords'),
EntryPoint(name='namespace_packages', value='setuptools.dist:check_nsp', group='distutils.setup_keywords'),
EntryPoint(name='package_data', value='setuptools.dist:check_package_data', group='distutils.setup_keywords'),
EntryPoint(name='packages', value='setuptools.dist:check_packages', group='distutils.setup_keywords'),
EntryPoint(name='python_requires', value='setuptools.dist:check_specifier', group='distutils.setup_keywords'),
EntryPoint(name='setup_requires', value='setuptools.dist:check_requirements', group='distutils.setup_keywords'),
EntryPoint(name='test_loader', value='setuptools.dist:check_importable', group='distutils.setup_keywords'),
EntryPoint(name='test_runner', value='setuptools.dist:check_importable', group='distutils.setup_keywords'),
EntryPoint(name='test_suite', value='setuptools.dist:check_test_suite', group='distutils.setup_keywords'),
EntryPoint(name='tests_require', value='setuptools.dist:check_requirements', group='distutils.setup_keywords'),
EntryPoint(name='use_2to3', value='setuptools.dist:invalid_unless_false', group='distutils.setup_keywords'),
EntryPoint(name='zip_safe', value='setuptools.dist:assert_bool', group='distutils.setup_keywords')],
'egg_info.writers': [EntryPoint(name='PKG-INFO', value='setuptools.command.egg_info:write_pkg_info', group='egg_info.writers'),
EntryPoint(name='dependency_links.txt', value='setuptools.command.egg_info:overwrite_arg', group='egg_info.writers'),
EntryPoint(name='depends.txt', value='setuptools.command.egg_info:warn_depends_obsolete', group='egg_info.writers'),
EntryPoint(name='eager_resources.txt', value='setuptools.command.egg_info:overwrite_arg', group='egg_info.writers'),
EntryPoint(name='entry_points.txt', value='setuptools.command.egg_info:write_entries', group='egg_info.writers'),
EntryPoint(name='namespace_packages.txt', value='setuptools.command.egg_info:overwrite_arg', group='egg_info.writers'),
EntryPoint(name='requires.txt', value='setuptools.command.egg_info:write_requirements', group='egg_info.writers'),
EntryPoint(name='top_level.txt', value='setuptools.command.egg_info:write_toplevel_names', group='egg_info.writers')],
'flake8.extension': [EntryPoint(name='C90', value='mccabe:McCabeChecker', group='flake8.extension'),
EntryPoint(name='E', value='flake8.plugins.pycodestyle:pycodestyle_logical', group='flake8.extension'),
EntryPoint(name='F', value='flake8.plugins.pyflakes:FlakesChecker', group='flake8.extension'),
EntryPoint(name='W', value='flake8.plugins.pycodestyle:pycodestyle_physical', group='flake8.extension')],
'flake8.report': [EntryPoint(name='default', value='flake8.formatting.default:Default', group='flake8.report'),
EntryPoint(name='pylint', value='flake8.formatting.default:Pylint', group='flake8.report'),
EntryPoint(name='quiet-filename', value='flake8.formatting.default:FilenameOnly', group='flake8.report'),
EntryPoint(name='quiet-nothing', value='flake8.formatting.default:Nothing', group='flake8.report')],
'pylama.linter': [EntryPoint(name='isort', value='isort.pylama_isort:Linter', group='pylama.linter')],
'pytest11': [EntryPoint(name='icdiff', value='pytest_icdiff', group='pytest11'),
EntryPoint(name='asyncio', value='pytest_asyncio.plugin', group='pytest11'),
EntryPoint(name='xdist', value='xdist.plugin', group='pytest11'),
EntryPoint(name='xdist.looponfail', value='xdist.looponfail', group='pytest11'),
EntryPoint(name='timeout', value='pytest_timeout', group='pytest11'),
EntryPoint(name='anyio', value='anyio.pytest_plugin', group='pytest11')],
'setuptools.finalize_distribution_options': [EntryPoint(name='keywords', value='setuptools.dist:Distribution._finalize_setup_keywords', group='setuptools.finalize_distribution_options'),
EntryPoint(name='parent_finalize', value='setuptools.dist:_Distribution.finalize_options', group='setuptools.finalize_distribution_options')],
'sphinx.html_themes': [EntryPoint(name='alabaster', value='alabaster', group='sphinx.html_themes'),
EntryPoint(name='basic-ng', value='sphinx_basic_ng', group='sphinx.html_themes'),
EntryPoint(name='furo', value='furo', group='sphinx.html_themes')],
'sqlite_utils': [EntryPoint(name='hello_world', value='sqlite_utils_hello_world', group='sqlite_utils')]}
Just found this migration guide: https://importlib-metadata.readthedocs.io/en/latest/migration.html
This broke in Python 3.8:
if plugin.__name__ not in DEFAULT_PLUGINS:
try:
if (importlib.resources.files(plugin.__name__) / "static").is_dir():
E AttributeError: module 'importlib.resources' has no attribute 'files'
Confirmed: https://docs.python.org/3/library/importlib.resources.html#importlib.resources.files
importlib.resources.files(package)
[...]
New in version 3.9.
I think I can fix this using https://importlib-resources.readthedocs.io/en/latest/using.html - maybe as a dependency only installed if the Python version is less than 3.9.
I'll imitate certbot
:
'importlib_resources>=1.3.1; python_version < "3.9"',
Looks like 1.3
is the minimum version needed for compatibility with the 3.9 standard library, according to https://github.com/python/importlib_resources/blob/main/README.rst#compatibility
if sys.version_info >= (3, 9): # pragma: no cover
import importlib.resources as importlib_resources
else: # pragma: no cover
import importlib_resources
That passed on 3.8 but should have failed: https://github.com/simonw/datasette/actions/runs/6266341481/job/17017099801 - the "Test DATASETTE_LOAD_PLUGINS" test shows errors but did not fail the CI run.
In my Python 3.8 environment I ran:
datasette install datasette-init datasette-json-html
And now datasette plugins
produces this error:
File "/Users/simon/Dropbox/Development/datasette/datasette/cli.py", line 192, in plugins
click.echo(json.dumps(app._plugins(all=all), indent=4))
File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 1136, in _plugins
ps.sort(key=lambda p: p["name"])
TypeError: '<' not supported between instances of 'NoneType' and 'NoneType'
In the debugger:
>>> import pdb
>>> pdb.pm()
> /Users/simon/Dropbox/Development/datasette/datasette/app.py(1136)_plugins()
-> ps.sort(key=lambda p: p["name"])
(Pdb) ps
[{'name': None, 'static_path': None, 'templates_path': None, 'hooks': ['prepare_connection', 'render_cell'], 'version': '1.0.1'}, {'name': None, 'static_path': None, 'templates_path': None, 'hooks': ['startup'], 'version': '0.2'}]
So the problem is the get_plugins()
function returning plugins with None
for their name:
datasette/datasette/plugins.py
Lines 61 to 91 in 80a9cd9
The problem is here:
86 distinfo = plugin_to_distinfo.get(plugin)
87 if distinfo is None:
88 breakpoint()
89 -> assert False
90 if distinfo.name is None:
91 breakpoint()
92 assert False
93 if distinfo:
94 plugin_info["version"] = distinfo.version
(Pdb) distinfo
(Pdb) plugin
<module 'datasette.sql_functions' from '/Users/simon/Dropbox/Development/datasette/datasette/sql_functions.py'>
That plugin_to_distinfo
is missing some stuff.
No that's not it actually, it's something else.
Got to this point:
DATASETTE_LOAD_PLUGINS=datasette-init python -i $(which datasette) plugins
That fails and drops me into a debugger:
File "/Users/simon/Dropbox/Development/datasette/datasette/cli.py", line 186, in plugins
app = Datasette([], plugins_dir=plugins_dir)
File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 405, in __init__
for plugin in get_plugins()
File "/Users/simon/Dropbox/Development/datasette/datasette/plugins.py", line 89, in get_plugins
plugin_info["name"] = distinfo.name or distinfo.project_name
AttributeError: 'PathDistribution' object has no attribute 'name'
Hunch: https://pypi.org/project/importlib-metadata/ may help here.
That does seem to fix the problem!
Still fails in Python 3.9: https://github.com/simonw/datasette/actions/runs/6266752548/job/17018363302
plugin_info["name"] = distinfo.name or distinfo.project_name
AttributeError: 'PathDistribution' object has no attribute 'name'
Test failed: datasette-json-html should not have been loaded
Tested that locally with Python 3.9 from pyenv
and it worked.
Tests all pass now.