Incompatibility with moto 5
jayvdb opened this issue · 8 comments
Problem description
Be sure your description clearly answers the following questions:
- https://build.opensuse.org/package/show/openSUSE:Factory/python-smart-open is now in "Fails to build" status because the tests use
@moto.mock_s3which doesnt work with moto 5 This problem doesnt exist locally because #798 pinned moto < 5. - What is the expected result? A release of smart_open with tests that are compatible with moto 5
- What are you seeing instead? The tests result in causes
AttributeError: module 'moto' has no attribute 'mock_s3'.
Steps/code to reproduce the problem
Remove the moto < 5 pin and retest.
Versions
openSUSE Tumbleweed is using smart_open v6.4.0 for py39-py311
Checklist
Before you create the issue, please make sure you have:
- Described the problem clearly
- Provided a minimal reproducible example, including any required data
- Provided the version numbers of the relevant software
Unless someone is willing to go through the code base and work out what the problem with moto 5 and onwards is, I think we'll re-release smart_open with the moto pinned to <5.
Something changed on the moto side going to version 5, I personally don't have the time to work out what that was.
pinning will not solve the distro packaging problem. Once a distro has choosen moto 5, all other packages need to either support that version, or be deleted from the distro. We dont have multiple versions of moto in the distro. On openSUSE Tumbleweed (which is a rolling release), there is an automated service which nominates a package for deletion if it hasnt built for 6 weeks. I can decline that deletion, but that only buys a bit more time. Usually at that stage, I figure out the problem and send a patch upstream.
This problem can be partially "solved" pretty easily by allowing testing of s3 support to be disabled.
e.g. instead of @moto.mock_s3()\ndef foo(..): ..., doing something like
def foo(..): ...
if hasattr(moto, 'mock_s3'):
foo = moto.mock_s3(foo)@piskvorky What do you think is the best course of action here? My opinion is that the above packaging approach (a distro chooses some version of a package, and then expects everything to work with that particular version) is poor. Personally, I don't understand why smart_open (or gensim, or anything else that's on PIP) even needs to be a an openSUSE package. There are already many package managers: pip being the main one, conda being another, for people who can't live without it); why introduce yet another one? More importantly, do we have the bandwidth to make sure smart_open works with yet another package manager?
@jayvdb Hypothetically, could you patch the smart_open tests as required as part of your package-building process?
Yes, openSUSE and all distro packaging systems AFAIK can apply patches as part of the package-building process. We usually also submit patches upstream, which I have done in the past here in this project.
You can see all the distros which package smart-open at https://repology.org/project/python:smart-open/versions . It is a result of being popular. A typical distro user doesnt want smart-open - they want some other package which uses smart-open - see https://www.wheelodex.org/projects/smart-open/rdepends/ for the likely candidates.
This issue is not about lots of packaging systems having varying needs. All distros work the same way - they choose one version of each dependency and eventually figure out how to get them all working together. A rolling release just does this a bit quicker and more often. This issue is about compatibility with latest moto. Eventually it will need to be done. This issue is about openSUSE identifying the incompatibility, and being on the hook for helping get it fixed.
@mpenkov Distro packaging is standard – having a stable, vetted, known-to-work version of a package in an official distribution can be beneficial.
However I agree that making such packaging work is "not our problem". In the sense that we're happy to accept PRs that improve smart_open / streamline the packaging (to the degree they don't break other people's workflows) but unfortunately have no capacity to get deeply involved. At least I don't :)
@jayvdb are you able to submit a PR? What is actually needed to support that new version of moto? As opposed to commenting out tests, which we probably don't want. Did they just rename methods / shuffle them around, or are the moto changes more substantial?
Is a new release planned for the next few weeks? There hasnt been a release for a while, and a security fix has been merged since the last release. If a release is coming, I'll wait until the release before updating the openSUSE package.
If not, I'll add a patch to the openSUSE package.
I'll make a new release this week