mypy bug with try/except conditional imports
timabbott opened this issue · 34 comments
try:
import simplejson
except ImportError:
import json as simplejson
resulst in the error error: Name 'simplejson' already defined
Pretty sure this is a dup, but I can't find the issue.
Note that #649 was closed; a buch of "leftover" bugs were opened in its
place, but I don't see one that matches this pattern, so I think it's a new
case.
On Mon, Jan 25, 2016 at 3:19 PM, Ryan Gonzalez notifications@github.com
wrote:
—
Reply to this email directly or view it on GitHub
#1153 (comment).
--Guido van Rossum (python.org/~guido)
Closed by mistake.
Another example from #2251 (reported by @RitwikGupta):
try:
# Python 3
from urllib.request import urlopen
except ImportError:
# Python 2
from urllib2 import urlopenAnd another from #2253:
This python code generates an error. It's a common idiom and it would be good if mypy could check for it in some way.
try: import cPickle as pickle except ImportError: import pickle$ mypy --py2 --silent-imports bad.py bad.py:4: error: Name 'pickle' already defined
The current workaround is to add a # type: ignore comment. For example:
try:
import cPickle as pickle
except ImportError:
import pickle # type: ignore # <<-- add thisFYI, this doesn't appear to work with sub-modules:
# This will error
try:
import foo.bar
except ImportError:
foo = None # type:ignoreThis seems to work though:
# This will pass
try:
import doesnt.exist # type:ignore
except ImportError:
doesnt = None
# So will this
try:
import collections.abc # type:ignore
except ImportError:
collections = None
assert doesnt is None
assert collections is not NoneThis is particularly difficult when the import that's happening is a typing import.
For example, this works:
$ mypy --version
mypy 0.530
$ cat no_try.py
from typing import Dict, Any
JSON = Dict[str, Any]
def accept(obj: JSON) -> Any:
return obj.pop('hello')
$ mypy --ignore-missing-imports no_try.py$ cat with_try.py
try:
from typing import Dict, Any
except:
from backports.typing import Dict, Any # type: ignore
JSON = Dict[str, Any]
def accept(obj: JSON) -> Any:
return obj.pop('hello')
$ mypy --ignore-missing-imports with_try.py
with_try.py:10: error: Invalid type "with_try.JSON"
with_try.py:11: error: JSON? has no attribute "pop"
This program should typecheck.
@quodlibetor I don't think your particular case is a bug. mypy has special treatment of typing module, so your imports should be written as:
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from typing import Dict, Any
else:
try:
from typing import Dict, Any
except ImportError:
from backports.typing import Dict, AnyIf your version of typing doesn't have TYPE_CHECKING, then use:
MYPY = False
if MYPY:
# special imports from typing
else:
# actual runtime implementationmypy will understand this.
Hmm, okay, I've verified that that works.
$ cat with_try.py
MYPY = False
if MYPY:
from typing import Dict, Any
else:
try:
from typing import Dict, Any
except:
from backports.typing import Dict, Any
JSON = Dict[str, Any]
def accept(obj: JSON) -> Any:
return obj.pop('hello')
$ mypy with_try.py
I think that I would still be inclined to describe the current behavior as a bug that has a workaround. Would you agree that it is at least as a UX problem?
The if MYPY trick brings the total number of lines of boilerplate to type check a module up to 8, for what would be a single-line import if you were only supporting the most modern python. Even just supporting python 3.5 if you want new items from the typing module means that we jump from 1 to 8 lines of code. It's not even "simple" code. It's also not the obvious first way that I would write that code... obviously.
Why do you need backports.typing? The typing module on PyPI supports all Python versions that mypy itself supports.
I did not know about the typing package on pypi. The last time I looked the only option was backports.typing. That that really ought to fix all my problems, or at least cause interesting new problems ;-) Thank you!
I recently stumbled upon a similar issue with the following code snippet:
try:
from yaml import CLoader as Loader
except ImportError:
from yaml import LoaderThe error message is different though, so it may actually be a different issue:
$ mypy /tmp/bug.py
/tmp/bug.py:4: error: Incompatible import of "Loader" (imported name has type "Type[Loader]", local name has type "Type[CLoader]")
This happens with mypy 0.670+dev.d5cf72b141ce93108c28864ef084c936964ae152 .
Any thoughts?
This is the same problem, but with a class instead of module. If you are not using Loader in annotations, then you can probably work around this by placing:
from typing import TYPE_CHECKING, Union, Type
if TYPE_CHECKING:
import yaml
Loader: Union[yaml.Cloader, yaml.Loader]above the try statement. But again this may only work if you are not using Loader in annotations elsewhere in the file.
Is there any suggestion of how to approach this for a definitive fix? Is it a mypy-related bug?
It seems to me as a not-so-rare case, given libraries might use except ImportError in order to provide fallbacks.
I would certainly rather not having any type: ignore annotation on my code base
Is there any suggestion of how to approach this for a definitive fix? Is it a mypy-related bug?
What do you mean by a "definitive fix"? Do you mean actually allow this in mypy? If yes, then this is non-trivial, definitely would not recommend this as a first issue.
Yes, that's what I mean. Not considering taking it as a first issue though, I've got my plate full already with #6568 on my mypy-spare time
Any hope of a fix any time soon? It's been nearly four years...
I recently started adding type annotations to an existing, large, mature project, and it has dozens of these try/import/except/alternative examples, and adding # type: ignore to all of them feels really gross...
I can't speak for the maintainers (nor do I want to), but my guess is that this is a relatively low priority in light of sunsetting Python 2. I have no doubt that there are legitimate cases of conditional imports of API-compatible libraries that don't involve writing cross-version compatible code, but my guess is that a majority of the uses of this technique are specifically for writing libraries that are both Python 2 and Python 3 compatible, which (as I understand) is now discouraged.
Again, just a guess though….
The majority of my cases are actually not about supporting one version of Python or another, but rather supporting different versions of different libraries or supporting the existence or non-existence of a third-party library and using or not using alternatives to those.
I agree that this is not restricted to dealing Python 2/3 compatibility issues, so this will continue to be relevant after sunsetting Python 2. The main reason that this is still unsupported is that the fix is non-trivial to implement, and the core team has their plate full with other work. If somebody wants to try fixing this, I'm happy to give hints and pointers.
Perhaps related? Code blocks like this:
try:
import pandas as pd
except ImportError:
pd = NoneCan't seem to stop mypy complaining about incompatible types here what I try to do ('(expression has type "None", variable has type Module').
@aldanor, here's a workaround, as mentioned above:
try:
import pandas as pd
except ImportError:
pd = None # type: ignoreThere is no better way to deal with this error as of now.
We encountered this in Twine with importlib.metadata vs importlib_metadata, but only as of mypy 0.750. There's a fix pending in pypa/twine#551 (which also covers an instance of #1393).
I made a repository to reproduce the errors and try out some fixes: https://github.com/bhrutledge/mypy-importlib-metadata. That's currently passing, but the commit history has more details.
Re: my previous comment, there's a suggested workaround (using sys.version) and related discussion starting at #1393 (comment). I applied the workaround in pypa/twine#551.
The sys.version workaround was introduced here: #698
A pattern that might be nicer for some?
if TYPE_CHECKING or sys.version_info < (3, 8, 0):
from typing_extensions import Literal
else:
from typing import LiteralBuilding on an answer from @cjerdonek, would this be an acceptable workaround instead of # type: ignore?:
try:
from py3_pkg import module as _module
except:
from py2_pkg import module
else:
module = _moduleIs this going to be fixed anytime soon?
No, it's not, and it's not clear that it should.
A majority of cases I've seen here are to work around Python stdlib changes, especially Python 2 to Python 3 stuff. sys.version_info checks are the preferred way to do this. Also Python 2 is dead.
if sys.version_info >= (3, 8):
import importlib.metadata as importlib_metadata
else:
import importlib_metadata
For the remaining cases, it's often not clear that it's sound, especially given nominal typing. Even if the API is structurally the same, nominal isinstance checks could result in surprising behaviour from mypy.
The only thing we should do in this space is #5018 where you can explicitly define exactly the structure you want with a Protocol, and mypy will confirm.
For all other cases, if you want to lie to the typechecker, please, go ahead and lie to the type checker (or type ignore):
if TYPE_CHECKING:
import json
else:
try:
import simplejson as json
except ImportError:
import json
FYI "type ignore" doesn't work well because in some cases it will cause Unused "type: ignore" comment errors elsewhere (e.g. GitHub Actions).