Introduce an Intersection
ilevkivskyi opened this issue · 242 comments
This question has already been discussed in #18 long time ago, but now I stumble on this in a practical question: How to annotate something that subclasses two ABC's. Currently, a workaround is to introduce a "mix" class:
from typing import Iterable, Container
class IterableContainer(Iterable[int], Container[int]):
...
def f(x: IterableContainer) -> None: ...
class Test(IterableContainer):
def __iter__(self): ...
def __contains__(self, item: int) -> bool: ...
f(Test())but mypy complains about this
error: Argument 1 of "__contains__" incompatible with supertype "Container"
But then I have found this code snippet in #18
def assertIn(item: T, thing: Intersection[Iterable[T], Container[T]]) -> None:
if item not in thing:
# Debug output
for it in thing:
print(it)Which is exactly what I want, and it is also much cleaner than introducing an auxiliary "mix" class. Maybe then introducing Intersection is a good idea, @JukkaL is it easy to implement it in mypy?
Mypy complains about your code because __contains__ should accept an argument of type object. It's debatable whether this is the right thing to do, but that's how it's specified in typeshed, and it allows Container to be covariant.
I'm worried that intersection types would be tricky to implement in mypy, though conceptually it should be feasible. I'd prefer supporting structural subtyping / protocols -- they would support your use case, as IterableContainer could be defined as a protocol (the final syntax might be different):
from typing import Iterable, Container
class IterableContainer(Iterable[int], Container[int], Protocol):
...
def f(x: IterableContainer) -> None: ...
class Test:
def __iter__(self): ...
def __contains__(self, item: int) -> bool: ...
f(Test()) # should be fine (except for the __contains__ argument type bit)
It would be really cool to implement protocols. Still, in this case intersection could be added as a "syntactic sugar", since there would be a certain asymmetry: Assume you want a type alias for something that implements either protocol, then you write:
IterableOrContainer = Union[Iterable[int], Container[int]]
But if you want a type alias for something that implements both, you would write:
class IterableContainer(Iterable[int], Container[int], Protocol): ...
I imagine such asymmetry could confuse a novice. Intersection could then be added (very roughly) as:
class _Intersection:
def __getitem__(self, bases):
full_bases = bases+(Protocol,)
class Inter(*full_bases): ...
return Inter
Intersection = _Intersection()then one could write:
IterableContainer = Intersection[Iterable[int], Container[int]]
Intersection[...] gets tricky once you consider type variables, callable types and all the other more special types as items. An intersection type that only supports protocols would be too special purpose to include, as it's not even clear how useful protocols would be.
I understand what you mean. That could be indeed tricky in general case.
Concerning protocols, I think structural subtyping would be quite natural for Python users, but only practice could show whether it will be useful. I think it will be useful.
This keeps coming up, in particular when people have code that they want to support both sets and sequences -- there is no good common type, and many people believe Iterable is the solution, but it isn't (it doesn't support __len__).
I think Intersection is a very natural thing (at least if one thinks about types as sets, as I usually do). Also, it naturally appears when one wants to support several ABCs/interfaces/protocols.
I don't think that one needs to choose between protocols and Intersection, on the contrary they will work very well in combination. For example, if one wants to have something that supports either "old-style" reversible protocol (i.e. has __len__ and __iter__ methods) or "new-style" (3.6+) reversible protocol (i.e. has __reversed__ method), then the corresponding type is Union[Reversible, Intersection[Sized, Iterable]].
It is easy to add Intersection to PEP 484 (it is already mentioned in PEP 483) and to typing.py, the more difficult part is to implement it in mypy (although @JukkaL mentioned this is feasible).
For cross-reference from #2702, this would be useful for type variables, e.g. T = TypeVar('T', bound=Intersection[t1, t2]).
Intersection[FooClass, BarMixin] is something I found myself missing today
If we had an intersection class in typing.py, what would we call it?
Intersection is linguistically symmetric with Union, but it's also rather long.
Intersect is shorter, but it's a verb. Meet is the type-theoretic version and also nice and short, but, again, you'd expect Union to be called Join if you call Intersection Meet.
As a data point, I first looked for Intersection in the docs.
Just as a random idea I was thinking about All (it would be more clear if Union would be called Any, but that name is already taken). In general, I don't think long name is a big issue, I have seen people writing from typing import Optional as Opt or even Optional as O depending on their taste. Also generic aliases help in such cases:
T = TypeVar('T')
CBack = Optional[Callable[[T], None]]
def process(data: bytes, on_error: CBack[bytes]) -> None:
...I just opened #483 hoping for exactly the same thing. I literally named it the same. I would be all for Intersection or All to allow to require a list of base classes.
Requests for Intersection appear here and there, maybe we should go ahead and support it in mypy? It can be first put in mypy_extensions or typing_extensions. It is a large piece of work, but should not be too hard. @JukkaL @gvanrossum what do you think?
I think we should note the use cases but not act on it immediately -- there are other tasks that IMO are more important.
OK, I will focus now on PEP 560 plus related changes to typing. Then later we can get back to Intersection, this can be a separate (mini-) PEP if necessary.
Btw, looking at the milestone "PEP 484 finalization", there are two important issues that need to be fixed soon: #208 (str/bytes/unicode) and #253 (semantics of @overload). The second will probably require some help from @JukkaL.
I agree that now's not the right time to add intersection types, but it may make sense later.
(been redirected here from the mailing list)
I think the Not type needs to be added in addition to Intersection:
Intersection[Any, Not[None]]Would mean anything but None.
How about the expression Type1 | Type2 and Type1 & Type2 alternative to Union and Intersection respectively.
example:
x: int & Const = 42
@rnarkk these have already been proposed many times, but have not been accepted.
The Not special form hasn't been proposed before to my knowledge. I suppose you could equivalently propose a Difference[Any, None] type.
What's the use case for that? It's not something I've ever missed in a medium-sized typed codebase at work and in lots of work on typeshed.
@JelleZijlstra My specific case was to specify Any but None.
Difference and other set-alike operators can be expressed using Union, Intersection and Not.
I don't think Not[T] fits in with the rest of the type system; it sounds more like something you'd want to do at runtime, and then specifically only for "not None".
This keeps coming up, in particular when people have code that they want to support both sets and sequences -- there is no good common type, and many people believe Iterable is the solution, but it isn't (it doesn't support
__len__).
@gvanrossum So what is the solution?
(see also my stackoverflow question)
Have you tried defining a custom protocol which subclasses the relevant protocols? Or you can explicitly define all the methods you care about in a custom protocol.
It seems that protocols are not available in typing under 3.6. The following code works and has no warnings in mypy.
from typing_extensions import Protocol
class SizedIterable(Protocol):
def __len__(self):
pass
def __iter__(self):
pass
def foo(some_thing: SizedIterable):
print(len(some_thing))
for part in some_thing:
print(part)
foo(['a', 'b', 'c'])
Thanks!
Cross-posting from python/mypy#3135, as a use case for Intersection that I don't believe is possible right now: class-decorators that add methods.
class FooBar(Protocol):
def bar(self) -> int:
return 1
T = TypeVar("T", bound=Type)
def add_bar(cls: T) -> Intersection[FooBar, T]:
def bar(self) -> int:
return 2
cls.bar = bar
@add_bar
class Foo: pass
Foo().bar()
Now I also understand that mypy doesn't support class decorators yet, but having a way to describe them correctly, and having mypy support them seem 2 different issues.
This is also highly needed for mixin classes to annotate self-type, for cases where mixin can only be mixed to some type or even also require another mixins.
class Base:
def do(self, params: dict):
raise NotImplementedError
class Mixin:
def _auxilary(self, params: dict) -> dict:
return dict(params, foo='bar')
def do(self: Intersection['Mixin', Base], params: dict):
super().do(self._auxilary(params)) # so IDE shows no warning about `.do` or `._auxilary`
Any workaround for this @gvanrossum
Cross-posting from python/mypy#3135, as a use case for Intersection that I don't believe is possible right now: class-decorators that add methods.
class FooBar(Protocol): def bar(self) -> int: return 1 T = TypeVar("T", bound=Type) def add_bar(cls: T) -> Intersection[FooBar, T]: def bar(self) -> int: return 2 cls.bar = bar @add_bar class Foo: pass Foo().bar()Now I also understand that mypy doesn't support class decorators yet, but having a way to describe them correctly, and having mypy support them seem 2 different issues.
Any update on this? Is it still on the road map? Mixins are a core feature of the language, I don't think this is an edge case. I know it can be worked around by defining base classes, but it means adding unnecessary boilerplate and overhead just to please type checkers.
Thanks for all the great work!
Is it still on the road map?
If you are asking about the mypy team, then it is unlikely the support will be added this year. There are many other features we need to support, so we will not have time for this.
Okay. Sad. Thanks for the answer.
I also have been really missing an Intersection type (and was surprised it didn't already exist). I've been writing a nontrivial application using mypy and generating stubs for a sizable codebase it uses, and it's a fairly pythonic thing to say things like "This method returns a file-like object that is also usable as a FnordHandle", or whatever.
I'm not up to date on my type theory but it seems to me that any typechecker that supports protocols already supports the necessary semantics — Intersection[A, B, ...] would have the same behavior as defining a protocol that subclasses from A, B, but adds no other attributes, right?
@wiml You can easily define an "intersection" protocol:
class Both(FileLike, SomethingElse, Protocol): # assuming both bases are protocols
passThe hard part is to support general intersection types like Intersection[Union[A, B], Tuple[C, D], Callable[[E], F]].
In lieu of a true intersection construct, what about a more limited Extends or Implements one, which captures the idea of a type that multiply inherits from a list of base classes?
class FileSomethingProtocol(FileLike, SomethingElse, Protocol, OtherThingsToo): pass
def helper(obj: Extends[FileLike, SomethingElse, Protocol]):
...
obj = FileSomethingProtocol()
helper(obj) # obj extends all the required bases
In lieu of a true intersection construct, what about a more limited
ExtendsorImplementsone, which captures the idea of a type that multiply inherits from a list of base classes?
This was proposed at early stages of PEP 544 discussions but was rejected, since it is easy to define a combining protocol or class, see https://www.python.org/dev/peps/pep-0544/#merging-and-extending-protocols
I think mypy will have general intersection types at some point, since it is a very natural feature in a language with multiple inheritance (also TypeScript has intersections). But not this year, we already have lots on our roadmap.
Just ran into this:
import matplotlib.axes as a
def plot_into(..., ax: Intersection[a.Axes, a.SubplotBase]):
ax.set_frame_on(False)
ax.set_xticks([])
ax.set_yticks([])
ax.get_subplotspec().subgridspec(...)
...For us, having Intersection would be an advantage as we use sphinx-autodoc-typehints and Intersection[...] would keep us from having to create, export, and document protocol classes (or ABCs).
(also I’m not sure if we can define a protocol as inheriting from two non-protocols)
@flying-sheep PEP-544 protocols are independent from the implementations, see https://gist.github.com/ysangkok/3668fc669c0aa8479a932405795fd4d3
Shouldn't the typing library be abstracted from whether or not a particular linter (mypy) handles a particular feature? The annotations act as a hint to the developer, if nothing else.
PyCharm and IntelliJ already appear to support "Intersection" (though I'd note they don't properly support Union at present!):
class A:
alpha = 1
class B:
beta = 2
def foo( x : object ):
assert isinstance( x, A ) and isinstance( x, B )
print( x.alpha )
print( x.beta )
print( x.gamma ) # ERROR! Cannot find reference 'gamma' in 'A | object | B'
PyCharm and IntelliJ already appear to support "Intersection" (though I'd note they don't properly support Union at present!)
Intersections are potentially much easier to support without proper support for all other PEP 484 features. Mypy also supports a limited form of intersection types, but it really is quite limited. Does PyCharm support arbitrary intersection types correctly, such as an intersection between a tuple, a callable and a type variable?
I think that it's important that we show that all typing features can be supported together. Each additional feature may increase the complexity of implementation exponentially, and having a standard that is effectively impractical to fully support would be unfortunate.
Does PyCharm support arbitrary intersection types correctly, such as an intersection between a tuple, a callable and a type variable?
Not as far as I'm aware - there's really no way of specifying such a construct since isinstance( x, TypeVar( y ) ) isn't a valid call.
My use case is to have stricter type checking on an aiohttp Application. These objects support a dict-like interface to provide a shared config across the application. So, I think what I'm looking for is something like:
class MyAppDict(TypedDict):
api_client: ClientSession
db: Pool
MyApp = Intersection[Application, MyAppDict]
Perhaps a class MyApp(Application, MyAppDict) could also work, except that TypedDict can't be used as a subclass like that.
The problem we’re trying to solve is that it’s currently impossible to say “I accept anything that subclasses/implements A and B” if one of A or B is a concrete class.
Therefore another possibility would be to treat classes as protocols.
Let’s say the syntax is Protocol[ConcreteClass] or so. Then we could e.g. do
from typing import Protocol
from sqlalchemy_mixins import SmartQueryMixin
class DeclarativeBase(Protocol):
metadata: MetaData
__tablename__: str
__table__: Table
__mapper__: Mapper
class SmartBase(DeclarativeBase, Protocol[SmartQueryMixin]):
passAnother potential use case: Often when we annotate that something is a np.ndarray, we also want to specify how many dimensions that array has. Intersection[Sequence[float], np.ndarray] specifies a one-dimensional numpy array of floats. Intersection[Sequence[Sequence[float]], np.ndarray] specifies a two-dimensional array, etc.
@charles-staats: Numpy itself might also benefit from this (numpy/numpy#16547 (comment)) to allow Intersection[np.Array[float], np.Shaped[N,M]]
Now that Python 3.10 introduced using | for Union shorthand, we could start thinking about using & for intersection shorthand like TypeScript does.
def fun(a: FooClass & BarMixin):
...@johnthagen Would be very welcome, yes. But is anyone stepping up to implement it in mypy?
@intgr I just want to add as motivation for anyone who's thinking about it, I think intersections would solve many existing issues.
python/mypy#9424 would be solved if mypy would do the correct thing of intersecting int with U rather than trying to narrow (neither type is narrower than the other). Similarly, the duplicate issue python/mypy#5720 would be solved by intersecting int with T.
This would also solve typing issues with libraries that add Mixins. In particular, I am using transitions which creates dynamic mixin methods for each "trigger" you implement.
For obvious reasons, these are not reflected for static typing analysis. I am happy to implement a protocol detailing the mixin values, but when type hinting self on a method that uses one such mixin, I get type errors for methods belonging to my natural class. If I could type hint to an Interface between the natural class and the protocol though, it'd work peachy keen!
Is this in active development? I encountered a number of situations in which this would have been useful. Would be invaluable for proper typing imo.
Just found myself looking for this. Any updates?
I'm not aware of any active plans in this area. If someone wants to see this feature done, they should work on drafting a PEP and a reference implementation in one of the major type checkers.
I would love that feature too, I really hope someone is working on it. But this is the only place where I see it discussed so I doubt that.
My use case is about dynamic classes created with type(). I don't want to create ~20 Mixin.
Maybe it's not the place to ask, but does anyone know why it's not in the python typing module since the beginning? It look like as basic to me as the Union type.
@thibaut-st have a read of the original issue #18
I would be happy with Protocols as a solution also, but right now they don't work since all bases of a Protocol have to also be a Protocol.
@mwerezak I agree, intersections that are limited to e.g. only protocols, or one concrete type + n protocols would be useful in my opinion.
Protocols don't solve the general problem, which is that mypy needs to keep track of intersection types internally. For example, assert isinstance needs to intersect the type--not set it. There are many examples where setting it causes problems.
Intersection types could be useful for fast ad-hoc protocols, especially IO protocols:
def foo(file: HasRead & HasSeek) -> None:
passI see that this issue is still open.
I'm not really aware about how the implementation of features in python is selected (or where to check what's planned), does someone know if the development is in the pipe?
I still have situation where it would be useful now and then.
As I wrote above, I don't think there are any active plans here. If you want to see it forward, I'd suggest you write an implementation for your type checker of choice and a PEP.
I may add an implementation to https://github.com/quora/pyanalyze in the near future, though.
I'd suggest you write an implementation for your type checker of choice and a PEP
I would love too, but I think it's beyond my capacity of development.
We would also benefit from Intersection[] type in typed-django / django-stubs. We need to model Manager.from_queryset and Queryset.from_manager methods. Right now we have a hacky Intersection[] ad-hoc implementation just for this case: https://github.com/typeddjango/django-stubs/blob/8f97bf880d3022772581ea4cf8b5bf5297a27bad/mypy_django_plugin/transformers/managers.py#L8
You can create arbitrary Intersections of Protocols.
class Foo(Protocol):
def foo(self) -> None: ...
class Bar(Protocol):
def bar(self) -> None: ...
class Impl:
def foo(self) -> None: ...
def bar(self) -> None: ...
class FooBar(Protocol, Foo, Bar): ...
def foo(it: FooBar):
print(it)
foo(Impl()) # correctly typecheckedIf only there was a way to specify that an instance of the protocol must have a set of bases:
class Foo: ...
class Bar: ...
class FooBar(Protocol, bases=(Foo, Bar)): # the set of bases that instances of this protocol must have in order to be considered a 'FooBar'
...But how neater and easier would it be to simply have to do something like that?
class Foo:
def foo(self):
pass
class Bar:
def bar(self):
pass
class FooBar1(Foo, Bar):
def far(self):
pass
class FooBar2(Foo, Bar):
def boo(self):
pass
def foobar_func(foobar: Foo & Bar): # or (foobar: Intersect[Foo, Bar])
foobar.foo()
foobar.bar()But I'll wait patiently to see if someone better than me can come with a convincing PEP.
After all it's not really an issue, more like a nice to have.
@KotlinIsland I don't follow. A protocol, by its very nature, doesn't demand nominal structure: it doesn't say which bases are required
@thibaut-st it can be an issue: if Foo and Bar aren't protocols and imported from a third-party library, I don't believe there's a decent workaround
@joelberkeley yeah I get that, and the issue here is intersections which would be better served with a dedicated type. Buuut if a protocol could define required bases you could create mixed nominal/structural types(although I don't know how useful that would be):
class Foo:
def foo() -> None: ...
class FooAndMore(Protocol, bases=(Foo,)):
def more() -> None: ...
class Impl(Foo):
def more() -> None:
passI'm sure at some point mixed structural/nominal types will be needed, this seems like a good way of doing it to me.
Or maybe just:
class Foo:
def foo() -> None: ...
class More(Protocol):
def more() -> None: ...
class Impl(Foo):
def more() -> None:
pass
FooAndMore = Foo & More
def eggs(f: Foo & More)Would have needed intersections today (and a couple times in the past), and now had to leave some parts untyped.
In my case, I have a decorator that modifies a class type that's passed into it by attaching extra metadata. As far as I know, there simply isn't a way to type that decorator properly at the moment. So now all classes using that decorator need to explicitly declare the extra metadata which the decorator will provide if they want mypy to be aware of those fields existing.
What should be the expected behavior in the case of conflicting types? For example:
class A(Protocol):
def foo(self) -> int: ...
class B(Protocol):
def foo(self) -> str: ...
def call_foo(x: A & B):
return x.foo() # oh no, x.foo() must be both an `int` and a `str`!Presumably, it would be int & str, which doesn't work (at least at runtime), but other base classes should fare better when intersected.
What should be the expected behavior in the case of conflicting types? For example:
class A(Protocol): def foo(self) -> int: ... class B(Protocol): def foo(self) -> str: ... def call_foo(x: A & B): return x.foo() # oh no, x.foo() must be both an `int` and a `str`!
I don't really see the issue, as it's the same case as multiple inheritance with the same method name.
Personnally I would use the same behavior, the first type have the precedence.
(in this example, x.foo() type in call_foo(x: A & B) should be int)
@thibaut-st I think that would break the Liskov Substitution Principle. An object of type A & B should be expected to behave both as an A and as a B. The only way to fulfill that would be for foo to return int & str.
For the same reason mypy gives a type error if you try to subclass A and B. See https://mypy-play.net/?mypy=latest&python=3.10&gist=23c0fe765069a25d7f6c4905483ab351
UPDATE:
It's actually been too long since I thought about this.... The reply below doesn't make sense at all. An Interection as discussed here, is an intersection like in PEP483:
Intersection[t1, t2, ...]. Types that are subtype of each of t1, etc are subtypes of this. (Compare to Union, which has at least one instead of each in its definition.)
So not a set intersection at all.
In this case I agree with @antonagestam -- it should just be disallowed if the things intersecting are not compatible, like in (multiple) inheritance.
Original (incorrect, for historical accuracy only):
I would make it work like in set theory:
A is the set {foo(self) -> int} and B the set {foo(self) -> str} (definitely simplifying here). The intersection is the empty set (so equal to class C(Protocol): pass). In this case probably you could want mypy to give a warning, however if both A and B share another method or property, that should be kept.
class A(Protocol):
x: int
def foo(self) -> int: ...
class B(Protocol):
x: int
def foo(self) -> str: ...
class B(Protocol):
x: int
So in this case A & B will be equal to C (for typing purposes).
I do feel this opens the door for many more corner cases.....
class Parent: ....
class Child(Parent): ....
class A(Protocol):
def foo(self) -> Parent: ...
def baz(self) -> Optional[int]: ....
def bar(self, *args) -> int: ....
class B(Protocol):
def foo(self) -> Child: ...
def baz(self) -> int: ....
def bar(self, x: int) -> int: ....
I do feel there are probably good answers for all of them if we think about them enough.
@reinhrst I agree, it probably makes a whole lot more sense to just forbid intersections of incompatible types, instead of expecting a return type of int & str. It would be interesting to compare this with other languages. What does TypeScript do for instance?
@reinhrst I agree, it probably makes a whole lot more sense to just forbid intersections of incompatible types, instead of expecting a return type of
int & str. It would be interesting to compare this with other languages. What does TypeScript do for instance?
type StrNum = string & number
type IsItNever = StrNum extends never ? true : false
It becomes never, in this example the IsItNever type would be true if that type is never(NoReturn in python land), and it is true.
Although I think whats happening there is very different to Python. In TypeScript, string and number are incompatible(and therefore become never) from a type standpoint because they are effectively final. In python there is theoretically nothing stopping a type being both of those types:
class StrInt(str, int):
passThis specific type is invalid as the definitions in str and int are incompatible(Definition of "__gt__" in base class "str" is incompatible with definition in base class "int" etc) and will fail at runtime due to a layout incompatibility(TypeError: multiple bases have instance lay-out conflict).
Due to that fact I would still expect str & int to become NoReturn but not for the same reasons as in TypeScript.
In python there is theoretically nothing stopping a type being both of those types
I thought so too, but that actually fails at runtime:
>>> class StrInt(str, int): ...
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: multiple bases have instance lay-out conflictNote that there actually is an overlap between Python's int and str objects: they both inherit from the object base class.
I think accessing __doc__ and __str__ of an int & str type should be allowed -- so maybe the type checker should only error when using attributes that have conflicting definitions, like __gt__?
However, if there are conflicting attributes, casting int & str back to say str would erase those conflicts. So maybe a downcast from Intersection back to a component type also needs to ensure there are no conflicting attributes?
@antonagestam And at type time, I talked to that in that comment.
@KotlinIsland Sorry about that, have to blame it on the current day of the week ;)
@intgr But, the whole point of having intersection types is to be able to do this:
def takes_a(val: A): ...
def takes_b(val: B): ...
def takes_ab(val: A & B):
takes_a(val)
takes_b(val)As long as A and B are compatible, that needs to be completely valid code in order for intersections to be useful. There shouldn't have to be any narrowing or casting in any of the involved functions in that example. So if the type checker knows that A and B aren't compatible, it doesn't really make sense to not error in the definition of takes_ab. And since takes_a and takes_b can be used individually with instances of A and B, it doesn't make sense for there to be a type error within those definitions.
@thibaut-st I think that would break the Liskov Substitution Principle. An object of type
A & Bshould be expected to behave both as an A and as a B. The only way to fulfill that would be for foo to returnint & str.For the same reason mypy gives a type error if you try to subclass A and B. See https://mypy-play.net/?mypy=latest&python=3.10&gist=23c0fe765069a25d7f6c4905483ab351
Yes, but you can use python and typing without Mypy, and in python you absolutely can subclass the example.
But it was just the first thing popping to my mind, I guess there is better solutions (or if not better, alternative ones).
My intuition tells me that intersection types should behave the same way as if we were to create a new class/protocol using @overloads. For instance, in the next example, C should be equivalent to A & B:
class A(Protocol):
age: int
verbose: int
element: Proto1 # `Proto1` is an arbitrary protocol
def foo(self, x: int) -> float: ...
class B(Protocol):
age: int
verbose: bool
element: Proto2 # `Proto2` is an arbitrary protocol
name: str
def foo(self, x: str) -> None: ...
def g(self, y: int) -> int: ...
class C(Protocol):
age: int # `age` is found in both `A` and `B` with type `int`; nothing wrong here
verbose: bool # the intersection between `bool` and `int` is `bool`
element: Proto1 & Proto2 # for arbitrary member variables, we calculate their intersection type
name: str # since `name` is found in `B`, it is required here
# any subtype of `A & B` must support the following overloads:
@overload
def foo(self, x: int) -> float: ...
@overload
def foo(self, x: str) -> None: ...
# `g` is found in `B`, so it must be present here as well:
def g(self, y: int) -> int: ...So far so good, I think we all agree here, right?
Now the question is what to do whenever we find "incompatible" overloads, as in:
class A(Protocol):
def foo(self) -> int: ...
def bar(self, x: int) -> bool: ...
class B(Protocol):
def foo(self) -> str: ...
def bar(self, x: bool) -> int: ...My opinion is that an error should be raised here, as happens when you try to @overload foo and bar with the signatures shown above.
Regarding the int & str example, I believe that this should be equivalent to trying to subclass both of them with class IntStr(int, str). Since IntStr(int, str) fails, int & str should fail too.
@ruancomelli If subclassing is the appropriate metaphor, then AB(A, B) will choose methods from A first, right? In that case, another option is for A & B to be an ordered operator that chooses methods from A in case of conflict.
That said, the error in case of conflict idea makes sense too.
In that case, another option is for A & B to be an ordered operator that chooses methods from A in case of conflict.
Well, that makes A & B incompatible with B, and my simple example above would fail:
def takes_a(val: A): ...
def takes_b(val: B):
# Since `B.foo()` returns `int` we should expect `ret` to be `int` here, but because we allowed
# incompatible types, e.g. AB(A, B), it will be str.
ret = val.foo()
def takes_ab(val: A & B):
takes_a(val)
takes_b(val)@antonagestam You're right. The error makes more sense then. (And therefore it's not as simple as relating it to inheritance.)
Well, perhaps my example worked against me... It seems more than intuitive that A & B should be exactly the same as B & A, in which case an ordered operator is unsuitable. Intuitively we should have issubtype(X, A & B) iff issubtype(X, A) and issubtype(X, B). For concrete classes A and B, this would also be equivalent to issubclass(X, A) and issubclass(X, B). All of this seems to be incompatible with the way multiple inheritance works in Python.
Perhaps we are mixing two concepts here, each one requiring its own operator?
- an
Intersection[A, B](orA & B) that would requireA & Bto be replaceable by bothAandBeverywhere. This operator would satisfy @antonagestam's example's requirements, but would raise a type error for e.g.int & strsince they are incompatible with each other. - an
InheritsFrom[A, B](please choose a better name) that would mean "anything that derives fromAandB, in that order". SoAB = InheritsFrom[A, B]would be equivalent to subclassingclass AB(A, B), considering method overriding and everything.
For now, I would focus only on the Intersection operator; the InheritsFrom looks a bit more niche and error-prone.
class A:
def foo(self, a: Foo) -> Baz: ...
class B:
def foo(self, b: Bar) -> Qux: ...
ab: A & B
reveal_type(ab.foo) # Callable[[Foo & Bar], Baz | Qux]I would think that when two classes are intersected, their methods would merge and the input parameters would become intersections, and their outputs would become unions.
This is assuming that the type operator & is commutative and not ordered.
class A: def foo(self, a: Foo) -> Baz: ... class B: def foo(self, b: Bar) -> Qux: ... ab: A & B reveal_type(ab.foo) # Callable[[Foo & Bar], Baz | Qux]I would think that when two classes are intersected, their methods would merge and the input parameters would become intersections, and their outputs would become unions.
This is assuming that the type operator
&is commutative and not ordered.
This breaks LSP. An A & B should be useable as an A, but Callable[[Foo & Bar], Baz | Qux] isn't compatible with Callable[[Foo], Baz]
Well, perhaps my example worked against me... It seems more than intuitive that
A & Bshould be exactly the same asB & A, in which case an ordered operator is unsuitable. Intuitively we should haveissubtype(X, A & B)iffissubtype(X, A) and issubtype(X, B). For concrete classesAandB, this would also be equivalent toissubclass(X, A) and issubclass(X, B). All of this seems to be incompatible with the way multiple inheritance works in Python.Perhaps we are mixing two concepts here, each one requiring its own operator?
* an `Intersection[A, B]` (or `A & B`) that would require `A & B` to be replaceable by both `A` and `B` everywhere. This operator would satisfy @antonagestam's example's requirements, but would raise a type error for e.g. `int & str` since they are incompatible with each other. * an `InheritsFrom[A, B]` (please choose a better name) that would mean "anything that derives from `A` and `B`, in that order". So `AB = InheritsFrom[A, B]` would be equivalent to subclassing `class AB(A, B)`, considering method overriding and everything.For now, I would focus only on the
Intersectionoperator; theInheritsFromlooks a bit more niche and error-prone.
I'm sure I'm missing something, but I can't figure out a way something would match A & B without being an inheriting class of (A, B). (and so, why intersection and inheritsFrom would differ)
this breaks LSP
Oh right, my bad. The methods would join as overloads.
class A:
def foo(self, a: Foo) -> Baz: ...
class B:
def foo(self, b: Bar) -> Qux: ...
ab: A & B
reveal_type(ab.foo) # overloaded method: (Foo) -> Baz and (Bar) -> Qux@KotlinIsland I think you're mistaken. They cannot be overloads since that violates LSP. And I think LSP applies because A & B needs to be usable as an A or a B.
Thinking about this a bit more, should this be allowed?
class A: pass
class B: pass
class AB(A, B): pass
class X:
def f(self, x: A) -> A:
...
class Y:
def f(self, x: B) -> B:
...
class XY(X, Y):
...
def f(x_and_y: X & Y, ab: A & B) -> A & B:
return x_and_y.f(ab)
f(XY(), AB()) # okay.In other words, intersecting two classes intersects all the methods, which means takes the intersection of all their parameters and return values. In this way, an A & B is usable as an A or a B.
Not really. Someone mentioned before that methods join as overloads, which is more accurate and less restrictive. Only if all the method parameters are equal, the return type should be the intersection, IMHO.
I'm sure I'm missing something, but I can't figure out a way something would match A & B without being an inheriting class of (A, B)
@thibaut-st if A or B (or both) are protocols or typing constructs, you won't necessarily be able to subclass them (take A = Union[int, float], for instance). But yes, for concrete types A and B, I believe that A & B must inherit from both A and B, even if indirectly.
and so, why intersection and inheritsFrom would differ
Disclaimer: InheritsFrom is just a draft idea, I don't know if it makes sense to have this. But the difference is that, given
class A:
def foo(self, x: int) -> None: ...
class B:
def foo(self, x: str) -> None: ...
class C:
def foo(self, x: int) -> str: ...A & Bwould be equivalent to a protocol with two overloads forfoo:(x: int) -> Noneand(x: str) -> None.InheritsFrom[A, B]'sfoowould have only one signature:(x: int) -> Nonebecause, if you were to subclassAB(A, B), thenAB.foowould be equivalent toA.foo.A & Cwould give an error, because you cannot have the overloads(x: int) -> Noneand(x: int) -> strat the same time (same input types, different outputs).InheritsFrom[A, C]would not raise any errors because, again,AC.foo's signature would just be equal toA.foo's signature due to inheritance rules.
@ruancomelli Is there a realistic use case of InheritsFrom? I think it's a good thought exercise, but most people who want the feature in this thread want Intersection.
Not really. Someone mentioned before that methods join as overloads, which is more accurate and less restrictive. Only if all the method parameters are equal, the return type should be the intersection, IMHO.
Methods don't join as overloads for an intersection since that's an LSP violation.
Methods don't join as overloads for an intersection since that's an LSP violation.
Care to elaborate? Expanding the accepted signatures should not break LSP. Do you have a particular example in mind?
@vnmabus What about this?
class A:
def foo(self, a: Foo) -> Baz: ...
class B:
def foo(self, a: Foo) -> Qux: ...
ab: A & B
reveal_type(ab.foo(foo)) # ??If it were an A, it would promise a reveal of Baz, if it were a B, it would promise Qux. I think it should promise Baz & Qux, whereas you're suggesting it should be Baz | Qux. (Edit: we both agree, and I misunderstood.)
I took that into account:
Only if all the method parameters are equal, the return type should be the intersection, IMHO.