[Python-Dev] Status of PEP 484 and the typing module

[Guido van Rossum]

Hi Mark,

We're down to the last few items here. I'm CC'ing python-dev so folks can
see how close we are. I'll answer point by point.

On Thu, May 21, 2015 at 6:24 AM, Mark Shannon <mark at hotpy.org> wrote:

> Hi,
>
> The PEP itself is looking fairly good.
>

I hope you'll accept it at least provisionally so we can iterate over the
finer points while a prototype of typing.py in in beta 1.


> However, I don't think that typing.py is ready yet, for a number of
> reasons:
>
> 1.
> As I've said before, there needs to be a distinction between classes and
> types.
> They is no need for Any, Generic, Generic's subtypes, or Union to subclass
> builtins.type.
>

I strongly disagree. They can appear in many positions where real classes
are acceptable, in particular annotations can have classes (e.g. int) or
types (e.g. Union[int, str]).


> Playing around with typing.py, it has also become clear to me that it
> is also important to distinguish type constructors from types.
>
> What do I mean by a type constructor?
> A type constructor makes types.
> "List" is an example of a type constructor. It constructs types such as
> List[T] and List[int].
> Saying that something is a List (as opposed to a list) should be rejected.
>

The PEP actually says that plain List (etc.) is equivalent to List[Any].
(Well, at least that's the intention; it's implied by the section about the
equivalence between Node() and Node[Any]().


> 2.
> Usability of typing as it stands:
>
> Let's try to make a class that implements a mutable mapping.
>
> >>> import typing as tp
> #Make some variables.
> >>> T = tp.TypeVar('T')
> >>> K = tp.TypeVar('K')
> >>> V = tp.TypeVar('V')
>
> #Then make our class:
>
> >>> class MM(tp.MutableMapping): pass
> ...
> #Oh that worked, but it shouldn't. MutableMapping is a type constructor.
>

It means MutableMapping[Any].


> #Let's make one
> >>> MM()
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/home/mark/repositories/typehinting/prototyping/typing.py", line
> 1095, in __new__
>     if _gorg(c) is Generic:
>   File "/home/mark/repositories/typehinting/prototyping/typing.py", line
> 887, in _gorg
>     while a.__origin__ is not None:
> AttributeError: type object 'Sized' has no attribute '__origin__'
>
> # ???
>

Sorry, that's a bug I introduced in literally the last change to typing.py.
I will fix it. The expected behavior is

TypeError: Can't instantiate abstract class MM with abstract methods __len__



> #Well let's try using type variables.
> class MM2(tp.MutableMapping[K, V]): pass
> ...
> >>> MM2()
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/home/mark/repositories/typehinting/prototyping/typing.py", line
> 1095, in __new__
>     if _gorg(c) is Generic:
>   File "/home/mark/repositories/typehinting/prototyping/typing.py", line
> 887, in _gorg
>     while a.__origin__ is not None:
> AttributeError: type object 'Sized' has no attribute '__origin__'
>

Ditto, and sorry.

>
> At this point, we have to resort to using 'Dict', which forces us to
> subclass 'dict' which may not be what we want as it may cause metaclass
> conflicts.
>
> 3.
> Memory consumption is also a worry. There is no caching, which means every
> time I use "List[int]" as an annotation, a new class object is created.
> Each class may only be a few KB, but collectively this could easily add up
> to several MBs.
> This should be easy to fix.
>

I can work on this after the beta-1 release. Until then, type aliases can
be used to avoid redundant type creation (and often they are clearer anyway
:-).


> 4.
> PY2, etc. really need to go.
> Assuming that this code type checks OK:
>
>  if typing.PY2:
>      type_safe_under_py2_only()
>  else:
>      type_safe_under_py3_only()
>
> Is the checker supposed to pass this:
>
>  if sys.hexversion < 0x03000000:
>      type_safe_under_py2_only()
>  else:
>      type_safe_under_py3_only()
>
> If it should pass, then why have PY2, etc. at all.
> If it should fail, well that is just stupid and annoying.
>
> Pylint already understands version checks, as does our (Semmle's) checker.
> I suspect most IDEs do as well.
>

I have to negotiate this with Jukka but I think he'll agree.


> 5.
> Removing isinstance() support:
>
> As I said before, this is the job of a checker not typing.py.
>
> It also introduces some strange situations:
> D = tp.Dict[str,int]
> d = {}
> assert isinstance(d, D)
> d["x"] = None
> assert isinstance(d, D)
>
> In the above case the first check passes, and the second fails.
> But d is either of type D or it isn't. It can't be both, as types
> are static properties of programs, unlike classes.
>

Well, isinstance() is a dynamic function. The type checker has no authority
over its behavior beyond its signature.


> And it's broken anyway:
> >>> D = tp.Dict[str,'D']
> >>> d = {"x": {}}
> >>> isinstance(d, D)
> False
>

That's because _ForwardRef doesn't implement __instancheck__ or
__subclasscheck__. It's easily fixed.

>
> Realistically, I don't see typing.py being ready in time for 3.5.
> I'd be happy to be proved wrong.
>
> Cheers,
> Mark.
>
>
> P.S.
> I am worried by the lack of formal specification. It all seems a bit
> hand-waving. A formal spec reduces the likelihood of some unforeseen corner
> case being a permanent wart.
>

Formal specs are not my cup of tea. :-( (I'm not proud of this, but it just
is a fact -- see how terrible a job I've done of the Python reference
manual.) The best I could come up with is PEP 483.


> Take the recursive type above. There is no mention of recursive types in
> the PEP and they are clearly possible. Are they allowed?
>

They should be allowed. I imagine you could create one for which a naive
isinstance() imeplementation ends up in an infinite loop. That can be fixed
too (we fixed this for printing self-referential lists and dicts).


> I'm guessing that Jukka's thesis should cover a lot of this.
> Has it been published yet?
>

Hopefully Jukka can answer that. :-)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150521/409b5458/attachment.html>