[Python-Dev] Experiences with Creating PEP 484 Stub Files

[Guido van Rossum]

[Just adding to Andrew's response]

On Tue, Feb 9, 2016 at 9:58 AM, Andrew Barnert via Python-Dev
<python-dev at python.org> wrote:
> On Feb 9, 2016, at 03:44, Phil Thompson <phil at riverbankcomputing.com> wrote:
>>
>> There are a number of things I'd like to express but cannot find a way to do so...
>>
>> - objects that implement the buffer protocol
>
> That seems like it should be filed as a bug with the typing repo. Presumably this is just an empty type that registers bytes, bytearray, and memoryview, and third-party classes have to register with it manually?

Hm, there's no way to talk about these in regular Python code either,
is there? I think that issue should be resolved first. Probably by
adding something to collections.abc. And then we can add the
corresponding name to typing.py. This will take time though (have to
wait for 3.6) so I'd recommend 'Any' for now (and filing those bugs).

>> - type objects

You can use 'type' for this (i.e. the builtin). You can't specify any
properties for types though; that's a feature request:
https://github.com/python/typing/issues/107 -- but it may be a while
before we address it (it's not entirely clear how it should work, and
we have many other pressing issues still).

>> - slice objects

> Can't you just use the concrete types type and slice tor these two? I don't think you need generic or abstract "any metaclass, whether inheriting from type or not" or "any class that meets the slice protocol", do you?

Can't you use 'slice' (i.e. the builtin)? Mypy supports that.

>> - capsules
>
> That one seems reasonable. But maybe there should just be a types.Capsule Type or types.PyCapsule, and then you can just check that the same as any other concrete type?
>
> But how often do you need to verify that something is a capsule, without knowing that it's the *right* capsule? At runtime, you'd usually use PyCapsule_IsValid, not PyCapsule_CheckExacf, right? So should the type checker be tracking the name too?
>
>> - sequences of fixed size (ie. specified in the same way as Tuple)

That's kind of a poor data structure. :-( Why can't you use Tuple here?

> How would you disambiguate between a sequence of one int and a sequence of 0 or more ints if they're both spelled "Sequence[int]"? That isn't a problem for Tuple, because it's assumed to be heterogeneous, so Tuple[int] can only be a 1-tuple. (This was actually discussed in some depth. I thought it would be a problem, because some types--including tuple itself--are sometimes used as homogenous arbitrary-length containers and sometimes as heterogeneous fixed-length containers, but Guido and others had some good answers for that, even if I can't remember what they were.)

We solved that by allowing Tuple[int, ...] to spell a homogeneous
tuple of integers.

>> - distinguishing between instance and class attributes.
>
> Where? Are you building a protocol that checks the data members of a type for conformance or something? If so, why is an object that has "spam" and "eggs" as instance attributes but "cheese" as a class attribute not usable as an object conforming to the protocol with all three attributes? (Also, does @property count as a class or instance attribute? What about an arbitrary data descriptor? Or a non-data descriptor?)

It's a known mypy bug. :-( It's somewhat convoluted to fix.
https://github.com/JukkaL/mypy/issues/1097

Some things Andrew snipped:

> The documentation is incomplete - there is no mention of Set or Tuple for example.

Tuple is here: https://docs.python.org/3/library/typing.html#typing.Tuple

collections.Set maps to typing.AbstractSet
(https://docs.python.org/3/library/typing.html#typing.AbstractSet;
present twice in the docs somehow :-( ). typing.Set (corresponding to
builtins.set) is indeed missing, I've a note of that:
http://bugs.python.org/issue26322.

> I found the documentation confusing regarding Optional. Intuitively it seems to be the way to specify arguments with default values. However it is explained in terms of (for example) Union[str, None] and I (intuitively but incorrectly) read that as meaning "a str or None" as opposed to "a str or nothing".

But it *does* mean 'str or None'. The *type* of an argument doesn't
have any bearing on whether it may be omitted from the argument list
by the caller -- these are orthogonal concepts (though sadly the word
optional might apply to both). It's possible (though unusual) to have
an optional argument that must be a str when given; it's also possible
to have a mandatory argument that may be a str or None.

Can you help improve the wording in the docs (preferably by filing an issue)?

> bytes can be used as shorthand for bytes, bytearray and memoryview - but what about objects that really only support bytes? Shouldn't the shorthand be called something like AnyBytes?

We debated that, but found it too annoying to have to import and write
write AnyBytes in so many places. The type checker may not be precise
for cases that only accept bytes, but hopefully it's more useful in
general this way.

> Is there any recommended way to test the validity and completeness of stub files? What's the recommended way to parse them?

That's also an open issue. For a quick check I tend to just point mypy
at a stub file, since it is the most mature implementation of PEP 484
to date (Google's pytype is still working on PEP 484 compatibility).
While this doesn't always catch all errors, it will at least find
syntax errors and cases that mypy doesn't support. :-)

-- 
--Guido van Rossum (python.org/~guido)