[Python-ideas] PEP 561: Distributing and Packaging Type Information

Ethan Smith ethan at ethanhs.me
Mon Sep 11 01:15:40 EDT 2017 

On Sun, Sep 10, 2017 at 6:21 PM, Jelle Zijlstra <jelle.zijlstra at gmail.com>
wrote:

>
>
> 2017-09-10 18:10 GMT-07:00 Ethan Smith <ethan at ethanhs.me>:
>
>>
>>
>> On Sun, Sep 10, 2017 at 5:39 PM, Jelle Zijlstra <jelle.zijlstra at gmail.com
>> > wrote:
>>
>>> Congratulations on your first PEP! This is solving an important problem
>>> for typing in Python, so I'm glad we're tackling it.
>>>
>>
>> Thanks!
>>
>>>
>>> <snip>
>>>
>> Packaging Type Information
>>>> --------------------------
>>>>
>>>> Packages must opt into supporting typing. This will be done though a distutils
>>>> extension [2]_, providing a ``typed`` keyword argument to the distutils
>>>> ``setup()`` command. The argument value will depend on the kind of type
>>>> information the package provides. The distutils extension will be added to the
>>>> ``typing`` package. Therefore a package maintainer may write
>>>>
>>>> Is the addition to the `typing` package just a legacy feature for
>>> Python versions without typing in the standard library? This should be made
>>> explicit.
>>>
>>
>> The intent here is that the typing package would be required for the
>> extra setup keyword to work, otherwise it would fail.
>>
> Then I would have to install the `typing` PyPI package even if I am only
> using Python 3.7+? That seems suboptimal. Perhaps the new keyword can be
> part of Python core in 3.7 and added to `typing_extensions` for 3.5 and 3.6.
>

That would be acceptable.

> ::
>>>>
>>>>     setup(
>>>>         ...
>>>>         setup_requires=["typing"],
>>>>         typed="inline",
>>>>         ...
>>>>     )
>>>>
>>>> Inline Typed Packages
>>>> '''''''''''''''''''''
>>>>
>>>> Packages that have inline type annotations simply have to pass the value
>>>> ``"inline"`` to the ``typed`` argument in ``setup()``.
>>>>
>>>> Stub Only Packages
>>>> ''''''''''''''''''
>>>>
>>>> For package maintainers wishing to ship stub files containing all of their
>>>> type information, it is prefered that the ``*.pyi`` stubs are alongside the
>>>> corresponding ``*.py`` files. However, the stubs may be put in a sub-folder
>>>> of the Python sources, with the same name the ``*.py`` files are in. For
>>>> example, the ``flyingcircus`` package would have its stubs in the folder
>>>> ``flyingcircus/flyingcircus/``. This path is chosen so that if stubs are
>>>>
>>>> What if `flyingcircus` already contains a subpackage called
>>> `flyingcircus`? This might be theoretical but there's probably a package
>>> out there that does this.
>>>
>>
>> I considered this. I considered it as worrying too much. The alternative
>> would be to special case the name and have type checkers follow that.
>>
>
>>
> That's fair.
>
>> not found in ``flyingcircus/`` the type checker may treat the subdirectory as
>>>> a normal package. The normal resolution order of checking ``*.pyi`` before
>>>> ``*.py`` will be maintained. The value of the ``typed`` argument to
>>>> ``setup()`` is ``"stubs"`` for this type of distribution. The author of the
>>>> package is suggested to use ``package_data`` to assure the stub files are
>>>> installed alongside the runtime Python code.
>>>>
>>>> It would be helpful to have an example of what this looks like. This
>>> PEP will likely end up being the reference document for people looking to
>>> add typing support to their packages.
>>>
>>
>> I plan on writing examples of the distutils plugin and a sample package
>> tomorrow if I have time.
>>
>>> Third Party Stub Packages
>>>> '''''''''''''''''''''''''
>>>>
>>>> Third parties seeking to distribute stub files are encouraged to contact the
>>>> maintainer of the package about distribution alongside the package. If the
>>>> maintainer does not wish to maintain or package stub files or type information
>>>> inline, then a "third party stub package" should be created. The structure is
>>>> similar, but slightly different from that of stub only packages. If the stubs
>>>> are for the library ``flyingcircus`` then the package should be named
>>>> ``flyingcircus-stubs`` and the stub files should be put in a sub-directory
>>>> named ``flyingcircus``. This allows the stubs to be checked as if they were in
>>>> a regular package. These packages should also pass ``"stubs"`` as the value
>>>> of ``typed`` argument in ``setup()``. These packages are suggested to use
>>>> ``package_data`` to package stub files.
>>>>
>>>> The version of the ``flyingcircus-stubs`` package should match the version of
>>>> the ``flyingcircus`` package it is providing types for.
>>>>
>>>> What if I made stubs for flyingcircus 1.0 in my flyingcircus-stubs
>>> package, but then realized that I made a terrible mistake in the stubs? I
>>> can't upload a new version of flyingcircus-stubs 1.0 to PyPI, and I can't
>>> make flyingcircus-stubs have some other version than 1.0 because of this
>>> requirement.
>>>
>>> Another option is as follows:
>>> - Stub packages are versioned independently of the packages they provide
>>> stubs for.
>>> - There is a special global (say __version__) that can be used in stub
>>> packages and gets resolved to the version of the package that is being
>>> checked.
>>>
>>> The contents of flyingcircus-stubs might look like:
>>>
>>> import enum
>>>
>>> class Swallow(enum.Enum):
>>>    african = 0
>>>    european = 1
>>>    if __version__ >= (2, 0):
>>>       asian = 2
>>>
>>> This option doesn't have the problem I described above, but it requires
>>> this magical __version__ variable.
>>>
>>
>> Guido has said he doesn't like this idea (https://github.com/python/typ
>> ing/issues/84#issuecomment-318256377), and I'm not convinced it is worth
>> the complications it involves
>>
> But Guido's comment implies that people could use an independent
> versioning scheme in their stubs package (his "django-1.1-stubs" example).
> The PEP makes the situation worse because the version is fixed.
>
> I agree that my proposal introduces a lot of complication, but I don't
> think what the PEP proposes is workable (if django-stubs 1.1 got the stubs
> for django 1.1 wrong, there is no second chance). We get enough bugs in
> typeshed to know that stub packages will need updates independent from the
> package they're providing stubs for.
>
> Hopefully we can come up with something that allows stub packages to be
> versioned separately. Guido's comment actually suggests a way forward: What
> if stub packages can declare (in their package metadata or something) what
> version of the package they are type checking? That way, django1.1-stubs
> 0.1 can provide buggy stubs for django 1.1, and then fix them in
> django1.1-stubs 0.2.
>

After thinking about this for a while, I have come to believe metadata is
the solution. Keeping the versioning in the name becomes a burden on both
the maintainer and PyPI. I think the best solution is to leverage existing
metadata and have the "stub only" packages list the package versions they
support via the install_requires keyword, and I believe type checkers can
verify that rather easily. So django-stubs would list e.g.
django>=1.1.0,django<1.2.0 or whatever the stub maintainer wishes to
support. Then the package of stubs can use normal versioning.

I think I have reached the point where I should go back and work on some
changes to the PEP barring objection to your feedback and/or my suggested
amendments.

> Type Checker Module Resolution Order
>>>> ------------------------------------
>>>>
>>>> The following is the order that type checkers supporting this PEP should
>>>> resolve modules containing type information:
>>>>
>>>> 1. User code - the files the type checker is running on.
>>>>
>>>> 2. Stubs or Python source in ``PYTHONPATH``. This is to allow the user
>>>>    complete control of which stubs to use, and patch broken stubs/inline
>>>>    types from packages.
>>>>
>>>> Current type checkers don't use PYTHONPATH as far as I know, because
>>> they may not run under the same Python version or environment as the code
>>> to be type checked. I don't think we should require type checkers to listen
>>> to PYTHONPATH; perhaps we should just say that type checkers should provide
>>> a way for users to put code at the beginning of the search path.
>>>
>>
>> I agree.
>>
>>> 3. Third party stub packages - these packages can supersede the installed
>>>>    untyped packages. They can be found at ``pkg-stubs`` for package ``pkg``,
>>>>    however it is encouraged to check their metadata to confirm that they opt
>>>>    into type checking.
>>>>
>>>> The metadata of the stubs package? I'm not sure why that makes much
>>> sense here.
>>>
>>
>> Essentially, a package opts into being checked via a setup() keyword.
>> That keyword is put in the packages metadata.
>>
>>> 4. Inline packages - finally, if there is nothing overriding the installed
>>>>    package, and it opts into type checking.
>>>>
>>>> 5. Typeshed (if used) - Provides the stdlib types and several third party libraries
>>>>
>>>> When resolving step (3) type checkers should assure the version of the stubs
>>>> match the installed runtime package.
>>>>
>>>> Type checkers that check a different Python version than the version they run
>>>> on must find the type information in the ``site-packages``/``dist-packages``
>>>> of that Python version. This can be queried e.g.
>>>> ``pythonX.Y -c 'import sys; print(sys.exec_prefix)'``. It is also recommended
>>>> that the type checker allow for the user to point to a particular Python
>>>> binary, in case it is not in the path.
>>>>
>>>> To check if a package has opted into type checking, type checkers are
>>>> recommended to use the ``pkg_resources`` module to query the package
>>>> metadata. If the ``typed`` package metadata has ``None`` as its value, the
>>>> package has not opted into type checking, and the type checker should skip that
>>>> package.
>>>>
>>>>
>>>> References
>>>> ==========
>>>>
>>>> .. [1] PEP 484, Storing and Distributing Stub Files
>>>>    (https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-files)
>>>>
>>>> .. [2] Distutils Extensions, Adding setup() arguments
>>>>    (http://setuptools.readthedocs.io/en/latest/setuptools.html#adding-setup-arguments)
>>>>
>>>> Copyright
>>>> =========
>>>>
>>>> This document has been placed in the public domain.
>>>>
>>>>
>>>> 

>>>> ..
>>>>    Local Variables:
>>>>    mode: indented-text
>>>>    indent-tabs-mode: nil
>>>>    sentence-end-double-space: t
>>>>    fill-column: 70
>>>>    coding: utf-8
>>>>    End:
>>>>
>>>>
>>>> _______________________________________________
>>>> Python-ideas mailing list
>>>> Python-ideas at python.org
>>>> https://mail.python.org/mailman/listinfo/python-ideas
>>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20170910/4f095b55/attachment-0001.html>

More information about the Python-ideas mailing list