[Python-ideas] PEP 561: Distributing and Packaging Type Information

Jelle Zijlstra jelle.zijlstra at gmail.com
Sun Sep 10 20:39:39 EDT 2017 

Congratulations on your first PEP! This is solving an important problem for
typing in Python, so I'm glad we're tackling it.

2017-09-10 16:05 GMT-07:00 Ethan Smith <ethan at ethanhs.me>:

> Hello,
>
> I have just published my first PEP, on packaging type information. I would
> appreciate comments and suggestions. The PEP can be found at
> https://www.python.org/dev/peps/pep-0561/
>
> I have also duplicated the text below.
>
> Thanks!
>
> ----------------------------------------------------------
>
>
> PEP: 561
> Title: Distributing and Packaging Type Information
> Author: Ethan Smith <ethan at ethanhs.me>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 09-Sep-2017
> Python-Version: 3.7
> Post-History:
>
>
> Abstract
> ========
>
> PEP 484 introduced type hints to Python, with goals of making typing
> gradual and easy to adopt. Currently, typing information must be distributed
> manually. This PEP provides a standardized means to package and distribute
> type information and an ordering for type checkers to resolve modules and
> collect this information for type checking using existing packaging
> architecture.
>
>
> Rationale
> =========
>
> PEP 484 has a brief section on distributing typing information. In this
> section [1]_ the PEP recommends using ``shared/typehints/pythonX.Y/`` for
> shipping stub files. However, manually adding a path to stub files for each
> third party library does not scale. The simplest approach people have taken
> is to add ``site-packages`` to their ``PYTHONPATH``, but this causes type
> checkers to fail on packages that are highly dynamic (e.g. sqlalchemy
> and Django).
>
> Furthermore, package authors are wishing to distribute code that has
> inline type information, and there currently is no standard method to
> distribute packages with inline type annotations or syntax that can
> simultaneously be used at runtime and in type checking.
>
> This feels like it should be the first paragraph: it describes the
important problem we're solving, and the first paragraph is just details on
why the previous solutions don't work. Perhaps you could talk more about
how people are running into problems because of the absence of a way to
distribute typed packages. For example, if you're working on a proprietary
codebase, it's likely that you're relying on other internal packages, but
there is no good way to do that right now (short of setting MYPYPATH). For
open source package, you can add them to typeshed, but that adds overhead
for the package maintainer and ties you to mypy's release cycle.

>
>
> Specification
> =============
>
> There are several motivations and methods of supporting typing in a package.                                                 This PEP recognizes three (3) types of packages that may be created:
>
> 1. The package maintainer would like to add type information inline.
>
> 2. The package maintainer would like to add type information via stubs.
>
> 3. A third party would like to share stub files for a package, but the
>    maintainer does not want to include them in the source of the package.
>
>
> Where does the typeshed repo fit in here? Does the PEP propose to
deprecate using typeshed for third-party packages, or should typeshed
continue to be the repository for major third-party packages? Either way,
it should be discussed.

> This PEP aims to support these scenarios and make them simple to add to
> packaging and deployment.
>
> The two major parts of this specification are the packaging specifications
> and the resolution order for resolving module type information. This spec
> is meant to replace the ``shared/typehints/pythonX.Y/`` spec of PEP 484 [1]_.
>
> Packaging Type Information
> --------------------------
>
> Packages must opt into supporting typing. This will be done though a distutils
> extension [2]_, providing a ``typed`` keyword argument to the distutils
> ``setup()`` command. The argument value will depend on the kind of type
> information the package provides. The distutils extension will be added to the
> ``typing`` package. Therefore a package maintainer may write
>
> Is the addition to the `typing` package just a legacy feature for Python
versions without typing in the standard library? This should be made
explicit.

>
> ::
>
>     setup(
>         ...
>         setup_requires=["typing"],
>         typed="inline",
>         ...
>     )
>
> Inline Typed Packages
> '''''''''''''''''''''
>
> Packages that have inline type annotations simply have to pass the value
> ``"inline"`` to the ``typed`` argument in ``setup()``.
>
> Stub Only Packages
> ''''''''''''''''''
>
> For package maintainers wishing to ship stub files containing all of their
> type information, it is prefered that the ``*.pyi`` stubs are alongside the
> corresponding ``*.py`` files. However, the stubs may be put in a sub-folder
> of the Python sources, with the same name the ``*.py`` files are in. For
> example, the ``flyingcircus`` package would have its stubs in the folder
> ``flyingcircus/flyingcircus/``. This path is chosen so that if stubs are
>
> What if `flyingcircus` already contains a subpackage called
`flyingcircus`? This might be theoretical but there's probably a package
out there that does this.

> not found in ``flyingcircus/`` the type checker may treat the subdirectory as
> a normal package. The normal resolution order of checking ``*.pyi`` before
> ``*.py`` will be maintained. The value of the ``typed`` argument to
> ``setup()`` is ``"stubs"`` for this type of distribution. The author of the
> package is suggested to use ``package_data`` to assure the stub files are
> installed alongside the runtime Python code.
>
> It would be helpful to have an example of what this looks like. This PEP
will likely end up being the reference document for people looking to add
typing support to their packages.

>
> Third Party Stub Packages
> '''''''''''''''''''''''''
>
> Third parties seeking to distribute stub files are encouraged to contact the
> maintainer of the package about distribution alongside the package. If the
> maintainer does not wish to maintain or package stub files or type information
> inline, then a "third party stub package" should be created. The structure is
> similar, but slightly different from that of stub only packages. If the stubs
> are for the library ``flyingcircus`` then the package should be named
> ``flyingcircus-stubs`` and the stub files should be put in a sub-directory
> named ``flyingcircus``. This allows the stubs to be checked as if they were in
> a regular package. These packages should also pass ``"stubs"`` as the value
> of ``typed`` argument in ``setup()``. These packages are suggested to use
> ``package_data`` to package stub files.
>
> The version of the ``flyingcircus-stubs`` package should match the version of
> the ``flyingcircus`` package it is providing types for.
>
> What if I made stubs for flyingcircus 1.0 in my flyingcircus-stubs
package, but then realized that I made a terrible mistake in the stubs? I
can't upload a new version of flyingcircus-stubs 1.0 to PyPI, and I can't
make flyingcircus-stubs have some other version than 1.0 because of this
requirement.

Another option is as follows:
- Stub packages are versioned independently of the packages they provide
stubs for.
- There is a special global (say __version__) that can be used in stub
packages and gets resolved to the version of the package that is being
checked.

The contents of flyingcircus-stubs might look like:

import enum

class Swallow(enum.Enum):
   african = 0
   european = 1
   if __version__ >= (2, 0):
      asian = 2

This option doesn't have the problem I described above, but it requires
this magical __version__ variable.

>
> Type Checker Module Resolution Order
> ------------------------------------
>
> The following is the order that type checkers supporting this PEP should
> resolve modules containing type information:
>
> 1. User code - the files the type checker is running on.
>
> 2. Stubs or Python source in ``PYTHONPATH``. This is to allow the user
>    complete control of which stubs to use, and patch broken stubs/inline
>    types from packages.
>
> Current type checkers don't use PYTHONPATH as far as I know, because they
may not run under the same Python version or environment as the code to be
type checked. I don't think we should require type checkers to listen to
PYTHONPATH; perhaps we should just say that type checkers should provide a
way for users to put code at the beginning of the search path.

>
> 3. Third party stub packages - these packages can supersede the installed
>    untyped packages. They can be found at ``pkg-stubs`` for package ``pkg``,
>    however it is encouraged to check their metadata to confirm that they opt
>    into type checking.
>
> The metadata of the stubs package? I'm not sure why that makes much sense
here.

>
> 4. Inline packages - finally, if there is nothing overriding the installed
>    package, and it opts into type checking.
>
> 5. Typeshed (if used) - Provides the stdlib types and several third party libraries
>
> When resolving step (3) type checkers should assure the version of the stubs
> match the installed runtime package.
>
> Type checkers that check a different Python version than the version they run
> on must find the type information in the ``site-packages``/``dist-packages``
> of that Python version. This can be queried e.g.
> ``pythonX.Y -c 'import sys; print(sys.exec_prefix)'``. It is also recommended
> that the type checker allow for the user to point to a particular Python
> binary, in case it is not in the path.
>
> To check if a package has opted into type checking, type checkers are
> recommended to use the ``pkg_resources`` module to query the package
> metadata. If the ``typed`` package metadata has ``None`` as its value, the
> package has not opted into type checking, and the type checker should skip that
> package.
>
>
> References
> ==========
>
> .. [1] PEP 484, Storing and Distributing Stub Files
>    (https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-files)
>
> .. [2] Distutils Extensions, Adding setup() arguments
>    (http://setuptools.readthedocs.io/en/latest/setuptools.html#adding-setup-arguments)
>
> Copyright
> =========
>
> This document has been placed in the public domain.
>
>
> 

> ..
>    Local Variables:
>    mode: indented-text
>    indent-tabs-mode: nil
>    sentence-end-double-space: t
>    fill-column: 70
>    coding: utf-8
>    End:
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20170910/07fa489d/attachment-0001.html>

More information about the Python-ideas mailing list