PEP 561: Distributing and Packaging Type Information
Hello all,
I have completed an implementation for PEP 561, and believe it is time to
share the PEP and implementation with python-dev
Python-ideas threads:
* PEP 561: Distributing and Packaging Type Information
* PEP 561 v2 - Packaging Static Type Information
* PEP 561: Distributing Type Information V3
The live version is here: https://www.python.org/dev/peps/pep-0561/
As always, duplicated below.
Ethan Smith
---------------------------------------------------
PEP: 561
Title: Distributing and Packaging Type Information
Author: Ethan Smith
Hi!
On Thu, Oct 26, 2017 at 03:42:19PM -0700, Ethan Smith
Post-History:
Not sure if postings to python-ideas count, but Post-History: 10-Sep-2017, 12-Sep-2017, 26-Oct-2017 Refs: https://mail.python.org/pipermail/python-ideas/2017-September/047015.html https://mail.python.org/pipermail/python-ideas/2017-September/047083.html
This PEP adds a new item to the ``*.distinfo/METADATA`` file
*.dist-info/ Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Thu, Oct 26, 2017 at 04:48:23PM -0700, Mariatta Wijaya
Not sure if postings to python-ideas count,
PEP 1 says:
Post-History is used to record the dates of when new versions of the PEP are posted to python-list and/or python-dev.
That's was added in 2003: https://hg.python.org/peps/annotate/96614829c145/pep-0001.txt https://github.com/python/peps/commit/0a690292ffe2cdc547dbad3bdbdb46672012b5... I don't remember if python-ideas has already been created. ;-)
So, no ?
I'm not so sure. :-)
Mariatta Wijaya
Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
I think python-ideas does count here. Many PEPs evolve mostly there.
On Oct 26, 2017 4:59 PM, "Oleg Broytman"
On Thu, Oct 26, 2017 at 04:48:23PM -0700, Mariatta Wijaya < mariatta.wijaya@gmail.com> wrote:
Not sure if postings to python-ideas count,
PEP 1 says:
Post-History is used to record the dates of when new versions of the PEP are posted to python-list and/or python-dev.
That's was added in 2003: https://hg.python.org/peps/annotate/96614829c145/pep-0001.txt https://github.com/python/peps/commit/0a690292ffe2cdc547dbad3bdbdb46 672012b536 I don't remember if python-ideas has already been created. ;-)
So, no ?
I'm not so sure. :-)
Mariatta Wijaya
Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ guido%40python.org
Ok I created an issue https://github.com/python/peps/issues/440, maybe
someone can work on updating the wordings in PEP 1 and PEP 12.
Thanks :)
Mariatta Wijaya
On Thu, Oct 26, 2017 at 5:03 PM, Guido van Rossum
I think python-ideas does count here. Many PEPs evolve mostly there.
Proposed pull request: https://github.com/python/peps/pull/441
On Thu, Oct 26, 2017 at 05:21:57PM -0700, Mariatta Wijaya
Ok I created an issue https://github.com/python/peps/issues/440, maybe someone can work on updating the wordings in PEP 1 and PEP 12. Thanks :)
Mariatta Wijaya
On Thu, Oct 26, 2017 at 5:03 PM, Guido van Rossum
wrote: I think python-ideas does count here. Many PEPs evolve mostly there.
Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Oct 26, 2017, at 20:03, Guido van Rossum
I think python-ideas does count here. Many PEPs evolve mostly there.
True, but there was some discussion of this way back when. The way I remember it was that, while there are many outlets to discuss PEPs (including those pointed to by the optional Discussions-To header), python-dev is the “forum of record”. This means that python-dev is the only mailing list you *have* to follow if you want to be informed of a PEP’s status in a timely manner. Thus Post-History is supposed to reflect the history of when the PEP is sent to python-dev. python-list is included because it’s the primary mailing list followed by people who aren’t developing Python but are still interested in it. Maybe this needs to be reconsidered here in 2017, but that’s the rationale for the wording of PEP 1. Cheers, -Barry
Heh, you're right that was the reasoning. But I think python-list is much
less valuable than python-ideas for PEP authors. So let's change it.
On Thu, Oct 26, 2017 at 6:38 PM, Barry Warsaw
On Oct 26, 2017, at 20:03, Guido van Rossum
wrote: I think python-ideas does count here. Many PEPs evolve mostly there.
True, but there was some discussion of this way back when.
The way I remember it was that, while there are many outlets to discuss PEPs (including those pointed to by the optional Discussions-To header), python-dev is the “forum of record”. This means that python-dev is the only mailing list you *have* to follow if you want to be informed of a PEP’s status in a timely manner. Thus Post-History is supposed to reflect the history of when the PEP is sent to python-dev. python-list is included because it’s the primary mailing list followed by people who aren’t developing Python but are still interested in it.
Maybe this needs to be reconsidered here in 2017, but that’s the rationale for the wording of PEP 1.
Cheers, -Barry
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ guido%40python.org
-- --Guido van Rossum (python.org/~guido)
On Oct 27, 2017, at 00:12, Guido van Rossum
Heh, you're right that was the reasoning. But I think python-list is much less valuable than python-ideas for PEP authors. So let's change it.
Sounds good. I just want to make sure we keep python-dev in the loop. This is a process change though, so I’ll work with the PR#441 author to get the update into the PEPs, and then make the announcement on the relevant mailing lists. Cheers, -Barry
Great!
On Fri, Oct 27, 2017 at 8:27 AM, Barry Warsaw
On Oct 27, 2017, at 00:12, Guido van Rossum
wrote: Heh, you're right that was the reasoning. But I think python-list is
much less valuable than python-ideas for PEP authors. So let's change it.
Sounds good. I just want to make sure we keep python-dev in the loop.
This is a process change though, so I’ll work with the PR#441 author to get the update into the PEPs, and then make the announcement on the relevant mailing lists.
Cheers, -Barry
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ guido%40python.org
-- --Guido van Rossum (python.org/~guido)
On Thu, Oct 26, 2017 at 4:48 PM, Mariatta Wijaya
Not sure if postings to python-ideas count,
PEP 1 says:
Post-History is used to record the dates of when new versions of the PEP are posted to python-list and/or python-dev.
So, no ?
Reading PEP 12, https://www.python.org/dev/peps/pep-0012/#id24 - Leave Post-History alone for now; you'll add dates to this header each time you post your PEP to python-list@python.org or python-dev@python.org. If you posted your PEP to the lists on August 14, 2001 and September 3, 2001, the Post-History header would look like: Post-History: 14-Aug-2001, 03-Sept-2001 You must manually add new dates and check them in. If you don't have check-in privileges, send your changes to the PEP editors. Perhaps it is outdated and needs to have python-ideas added? python-ideas was created around 2006 (according to the archives), so after PEP 1/12 were written.
This PEP adds a new item to the ``*.distinfo/METADATA`` file
*.dist-info/ Thank you for catching that. I will fix that with my next round of edits.
On Thu, Oct 26, 2017 at 3:42 PM, Ethan Smith
However, the stubs may be put in a sub-folder of the Python sources, with the same name the ``*.py`` files are in. For example, the ``flyingcircus`` package would have its stubs in the folder ``flyingcircus/flyingcircus/``. This path is chosen so that if stubs are not found in ``flyingcircus/`` the type checker may treat the subdirectory as a normal package.
I admit that I find this aesthetically unpleasant. Wouldn't something like __typestubs__/ be a more Pythonic name? (And also avoid potential name clashes, e.g. my async_generator package has a top-level export called async_generator; normally you do 'from async_generator import async_generator'. I think that might cause problems if I created an async_generator/async_generator/ directory, especially post-PEP 420.) I also don't understand the given rationale -- it sounds like you want to be able say well, if ${SOME_DIR_ON_PYTHONPATH}/flyingcircus/ doesn't contain stubs, then just stick the ${SOME_DIR_ON_PYTHONPATH}/flyingcircus/ directory *itself* onto PYTHONPATH, and then try again. But that's clearly the wrong thing, because then you'll also be adding a bunch of other random junk into that directory into the top-level namespace. For example, suddenly the flyingcircus.summarise_proust module has become a top-level summarise_proust package. I must be misunderstanding something?
Type Checker Module Resolution Order ------------------------------------
The following is the order that type checkers supporting this PEP should resolve modules containing type information:
1. User code - the files the type checker is running on.
2. Stubs or Python source manually put in the beginning of the path. Type checkers should provide this to allow the user complete control of which stubs to use, and patch broken stubs/inline types from packages.
3. Third party stub packages - these packages can supersede the installed untyped packages. They can be found at ``pkg-stubs`` for package ``pkg``, however it is encouraged to check the package's metadata using packaging query APIs such as ``pkg_resources`` to assure that the package is meant for type checking, and is compatible with the installed version.
Am I right that this means you need to be able to map from import names to distribution names? I.e., if you see 'import foo', you need to figure out which *.dist-info directory contains metadata for the 'foo' package? How do you plan to do this? The problem is that technically, import names and distribution names are totally unrelated namespaces -- for example, the '_pytest' package comes from the 'pytest' distribution, the 'pylab' package comes from 'matplotlib', and 'pip install scikit-learn' gives you a package imported as 'sklearn'. Namespace packages are also challenging, because a single top-level package might actually be spread across multiple distributions. -n -- Nathaniel J. Smith -- https://vorpus.org
On Fri, Oct 27, 2017 at 12:44 AM, Nathaniel Smith
On Thu, Oct 26, 2017 at 3:42 PM, Ethan Smith
wrote: However, the stubs may be put in a sub-folder of the Python sources, with the same name the ``*.py`` files are in. For example, the ``flyingcircus`` package would have its stubs in the folder ``flyingcircus/flyingcircus/``. This path is chosen so that if stubs are not found in ``flyingcircus/`` the type checker may treat the subdirectory as a normal package.
I admit that I find this aesthetically unpleasant. Wouldn't something like __typestubs__/ be a more Pythonic name? (And also avoid potential name clashes, e.g. my async_generator package has a top-level export called async_generator; normally you do 'from async_generator import async_generator'. I think that might cause problems if I created an async_generator/async_generator/ directory, especially post-PEP 420.)
I agree, this is unpleasant, I am now of the thought that if maintainers do not wish to ship stubs alongside their Python code, they should just create separate stub-only packages. I don't think there is a particular need to special case this for minor convenience. <snip>
Type Checker Module Resolution Order ------------------------------------
The following is the order that type checkers supporting this PEP should resolve modules containing type information:
1. User code - the files the type checker is running on.
2. Stubs or Python source manually put in the beginning of the path. Type checkers should provide this to allow the user complete control of which stubs to use, and patch broken stubs/inline types from packages.
3. Third party stub packages - these packages can supersede the installed untyped packages. They can be found at ``pkg-stubs`` for package ``pkg``, however it is encouraged to check the package's metadata using packaging query APIs such as ``pkg_resources`` to assure that the package is meant for type checking, and is compatible with the installed version.
Am I right that this means you need to be able to map from import names to distribution names? I.e., if you see 'import foo', you need to figure out which *.dist-info directory contains metadata for the 'foo' package? How do you plan to do this?
The problem is that technically, import names and distribution names
are totally unrelated namespaces -- for example, the '_pytest' package comes from the 'pytest' distribution, the 'pylab' package comes from 'matplotlib', and 'pip install scikit-learn' gives you a package imported as 'sklearn'. Namespace packages are also challenging, because a single top-level package might actually be spread across multiple distributions.
This is a problem. What I now realize is that the typing metadata is needed for *packages* and not distributions. I will work on a new proposal that makes the metadata per-package. It will require a slightly more complicated proposal, but I feel that it is necessary. Thank you for realizing this issue with my proposal, I probably should have caught it earlier. -n
-- Nathaniel J. Smith -- https://vorpus.org
On Thu, 26 Oct 2017 15:42:19 -0700
Ethan Smith
Stub Only Packages ''''''''''''''''''
For package maintainers wishing to ship stub files containing all of their type information, it is prefered that the ``*.pyi`` stubs are alongside the corresponding ``*.py`` files. However, the stubs may be put in a sub-folder of the Python sources, with the same name the ``*.py`` files are in. For example, the ``flyingcircus`` package would have its stubs in the folder ``flyingcircus/flyingcircus/``. This path is chosen so that if stubs are not found in ``flyingcircus/`` the type checker may treat the subdirectory as a normal package. The normal resolution order of checking ``*.pyi`` before ``*.py`` will be maintained.
I am not sure I understand the rationale for this. What would be the problem with looking for the stubs in a directory named, e.g; "flyingcircus/__typing__"? Regards Antoine.
On Fri, 27 Oct 2017 11:31:04 +0200
Antoine Pitrou
On Thu, 26 Oct 2017 15:42:19 -0700 Ethan Smith
wrote: Stub Only Packages ''''''''''''''''''
For package maintainers wishing to ship stub files containing all of their type information, it is prefered that the ``*.pyi`` stubs are alongside the corresponding ``*.py`` files. However, the stubs may be put in a sub-folder of the Python sources, with the same name the ``*.py`` files are in. For example, the ``flyingcircus`` package would have its stubs in the folder ``flyingcircus/flyingcircus/``. This path is chosen so that if stubs are not found in ``flyingcircus/`` the type checker may treat the subdirectory as a normal package. The normal resolution order of checking ``*.pyi`` before ``*.py`` will be maintained.
I am not sure I understand the rationale for this. What would be the problem with looking for the stubs in a directory named, e.g; "flyingcircus/__typing__"?
I just saw Nathaniel asked the same question above. Sorry for the noise! Regards Antoine.
participants (8)
-
Antoine Pitrou
-
Barry Warsaw
-
Ethan Smith
-
Guido van Rossum
-
Guido van Rossum
-
Mariatta Wijaya
-
Nathaniel Smith
-
Oleg Broytman