Requiring PEPs to add/remove modules in the stdlib (and dropping the concept of "provisional")
I had kicked off a discussion a while back at https://discuss.python.org/t/how-do-we-want-to-manage-additions-removals-to-... about how to manage the stdlib (this has nothing to with *what* the stdlib is and thus what belongs in it). It finally bubbled up in the SC agenda and after discussing things we have three proposals:
- Update PEP 2 to say a PEP is necessary to add a module to the stdlib
- Update PEP 4 to say that a PEP is necessary to deprecate/remove a module
- Mark PEP 411 as obsolete and thus dropping the idea of provisional modules
The PEP requirements are because the stdlib is important to manage appropriately and since it's a shared resource it should be something we all discuss. The PEP process seems like a good mechanism for both keeping what we bring in under control while also making sure people don't break people's code needlessly with removals.
The dropping of the concept of provisional modules is from simply having not seen any true benefit that could have been had in other ways (e.g. typing, asyncio, importlib.metadata). In pretty much all cases the APIs could have evolved publicly first and then been brought into the stdlib once stable (if at all), so the concept of "provisional" just doesn't seem worth keeping around.
We wanted to see if there was consensus around any of these ideas, hence this email. 🙂
PEP 2 will have to be “un-superseded”, and presumably the devguide (which PEP 2 points to) will also need to be updated.
I think it’s probably fine to drop the idea of provisional APIs. Aside from the limit benefit of evolution within the stdlib, code still gets written against provisional APIs and people still complain when they break in non-backward compatible ways, so we might as well avoid the whole thing.
-Barry
On Mar 22, 2022, at 16:26, Brett Cannon <brett@python.org> wrote:
I had kicked off a discussion a while back at https://discuss.python.org/t/how-do-we-want-to-manage-additions-removals-to-... about how to manage the stdlib (this has nothing to with what the stdlib is and thus what belongs in it). It finally bubbled up in the SC agenda and after discussing things we have three proposals: • Update PEP 2 to say a PEP is necessary to add a module to the stdlib • Update PEP 4 to say that a PEP is necessary to deprecate/remove a module • Mark PEP 411 as obsolete and thus dropping the idea of provisional modules The PEP requirements are because the stdlib is important to manage appropriately and since it's a shared resource it should be something we all discuss. The PEP process seems like a good mechanism for both keeping what we bring in under control while also making sure people don't break people's code needlessly with removals.
The dropping of the concept of provisional modules is from simply having not seen any true benefit that could have been had in other ways (e.g. typing, asyncio, importlib.metadata). In pretty much all cases the APIs could have evolved publicly first and then been brought into the stdlib once stable (if at all), so the concept of "provisional" just doesn't seem worth keeping around.
We wanted to see if there was consensus around any of these ideas, hence this email. 🙂
python-committers mailing list -- python-committers@python.org To unsubscribe send an email to python-committers-leave@python.org https://mail.python.org/mailman3/lists/python-committers.python.org/ Message archived at https://mail.python.org/archives/list/python-committers@python.org/message/5... Code of Conduct: https://www.python.org/psf/codeofconduct/
On Tue, Mar 22, 2022 at 4:36 PM Barry Warsaw <barry@python.org> wrote:
PEP 2 will have to be “un-superseded”, and presumably the devguide (which PEP 2 points to) will also need to be updated.
we discussed that a bit. the dev guide makes sense as a "how to do it" purpose document, useful for constructing PRs. The PEP makes sense as a "here's the policy before merging or spending too much time on it". they'd link to each other, but they have distinct roles.
I think it’s probably fine to drop the idea of provisional APIs. Aside from the limit benefit of evolution within the stdlib, code still gets written against provisional APIs and people still complain when they break in non-backward compatible ways, so we might as well avoid the whole thing.
-Barry
On Mar 22, 2022, at 16:26, Brett Cannon <brett@python.org> wrote:
I had kicked off a discussion a while back at https://discuss.python.org/t/how-do-we-want-to-manage-additions-removals-to-... about how to manage the stdlib (this has nothing to with what the stdlib is and thus what belongs in it). It finally bubbled up in the SC agenda and after discussing things we have three proposals: • Update PEP 2 to say a PEP is necessary to add a module to the stdlib • Update PEP 4 to say that a PEP is necessary to deprecate/remove a module • Mark PEP 411 as obsolete and thus dropping the idea of provisional modules The PEP requirements are because the stdlib is important to manage appropriately and since it's a shared resource it should be something we all discuss. The PEP process seems like a good mechanism for both keeping what we bring in under control while also making sure people don't break people's code needlessly with removals.
The dropping of the concept of provisional modules is from simply having not seen any true benefit that could have been had in other ways (e.g. typing, asyncio, importlib.metadata). In pretty much all cases the APIs could have evolved publicly first and then been brought into the stdlib once stable (if at all), so the concept of "provisional" just doesn't seem worth keeping around.
We wanted to see if there was consensus around any of these ideas, hence this email. 🙂
python-committers mailing list -- python-committers@python.org To unsubscribe send an email to python-committers-leave@python.org https://mail.python.org/mailman3/lists/python-committers.python.org/ Message archived at https://mail.python.org/archives/list/python-committers@python.org/message/5... Code of Conduct: https://www.python.org/psf/codeofconduct/
python-committers mailing list -- python-committers@python.org To unsubscribe send an email to python-committers-leave@python.org https://mail.python.org/mailman3/lists/python-committers.python.org/ Message archived at https://mail.python.org/archives/list/python-committers@python.org/message/S... Code of Conduct: https://www.python.org/psf/codeofconduct/
Sgtm
Sent from my comm
On Mar 22, 2022, at 21:31, Gregory P. Smith <greg@krypto.org> wrote:

On Tue, Mar 22, 2022 at 4:36 PM Barry Warsaw <barry@python.org<mailto:barry@python.org>> wrote: PEP 2 will have to be “un-superseded”, and presumably the devguide (which PEP 2 points to) will also need to be updated.
we discussed that a bit. the dev guide makes sense as a "how to do it" purpose document, useful for constructing PRs. The PEP makes sense as a "here's the policy before merging or spending too much time on it". they'd link to each other, but they have distinct roles.
I think it’s probably fine to drop the idea of provisional APIs. Aside from the limit benefit of evolution within the stdlib, code still gets written against provisional APIs and people still complain when they break in non-backward compatible ways, so we might as well avoid the whole thing.
-Barry
On Mar 22, 2022, at 16:26, Brett Cannon <brett@python.org<mailto:brett@python.org>> wrote:
I had kicked off a discussion a while back at https://discuss.python.org/t/how-do-we-want-to-manage-additions-removals-to-... about how to manage the stdlib (this has nothing to with what the stdlib is and thus what belongs in it). It finally bubbled up in the SC agenda and after discussing things we have three proposals: • Update PEP 2 to say a PEP is necessary to add a module to the stdlib • Update PEP 4 to say that a PEP is necessary to deprecate/remove a module • Mark PEP 411 as obsolete and thus dropping the idea of provisional modules The PEP requirements are because the stdlib is important to manage appropriately and since it's a shared resource it should be something we all discuss. The PEP process seems like a good mechanism for both keeping what we bring in under control while also making sure people don't break people's code needlessly with removals.
The dropping of the concept of provisional modules is from simply having not seen any true benefit that could have been had in other ways (e.g. typing, asyncio, importlib.metadata). In pretty much all cases the APIs could have evolved publicly first and then been brought into the stdlib once stable (if at all), so the concept of "provisional" just doesn't seem worth keeping around.
We wanted to see if there was consensus around any of these ideas, hence this email. 🙂
python-committers mailing list -- python-committers@python.org<mailto:python-committers@python.org> To unsubscribe send an email to python-committers-leave@python.org<mailto:python-committers-leave@python.org> https://mail.python.org/mailman3/lists/python-committers.python.org/ Message archived at https://mail.python.org/archives/list/python-committers@python.org/message/5... Code of Conduct: https://www.python.org/psf/codeofconduct/
python-committers mailing list -- python-committers@python.org<mailto:python-committers@python.org> To unsubscribe send an email to python-committers-leave@python.org<mailto:python-committers-leave@python.org> https://mail.python.org/mailman3/lists/python-committers.python.org/ Message archived at https://mail.python.org/archives/list/python-committers@python.org/message/S... Code of Conduct: https://www.python.org/psf/codeofconduct/
python-committers mailing list -- python-committers@python.org To unsubscribe send an email to python-committers-leave@python.org https://mail.python.org/mailman3/lists/python-committers.python.org/ Message archived at https://mail.python.org/archives/list/python-committers@python.org/message/G... Code of Conduct: https://www.python.org/psf/codeofconduct/
On Tue, Mar 22, 2022 at 04:26:57PM -0700, Brett Cannon wrote:
- Update PEP 2 to say a PEP is necessary to add a module to the stdlib
- Update PEP 4 to say that a PEP is necessary to deprecate/remove a module
Does that include modules flagged as private?
E.g. the public interface is weakref but there is also a _weakref module.
Maybe we want to keep tighter control over the top level stdlib modules (such as _weakref) but I hope that what happens inside a package is considered internal to the package, e.g. concurrent.futures._base.
If we are discussing these issues, how about refactoring a single file module to a package, with no change to the API? E.g.
# Before hovercraft.py
# refactor to hovercraft/__init__.py hovercraft/_privatestuff.py
-- Steve
On Tue, Mar 22, 2022 at 5:02 PM Steven D'Aprano <steve@pearwood.info> wrote:
On Tue, Mar 22, 2022 at 04:26:57PM -0700, Brett Cannon wrote:
- Update PEP 2 to say a PEP is necessary to add a module to the stdlib
- Update PEP 4 to say that a PEP is necessary to deprecate/remove a module
Does that include modules flagged as private?
E.g. the public interface is weakref but there is also a _weakref module.
I for one just had "public" in mind while we were discussing this
_underscore private things are implementation details so long as they do something unfortunate and leak out into things like pickle state. (and if anything does that, it is a bug that should be fixed, but takes deprecation time to drop old support for - i have no examples of this in mind, it is hopefully made up)
Maybe we want to keep tighter control over the top level stdlib modules
(such as _weakref) but I hope that what happens inside a package is considered internal to the package, e.g. concurrent.futures._base.
If we are discussing these issues, how about refactoring a single file module to a package, with no change to the API? E.g.
# Before hovercraft.py
# refactor to hovercraft/__init__.py hovercraft/_privatestuff.py
We've done that in the past without incident that I can recall. logging and unittest for example. It's still "import hovercraft" and is where you'd store your eels regardless of module vs package. I don't think that you need a PEP for that unless you can identify significant negative consequences.
-- Steve
python-committers mailing list -- python-committers@python.org To unsubscribe send an email to python-committers-leave@python.org https://mail.python.org/mailman3/lists/python-committers.python.org/ Message archived at https://mail.python.org/archives/list/python-committers@python.org/message/G... Code of Conduct: https://www.python.org/psf/codeofconduct/
On 3/22/22 16:26, Brett Cannon wrote:
[...] after discussing things we have three proposals:
- Update PEP 2 to say a PEP is necessary to add a module to the stdlib
- Update PEP 4 to say that a PEP is necessary to deprecate/remove a module
- Mark PEP 411 as obsolete and thus dropping the idea of provisional modules
+1 to all three
-- ~Ethan~
On Tue, 22 Mar 2022 at 23:27, Brett Cannon <brett@python.org> wrote:
Update PEP 2 to say a PEP is necessary to add a module to the stdlib Update PEP 4 to say that a PEP is necessary to deprecate/remove a module Mark PEP 411 as obsolete and thus dropping the idea of provisional modules
These all seem reasonable to me.
There's an implication for the Packaging community in that we have used "Provisional" status for PEPs (specifications). But personally, I'm fine with dropping that option - I'm on record saying that I don't like provisional status for packaging standards and won't use it in future.
Assuming this proposal gets approved, I will link to the PEP 411 change on the relevant packaging thread, and get the packaging process docs updated to match.
Paul
On Wed, Mar 23, 2022 at 2:23 AM Paul Moore <p.f.moore@gmail.com> wrote:
On Tue, 22 Mar 2022 at 23:27, Brett Cannon <brett@python.org> wrote:
Update PEP 2 to say a PEP is necessary to add a module to the stdlib Update PEP 4 to say that a PEP is necessary to deprecate/remove a module Mark PEP 411 as obsolete and thus dropping the idea of provisional modules
These all seem reasonable to me.
There's an implication for the Packaging community in that we have used "Provisional" status for PEPs (specifications). But personally, I'm fine with dropping that option - I'm on record saying that I don't like provisional status for packaging standards and won't use it in future.
This doesn't affect provisional *PEPs*, only *modules*; provisional PEP acceptances are covered under a different PEP (probably PEP 1).
-Brett
Assuming this proposal gets approved, I will link to the PEP 411 change on the relevant packaging thread, and get the packaging process docs updated to match.
Paul
participants (7)
-
Barry Warsaw -
Brett Cannon -
Ethan Furman -
Gregory P. Smith -
Jason R. Coombs -
Paul Moore -
Steven D'Aprano