Mailman 3 Requiring PEPs to add/remove modules in the stdlib (and dropping the concept of "provisional") - python-committers

newer
[RELEASE] Python 3.10.4 and 3.9.12...

Requiring PEPs to add/remove modules in the stdlib (and dropping the concept of "provisional")

older
[RELEASE] Python 3.10.3, 3.9.11,...

Brett Cannon

22 Mar 2022 22 Mar '22

11:26 p.m.

I had kicked off a discussion a while back at https://discuss.python.org/t/how-do-we-want-to-manage-additions-removals-to-... about how to manage the stdlib (this has nothing to with *what* the stdlib is and thus what belongs in it). It finally bubbled up in the SC agenda and after discussing things we have three proposals:

Update PEP 2 to say a PEP is necessary to add a module to the stdlib
Update PEP 4 to say that a PEP is necessary to deprecate/remove a module
Mark PEP 411 as obsolete and thus dropping the idea of provisional modules

The PEP requirements are because the stdlib is important to manage appropriately and since it's a shared resource it should be something we all discuss. The PEP process seems like a good mechanism for both keeping what we bring in under control while also making sure people don't break people's code needlessly with removals.

The dropping of the concept of provisional modules is from simply having not seen any true benefit that could have been had in other ways (e.g. typing, asyncio, importlib.metadata). In pretty much all cases the APIs could have evolved publicly first and then been brought into the stdlib once stable (if at all), so the concept of "provisional" just doesn't seem worth keeping around.

We wanted to see if there was consensus around any of these ideas, hence this email. 🙂

Attachments:

attachment.html (text/html — 1.6 KB)

Show replies by date

Barry Warsaw

22 Mar 22 Mar

11:34 p.m.

PEP 2 will have to be “un-superseded”, and presumably the devguide (which PEP 2 points to) will also need to be updated.

I think it’s probably fine to drop the idea of provisional APIs. Aside from the limit benefit of evolution within the stdlib, code still gets written against provisional APIs and people still complain when they break in non-backward compatible ways, so we might as well avoid the whole thing.

-Barry

...

On Mar 22, 2022, at 16:26, Brett Cannon brett@python.org wrote:

I had kicked off a discussion a while back at https://discuss.python.org/t/how-do-we-want-to-manage-additions-removals-to-... about how to manage the stdlib (this has nothing to with what the stdlib is and thus what belongs in it). It finally bubbled up in the SC agenda and after discussing things we have three proposals: • Update PEP 2 to say a PEP is necessary to add a module to the stdlib • Update PEP 4 to say that a PEP is necessary to deprecate/remove a module • Mark PEP 411 as obsolete and thus dropping the idea of provisional modules The PEP requirements are because the stdlib is important to manage appropriately and since it's a shared resource it should be something we all discuss. The PEP process seems like a good mechanism for both keeping what we bring in under control while also making sure people don't break people's code needlessly with removals.

The dropping of the concept of provisional modules is from simply having not seen any true benefit that could have been had in other ways (e.g. typing, asyncio, importlib.metadata). In pretty much all cases the APIs could have evolved publicly first and then been brought into the stdlib once stable (if at all), so the concept of "provisional" just doesn't seem worth keeping around.

We wanted to see if there was consensus around any of these ideas, hence this email. 🙂

python-committers mailing list -- python-committers@python.org To unsubscribe send an email to python-committers-leave@python.org https://mail.python.org/mailman3/lists/python-committers.python.org/ Message archived at https://mail.python.org/archives/list/python-committers@python.org/message/5... Code of Conduct: https://www.python.org/psf/codeofconduct/

Gregory P. Smith

23 Mar 23 Mar

1:30 a.m.

On Tue, Mar 22, 2022 at 4:36 PM Barry Warsaw barry@python.org wrote:

...

PEP 2 will have to be “un-superseded”, and presumably the devguide (which PEP 2 points to) will also need to be updated.

we discussed that a bit. the dev guide makes sense as a "how to do it" purpose document, useful for constructing PRs. The PEP makes sense as a "here's the policy before merging or spending too much time on it". they'd link to each other, but they have distinct roles.

...

I think it’s probably fine to drop the idea of provisional APIs. Aside from the limit benefit of evolution within the stdlib, code still gets written against provisional APIs and people still complain when they break in non-backward compatible ways, so we might as well avoid the whole thing.

-Barry

...
On Mar 22, 2022, at 16:26, Brett Cannon brett@python.org wrote:

I had kicked off a discussion a while back at https://discuss.python.org/t/how-do-we-want-to-manage-additions-removals-to-... about how to manage the stdlib (this has nothing to with what the stdlib is and thus what belongs in it). It finally bubbled up in the SC agenda and after discussing things we have three proposals: • Update PEP 2 to say a PEP is necessary to add a module to the stdlib • Update PEP 4 to say that a PEP is necessary to deprecate/remove a module • Mark PEP 411 as obsolete and thus dropping the idea of provisional modules The PEP requirements are because the stdlib is important to manage appropriately and since it's a shared resource it should be something we all discuss. The PEP process seems like a good mechanism for both keeping what we bring in under control while also making sure people don't break people's code needlessly with removals.

The dropping of the concept of provisional modules is from simply having not seen any true benefit that could have been had in other ways (e.g. typing, asyncio, importlib.metadata). In pretty much all cases the APIs could have evolved publicly first and then been brought into the stdlib once stable (if at all), so the concept of "provisional" just doesn't seem worth keeping around.

We wanted to see if there was consensus around any of these ideas, hence this email. 🙂

python-committers mailing list -- python-committers@python.org To unsubscribe send an email to python-committers-leave@python.org https://mail.python.org/mailman3/lists/python-committers.python.org/ Message archived at https://mail.python.org/archives/list/python-committers@python.org/message/5... Code of Conduct: https://www.python.org/psf/codeofconduct/

python-committers mailing list -- python-committers@python.org To unsubscribe send an email to python-committers-leave@python.org https://mail.python.org/mailman3/lists/python-committers.python.org/ Message archived at https://mail.python.org/archives/list/python-committers@python.org/message/S... Code of Conduct: https://www.python.org/psf/codeofconduct/

Jason R. Coombs

1:33 a.m.

Sgtm

Sent from my comm

On Mar 22, 2022, at 21:31, Gregory P. Smith greg@krypto.org wrote:

On Tue, Mar 22, 2022 at 4:36 PM Barry Warsaw mailto:barry@python.org> wrote: PEP 2 will have to be “un-superseded”, and presumably the devguide (which PEP 2 points to) will also need to be updated.

-Barry

...

On Mar 22, 2022, at 16:26, Brett Cannon mailto:brett@python.org> wrote:

I had kicked off a discussion a while back at https://discuss.python.org/t/how-do-we-want-to-manage-additions-removals-to-... about how to manage the stdlib (this has nothing to with what the stdlib is and thus what belongs in it). It finally bubbled up in the SC agenda and after discussing things we have three proposals: • Update PEP 2 to say a PEP is necessary to add a module to the stdlib • Update PEP 4 to say that a PEP is necessary to deprecate/remove a module • Mark PEP 411 as obsolete and thus dropping the idea of provisional modules The PEP requirements are because the stdlib is important to manage appropriately and since it's a shared resource it should be something we all discuss. The PEP process seems like a good mechanism for both keeping what we bring in under control while also making sure people don't break people's code needlessly with removals.

The dropping of the concept of provisional modules is from simply having not seen any true benefit that could have been had in other ways (e.g. typing, asyncio, importlib.metadata). In pretty much all cases the APIs could have evolved publicly first and then been brought into the stdlib once stable (if at all), so the concept of "provisional" just doesn't seem worth keeping around.

We wanted to see if there was consensus around any of these ideas, hence this email. 🙂

python-committers mailing list -- python-committers@python.orgmailto:python-committers@python.org To unsubscribe send an email to python-committers-leave@python.orgmailto:python-committers-leave@python.org https://mail.python.org/mailman3/lists/python-committers.python.org/ Message archived at https://mail.python.org/archives/list/python-committers@python.org/message/5... Code of Conduct: https://www.python.org/psf/codeofconduct/

python-committers mailing list -- python-committers@python.orgmailto:python-committers@python.org To unsubscribe send an email to python-committers-leave@python.orgmailto:python-committers-leave@python.org https://mail.python.org/mailman3/lists/python-committers.python.org/ Message archived at https://mail.python.org/archives/list/python-committers@python.org/message/S... Code of Conduct: https://www.python.org/psf/codeofconduct/

python-committers mailing list -- python-committers@python.org To unsubscribe send an email to python-committers-leave@python.org https://mail.python.org/mailman3/lists/python-committers.python.org/ Message archived at https://mail.python.org/archives/list/python-committers@python.org/message/G... Code of Conduct: https://www.python.org/psf/codeofconduct/

Steven D'Aprano

midnight

On Tue, Mar 22, 2022 at 04:26:57PM -0700, Brett Cannon wrote:

...

Update PEP 2 to say a PEP is necessary to add a module to the stdlib

Update PEP 4 to say that a PEP is necessary to deprecate/remove a module

Does that include modules flagged as private?

E.g. the public interface is weakref but there is also a _weakref module.

Maybe we want to keep tighter control over the top level stdlib modules (such as _weakref) but I hope that what happens inside a package is considered internal to the package, e.g. concurrent.futures._base.

If we are discussing these issues, how about refactoring a single file module to a package, with no change to the API? E.g.

# Before hovercraft.py

# refactor to hovercraft/__init__.py hovercraft/_privatestuff.py

-- Steve

Gregory P. Smith

1:28 a.m.

On Tue, Mar 22, 2022 at 5:02 PM Steven D'Aprano steve@pearwood.info wrote:

...

On Tue, Mar 22, 2022 at 04:26:57PM -0700, Brett Cannon wrote:

...

Update PEP 2 to say a PEP is necessary to add a module to the stdlib

Update PEP 4 to say that a PEP is necessary to deprecate/remove a module

Does that include modules flagged as private?

E.g. the public interface is weakref but there is also a _weakref module.

I for one just had "public" in mind while we were discussing this

_underscore private things are implementation details so long as they do something unfortunate and leak out into things like pickle state. (and if anything does that, it is a bug that should be fixed, but takes deprecation time to drop old support for - i have no examples of this in mind, it is hopefully made up)

Maybe we want to keep tighter control over the top level stdlib modules

...

(such as _weakref) but I hope that what happens inside a package is considered internal to the package, e.g. concurrent.futures._base.

If we are discussing these issues, how about refactoring a single file module to a package, with no change to the API? E.g.

# Before hovercraft.py

# refactor to hovercraft/__init__.py hovercraft/_privatestuff.py

We've done that in the past without incident that I can recall. logging and unittest for example. It's still "import hovercraft" and is where you'd store your eels regardless of module vs package. I don't think that you need a PEP for that unless you can identify significant negative consequences.

...

-- Steve

python-committers mailing list -- python-committers@python.org To unsubscribe send an email to python-committers-leave@python.org https://mail.python.org/mailman3/lists/python-committers.python.org/ Message archived at https://mail.python.org/archives/list/python-committers@python.org/message/G... Code of Conduct: https://www.python.org/psf/codeofconduct/

Ethan Furman

12:52 a.m.

On 3/22/22 16:26, Brett Cannon wrote:

...

[...] after discussing things we have three proposals:

Update PEP 2 to say a PEP is necessary to add a module to the stdlib

Update PEP 4 to say that a PEP is necessary to deprecate/remove a module

Mark PEP 411 as obsolete and thus dropping the idea of provisional modules

+1 to all three

-- ~Ethan~

Paul Moore

9:22 a.m.

On Tue, 22 Mar 2022 at 23:27, Brett Cannon brett@python.org wrote:

...

Update PEP 2 to say a PEP is necessary to add a module to the stdlib Update PEP 4 to say that a PEP is necessary to deprecate/remove a module Mark PEP 411 as obsolete and thus dropping the idea of provisional modules

These all seem reasonable to me.

There's an implication for the Packaging community in that we have used "Provisional" status for PEPs (specifications). But personally, I'm fine with dropping that option - I'm on record saying that I don't like provisional status for packaging standards and won't use it in future.

Assuming this proposal gets approved, I will link to the PEP 411 change on the relevant packaging thread, and get the packaging process docs updated to match.

Paul

Brett Cannon

7:53 p.m.

On Wed, Mar 23, 2022 at 2:23 AM Paul Moore p.f.moore@gmail.com wrote:

...

On Tue, 22 Mar 2022 at 23:27, Brett Cannon brett@python.org wrote:

...
Update PEP 2 to say a PEP is necessary to add a module to the stdlib Update PEP 4 to say that a PEP is necessary to deprecate/remove a module Mark PEP 411 as obsolete and thus dropping the idea of provisional modules

These all seem reasonable to me.

There's an implication for the Packaging community in that we have used "Provisional" status for PEPs (specifications). But personally, I'm fine with dropping that option - I'm on record saying that I don't like provisional status for packaging standards and won't use it in future.

This doesn't affect provisional *PEPs*, only *modules*; provisional PEP acceptances are covered under a different PEP (probably PEP 1).

-Brett

...

Assuming this proposal gets approved, I will link to the PEP 411 change on the relevant packaging thread, and get the packaging process docs updated to match.

Paul