On 06/28/2016 02:05 PM, Yury Selivanov wrote:

Long story short, I've discovered that asyncio is broken in 3.5.2. Specifically, there is a callbacks race in `loop.sock_connect` which can make subsequent `loop.sock_sendall` calls to hang forever. This thing is very tricky and hard to detect and debug; I had to spend a few hours investigating what's going on with a failing unittest in uvloop (asyncio-compatible event loop). I can only imagine how hard it would be to understand what's going on in a larger codebase.

For those who is interested, here's a PR for asyncio repo: https://github.com/python/asyncio/pull/366 It explains the bug in detail and there has a proposed patch to fix the problem.

Larry and the release team: would it be possible to make an "emergency" 3.5.3 release?

I've looped in the rest of the 3.5 release team. By the way, I don't know why you Cc'd Nick and Brett. While they're fine fellows, they aren't on the release team, and they aren't involved in these sorts of decisions. //arry/

On Jun 29, 2016, at 12:42 AM, Larry Hastings wrote:

By the way, I don't know why you Cc'd Nick and Brett. While they're fine fellows, they aren't on the release team, and they aren't involved in these sorts of decisions.

We're all involved in those sort of decisions. Raymond

On 06/28/2016 02:51 PM, Larry Hastings wrote:

On 06/28/2016 02:05 PM, Yury Selivanov wrote:

...

Larry and the release team: would it be possible to make an "emergency" 3.5.3 release?

I'd like to hear from the other asyncio reviewers: is this bug bad enough to merit such an "emergency" release?

Thanks,

//arry/

There has been a distinct lack of "dear god yes Larry" emails so far. This absence suggests that, no, it is not a bad enough bug to merit such a release. If we stay to our usual schedule, I expect 3.5.3 to ship December-ish. //arry/

On 2 July 2016 at 16:17, Ludovic Gasc wrote:

Hi everybody,

I fully understand that AsyncIO is a drop in the ocean of CPython, you're working to prepare the entire 3.5.3 release for December, not yet ready. However, you might create a 3.5.2.1 release with only this AsyncIO fix ?

That would be more work than just doing a 3.5.3 release, though - the problem isn't with the version number bump, it's with asking the release team to do additional work without clearly explaining the rationale for the request (more on that below). While some parts of the release process are automated, there's still a lot of steps to run through by a number of different people: https://www.python.org/dev/peps/pep-0101/. The first key question to answer in this kind of situation is: "Is there code that will run correctly on 3.5.1 that will now fail on 3.5.2?" (i.e. it's a regression introduced by the asyncio and coroutine changes in the point release rather than something that was already broken in 3.5.0 and 3.5.1). If the answer is "No", then it doesn't inhibit the 3.5.2 rollout in any way, and folks can wait until 3.5.3 for the fix. However, if the answer is "Yes, it's a new regression in 3.5.2" (as in this case), then the next question becomes "Is there an agreed resolution for the regression?" The answer to that is currently "No" - Yury's PR against the asyncio repo is still being discussed. Once the answer to that question is "Yes", *then* the question of releasing a high priority fix in a Python 3.5.3 release can be properly considered by answering the question "Of the folks using asyncio, what proportion of them are likely to encounter problems in upgrading to Python 3.5.2, and is there a workaround they can apply or alternate approach they can use to avoid the problem?". At the moment, Yury's explanation of the fix in the PR is (understandably) addressed at getting the problem resolved within the context of asyncio, and hence just describes the particular APIs affected, and the details of the incorrect behaviour. While that's an important step in the process, it doesn't provide a clear assessment of the *consequences* of the bug aimed at folks that aren't themselves deeply immersed in using asyncio, so we can't tell if the problem is "Some idiomatic code frequently recommended in user facing examples and used in third party asyncio based libraries may hang client processes" (which would weigh in favour of an early 3.5.3 release before people start encountering the regression in practice) or "Some low level API's not recommended for general use may hang if used in a particular non-idiomatic combination only likely to be encountered by event loop implementors" (which would suggest it may be OK to stick with the normal maintenance release cadence).

If 3.5.2.1 or 3.5.3 are impossible to release before december,

Early maintenance releases are definitely possible, but the consequences of particular regressions need to be put into terms that make sense to the release team, which generally means stepping up from "APIs X, Y, and Z broke in this way" to "Users doing A, B, and C will be affected in this way". As an example of a case where an early maintenance release took place: several years ago, Python 2.6.3 happened to break both "from logging import *" (due to a missing entry in test___all__ letting an error in logging.__all__ through) and building extension modules with setuptools (due to a change in a private API that setuptools was monkeypatching). Those were considered significant enough for the 2.6.4 release to happen early.

what are the alternative solutions for AsyncIO users ? 1. Use 3.5.1 and hope that Linux distributions won't use 3.5.2 ?

Linux distributions have mechanisms to carry patches (indeed, selective application of patches is one of the main benefits of using system packages over upstream ones), so any distro that rebases on 3.5.2 can be encouraged to add the fix once it lands regardless of whether or not Larry approves an additional maintenance release outside the normal cadence. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Another thought recently occurred to me. Do releases really have to be such big productions? A recent ACM article by Tom Limoncelli[1] reminded me that we're doing releases the old-fashioned way -- infrequently, and with lots of manual labor. Maybe we could (eventually) try to strive for a lighter-weight, more automated release process? It would be less work, and it would reduce stress for authors of stdlib modules and packages -- there's always the next release. I would think this wouldn't obviate the need for carefully planned and timed "big deal" feature releases, but it could make the bug fix releases *less* of a deal, for everyone. [1] http://cacm.acm.org/magazines/2016/7/204027-the-small-batches-principle/abst... (sadly requires login) -- --Guido van Rossum (python.org/~guido)

[forking the conversation since the subject has shifted] On Sun, 3 Jul 2016 at 09:50 Steve Dower wrote:

Many of our users prefer stability (the sort who plan operating system updates years in advance), but generally I'm in favour of more frequent releases.

So there's our 18 month cadence for feature/minor releases, and then there's the 6 month cadence for bug-fix/micro releases. At the language summit there was the discussion kicked off by Ned about our release schedule and a group of us had a discussion afterward where a more strict release cadence of 12 months with the release date tied to a consistent month -- e.g. September of every year -- instead of our hand-wavy "about 18 months after the last feature release"; people in the discussion seemed to like the 12 months consistency idea. I think making releases on a regular, annual schedule requires simply a decision by us to do it since the time scale we are talking about is still so large it shouldn't impact the workload of RMs & friends *that* much (I think). As for upping the bug-fix release cadence, if we can automate that then perhaps we can up the frequency (maybe once every quarter), but I'm not sure what kind of overhead that would add and thus how much would need to be automated to make that release cadence work. Doing this kind of shrunken cadence for bug-fix releases would require the RM & friends to decide what would need to be automated to shrink the release schedule to make it viable (e.g. "if we automated steps N & M of the release process then I would be okay releasing every 3 months instead of 6"). For me, I say we shift to an annual feature release in a specific month every year, and switch to a quarterly bug-fix releases only if we can add zero extra work to RMs & friends.

It will likely require more complex branching though, presumably based on the LTS model everyone else uses.

Why is that? You can almost view our feature releases as LTS releases, at which point our current branching structure is no different.

One thing we've discussed before is separating core and stdlib releases. I'd be really interested to see a release where most of the stdlib is just preinstalled (and upgradeable) PyPI packages. We can pin versions/bundle wheels for stable releases and provide a fast track via pip to update individual packages.

Probably no better opportunity to make such a fundamental change as we move to a new VCS...

<deep breath /> Topic 1 ======= If we separate out the stdlib, we first need to answer why we are doing this? The arguments supporting this idea is (1) it might simplify more frequent releases of Python (but that's a guess), (2) it would make the stdlib less CPython-dependent (if purely by the fact of perception and ease of testing using CI against other interpreters when they have matching version support), and (3) it might make it easier for us to get more contributors who are comfortable helping with just the stdlib vs CPython itself (once again, this might simply be through perception). So if we really wanted to go this route of breaking out the stdlib, I think we have two options. One is to have the cpython repo represent the CPython interpreter and then have a separate stdlib repo. The other option is to still have cpython represent the interpreter but then each stdlib module have their own repository. Since the single repo for the stdlib is not that crazy, I'll talk about the crazier N repo idea (in all scenarios we would probably have a repo that pulled in cpython and the stdlib through either git submodules or subtrees and that would represent a CPython release repo). In this scenario, having each module/package have its own repo could get us a couple of things. One is that it might help simplify module maintenance by allowing each module to have its own issue tracker, set of contributors, etc. This also means it will make it obvious what modules are being neglected which will either draw attention and get help or honestly lead to a deprecation if no one is willing to help maintain it. Separate repos would also allow for easier backport releases (e.g. what asyncio and typing have been doing since they were created). If a module is maintained as if it was its own project then it makes it easier to make releases separated from the stdlib itself (although the usefulness is minimized as long as sys.path has site-packages as its last entry). Separate releases allows for faster releases of the stand-alone module, e.g. if only asyncio has a bug then asyncio can cut their own release and the rest of the stdlib doesn't need to care. Then when a new CPython release is done we can simply bundle up the stable release at the moment and essentially make our mythical sumo release be the stdlib release itself (and this would help stop modules like asyncio and typing from simply copying modules into the stdlib from their external repo if we just pulled in their repo using submodules or subtrees in a master repo). And yes, I realize this might lead to a ton of repos, but maybe that's an important side effect. We have so much code in our stdlib that it's hard to maintain and fixes can get dropped on the floor. If this causes us to re-prioritize what should be in the stdlib and trim it back to things we consider critical to have in all Python releases, then IMO that's as a huge win in maintainability and workload savings instead of carrying forward neglected code (or at least help people focus on modules they care about and let others know where help is truly needed). Topic 2 ======= Independent releases of the stdlib could be done, although if we break the stdlib up into individual repos then it shifts the conversation as individual modules could simply do their own releases independent of the big stdlib release. Personally I don't see a point of doing a stdlib release separate from CPython, but I could see doing a more frequent release of CPython where the only thing that changed is the stdlib itself (but I don't know if that would even alleviate the RM workload). For me, I'm more interested in thinking about breaking the stdlib modules into their own repos and making a CPython release more of a collection of python-dev-approved modules that are maintained under the python organization on GitHub and follow our compatibility guidelines and code quality along with the CPython interpreter. This would also make it much easier for custom distros, e.g. a cloud-targeted CPython release that ignored all GUI libraries. -Brett

Cheers, Steve

Top-posted from my Windows Phone ------------------------------ From: Guido van Rossum Sent: ‎7/‎3/‎2016 7:42 To: Python-Dev Cc: Nick Coghlan Subject: Re: [Python-Dev] Request for CPython 3.5.3 release

Another thought recently occurred to me. Do releases really have to be such big productions? A recent ACM article by Tom Limoncelli[1] reminded me that we're doing releases the old-fashioned way -- infrequently, and with lots of manual labor. Maybe we could (eventually) try to strive for a lighter-weight, more automated release process? It would be less work, and it would reduce stress for authors of stdlib modules and packages -- there's always the next release. I would think this wouldn't obviate the need for carefully planned and timed "big deal" feature releases, but it could make the bug fix releases *less* of a deal, for everyone.

[1] http://cacm.acm.org/magazines/2016/7/204027-the-small-batches-principle/abst... (sadly requires login)

-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org

As an observer and user— It may be worth asking the Rust team what the main pain points are in coordinating and managing their releases. Some context for those unfamiliar: Rust uses a Chrome- or Firefox-like release train approach, with stable and beta releases every six weeks. Each release cycle includes both the compiler and the standard library. They use feature flags on "nightly" (the master branch) and cut release branches for actually gets shipped in each release. This has the advantage of letting new features and functionality ship whenever they're ready, rather than waiting for Big Bang releases. Because of strong commitments to stability and backwards compatibility as part of that, it hasn't led to any substantial breakage along the way, either. There is also some early discussion of how they might add LTS releases into that mix. The Rust standard library is currently bundled into the same repository as the compiler. Although the stdlib is currently being modularized and somewhat decoupled from the compiler, I don't believe they intend to separate it from the compiler repository or release in that process (not least because there's no need to further speed up their release cadence!). None of that is meant to suggest Python adopt that specific cadence (though I have found it quite nice), but simply to observe that the Rust team might have useful info on upsides, downsides, and particular gotchas as Python considers changing its own release process. Regards, Chris Krycho

On Jul 3, 2016, at 16:22, Brett Cannon wrote:

[forking the conversation since the subject has shifted]

...

On Sun, 3 Jul 2016 at 09:50 Steve Dower wrote: Many of our users prefer stability (the sort who plan operating system updates years in advance), but generally I'm in favour of more frequent releases.

So there's our 18 month cadence for feature/minor releases, and then there's the 6 month cadence for bug-fix/micro releases. At the language summit there was the discussion kicked off by Ned about our release schedule and a group of us had a discussion afterward where a more strict release cadence of 12 months with the release date tied to a consistent month -- e.g. September of every year -- instead of our hand-wavy "about 18 months after the last feature release"; people in the discussion seemed to like the 12 months consistency idea. I think making releases on a regular, annual schedule requires simply a decision by us to do it since the time scale we are talking about is still so large it shouldn't impact the workload of RMs & friends that much (I think).

As for upping the bug-fix release cadence, if we can automate that then perhaps we can up the frequency (maybe once every quarter), but I'm not sure what kind of overhead that would add and thus how much would need to be automated to make that release cadence work. Doing this kind of shrunken cadence for bug-fix releases would require the RM & friends to decide what would need to be automated to shrink the release schedule to make it viable (e.g. "if we automated steps N & M of the release process then I would be okay releasing every 3 months instead of 6").

For me, I say we shift to an annual feature release in a specific month every year, and switch to a quarterly bug-fix releases only if we can add zero extra work to RMs & friends.

...

It will likely require more complex branching though, presumably based on the LTS model everyone else uses.

Why is that? You can almost view our feature releases as LTS releases, at which point our current branching structure is no different.

...

One thing we've discussed before is separating core and stdlib releases. I'd be really interested to see a release where most of the stdlib is just preinstalled (and upgradeable) PyPI packages. We can pin versions/bundle wheels for stable releases and provide a fast track via pip to update individual packages.

Probably no better opportunity to make such a fundamental change as we move to a new VCS...

<deep breath />

Topic 1 ======= If we separate out the stdlib, we first need to answer why we are doing this? The arguments supporting this idea is (1) it might simplify more frequent releases of Python (but that's a guess), (2) it would make the stdlib less CPython-dependent (if purely by the fact of perception and ease of testing using CI against other interpreters when they have matching version support), and (3) it might make it easier for us to get more contributors who are comfortable helping with just the stdlib vs CPython itself (once again, this might simply be through perception).

So if we really wanted to go this route of breaking out the stdlib, I think we have two options. One is to have the cpython repo represent the CPython interpreter and then have a separate stdlib repo. The other option is to still have cpython represent the interpreter but then each stdlib module have their own repository.

Since the single repo for the stdlib is not that crazy, I'll talk about the crazier N repo idea (in all scenarios we would probably have a repo that pulled in cpython and the stdlib through either git submodules or subtrees and that would represent a CPython release repo). In this scenario, having each module/package have its own repo could get us a couple of things. One is that it might help simplify module maintenance by allowing each module to have its own issue tracker, set of contributors, etc. This also means it will make it obvious what modules are being neglected which will either draw attention and get help or honestly lead to a deprecation if no one is willing to help maintain it.

Separate repos would also allow for easier backport releases (e.g. what asyncio and typing have been doing since they were created). If a module is maintained as if it was its own project then it makes it easier to make releases separated from the stdlib itself (although the usefulness is minimized as long as sys.path has site-packages as its last entry). Separate releases allows for faster releases of the stand-alone module, e.g. if only asyncio has a bug then asyncio can cut their own release and the rest of the stdlib doesn't need to care. Then when a new CPython release is done we can simply bundle up the stable release at the moment and essentially make our mythical sumo release be the stdlib release itself (and this would help stop modules like asyncio and typing from simply copying modules into the stdlib from their external repo if we just pulled in their repo using submodules or subtrees in a master repo).

And yes, I realize this might lead to a ton of repos, but maybe that's an important side effect. We have so much code in our stdlib that it's hard to maintain and fixes can get dropped on the floor. If this causes us to re-prioritize what should be in the stdlib and trim it back to things we consider critical to have in all Python releases, then IMO that's as a huge win in maintainability and workload savings instead of carrying forward neglected code (or at least help people focus on modules they care about and let others know where help is truly needed).

Topic 2 ======= Independent releases of the stdlib could be done, although if we break the stdlib up into individual repos then it shifts the conversation as individual modules could simply do their own releases independent of the big stdlib release. Personally I don't see a point of doing a stdlib release separate from CPython, but I could see doing a more frequent release of CPython where the only thing that changed is the stdlib itself (but I don't know if that would even alleviate the RM workload).

For me, I'm more interested in thinking about breaking the stdlib modules into their own repos and making a CPython release more of a collection of python-dev-approved modules that are maintained under the python organization on GitHub and follow our compatibility guidelines and code quality along with the CPython interpreter. This would also make it much easier for custom distros, e.g. a cloud-targeted CPython release that ignored all GUI libraries.

-Brett

...

Cheers, Steve

Top-posted from my Windows Phone From: Guido van Rossum Sent: ‎7/‎3/‎2016 7:42 To: Python-Dev Cc: Nick Coghlan Subject: Re: [Python-Dev] Request for CPython 3.5.3 release

Another thought recently occurred to me. Do releases really have to be such big productions? A recent ACM article by Tom Limoncelli[1] reminded me that we're doing releases the old-fashioned way -- infrequently, and with lots of manual labor. Maybe we could (eventually) try to strive for a lighter-weight, more automated release process? It would be less work, and it would reduce stress for authors of stdlib modules and packages -- there's always the next release. I would think this wouldn't obviate the need for carefully planned and timed "big deal" feature releases, but it could make the bug fix releases *less* of a deal, for everyone.

[1] http://cacm.acm.org/magazines/2016/7/204027-the-small-batches-principle/abst... (sadly requires login)

-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris%40chriskrycho.com

On 3 July 2016 at 22:04, Brett Cannon wrote:

This last bit is what I would advocate if we broke the stdlib out unless an emergency patch release is warranted for a specific module (e.g. like asyncio that started this discussion). Obviously backporting is its own thing.

It's also worth noting that pip has no mechanism for installing an updated stdlib module, as everything goes into site-packages, and the stdlib takes precedence over site-packages unless you get into sys.path hacking abominations like setuptools uses (or at least used to use, I don't know if it still does). So as things stand, independent patch releases of stdlib modules would need to be manually copied into place. Allowing users to override the stdlib opens up a different can of worms - not necessarily one that we couldn't resolve, but IIRC, it was always a deliberate policy that overriding the stdlib wasn't possible (that's why backports have names like unittest2...) Paul

I actually thought about Rust when thinking about 3 month releases (I know they release faster though). What i would want to know is whether the RMs for Rust are employed by Mozilla and thus have work time to do but it vs Python RMs & friends who vary ob whether they get work time. On Sun, Jul 3, 2016, 13:54 Chris Krycho wrote:

As an observer and user—

It may be worth asking the Rust team what the main pain points are in coordinating and managing their releases.

Some context for those unfamiliar: Rust uses a Chrome- or Firefox-like release train approach, with stable and beta releases every six weeks. Each release cycle includes both the compiler and the standard library. They use feature flags on "nightly" (the master branch) and cut release branches for actually gets shipped in each release. This has the advantage of letting new features and functionality ship whenever they're ready, rather than waiting for Big Bang releases. Because of strong commitments to stability and backwards compatibility as part of that, it hasn't led to any substantial breakage along the way, either.

There is also some early discussion of how they might add LTS releases into that mix.

The Rust standard library is currently bundled into the same repository as the compiler. Although the stdlib is currently being modularized and somewhat decoupled from the compiler, I don't believe they intend to separate it from the compiler repository or release in that process (not least because there's no need to further speed up their release cadence!).

None of that is meant to suggest Python adopt that specific cadence (though I have found it *quite* nice), but simply to observe that the Rust team might have useful info on upsides, downsides, and particular gotchas as Python considers changing its own release process.

Regards, Chris Krycho

On Jul 3, 2016, at 16:22, Brett Cannon wrote:

[forking the conversation since the subject has shifted]

On Sun, 3 Jul 2016 at 09:50 Steve Dower wrote:

...

Many of our users prefer stability (the sort who plan operating system updates years in advance), but generally I'm in favour of more frequent releases.

So there's our 18 month cadence for feature/minor releases, and then there's the 6 month cadence for bug-fix/micro releases. At the language summit there was the discussion kicked off by Ned about our release schedule and a group of us had a discussion afterward where a more strict release cadence of 12 months with the release date tied to a consistent month -- e.g. September of every year -- instead of our hand-wavy "about 18 months after the last feature release"; people in the discussion seemed to like the 12 months consistency idea. I think making releases on a regular, annual schedule requires simply a decision by us to do it since the time scale we are talking about is still so large it shouldn't impact the workload of RMs & friends *that* much (I think).

As for upping the bug-fix release cadence, if we can automate that then perhaps we can up the frequency (maybe once every quarter), but I'm not sure what kind of overhead that would add and thus how much would need to be automated to make that release cadence work. Doing this kind of shrunken cadence for bug-fix releases would require the RM & friends to decide what would need to be automated to shrink the release schedule to make it viable (e.g. "if we automated steps N & M of the release process then I would be okay releasing every 3 months instead of 6").

For me, I say we shift to an annual feature release in a specific month every year, and switch to a quarterly bug-fix releases only if we can add zero extra work to RMs & friends.

...

It will likely require more complex branching though, presumably based on the LTS model everyone else uses.

Why is that? You can almost view our feature releases as LTS releases, at which point our current branching structure is no different.

...

One thing we've discussed before is separating core and stdlib releases. I'd be really interested to see a release where most of the stdlib is just preinstalled (and upgradeable) PyPI packages. We can pin versions/bundle wheels for stable releases and provide a fast track via pip to update individual packages.

Probably no better opportunity to make such a fundamental change as we move to a new VCS...

<deep breath />

Topic 1 ======= If we separate out the stdlib, we first need to answer why we are doing this? The arguments supporting this idea is (1) it might simplify more frequent releases of Python (but that's a guess), (2) it would make the stdlib less CPython-dependent (if purely by the fact of perception and ease of testing using CI against other interpreters when they have matching version support), and (3) it might make it easier for us to get more contributors who are comfortable helping with just the stdlib vs CPython itself (once again, this might simply be through perception).

So if we really wanted to go this route of breaking out the stdlib, I think we have two options. One is to have the cpython repo represent the CPython interpreter and then have a separate stdlib repo. The other option is to still have cpython represent the interpreter but then each stdlib module have their own repository.

Since the single repo for the stdlib is not that crazy, I'll talk about the crazier N repo idea (in all scenarios we would probably have a repo that pulled in cpython and the stdlib through either git submodules or subtrees and that would represent a CPython release repo). In this scenario, having each module/package have its own repo could get us a couple of things. One is that it might help simplify module maintenance by allowing each module to have its own issue tracker, set of contributors, etc. This also means it will make it obvious what modules are being neglected which will either draw attention and get help or honestly lead to a deprecation if no one is willing to help maintain it.

Separate repos would also allow for easier backport releases (e.g. what asyncio and typing have been doing since they were created). If a module is maintained as if it was its own project then it makes it easier to make releases separated from the stdlib itself (although the usefulness is minimized as long as sys.path has site-packages as its last entry). Separate releases allows for faster releases of the stand-alone module, e.g. if only asyncio has a bug then asyncio can cut their own release and the rest of the stdlib doesn't need to care. Then when a new CPython release is done we can simply bundle up the stable release at the moment and essentially make our mythical sumo release be the stdlib release itself (and this would help stop modules like asyncio and typing from simply copying modules into the stdlib from their external repo if we just pulled in their repo using submodules or subtrees in a master repo).

And yes, I realize this might lead to a ton of repos, but maybe that's an important side effect. We have so much code in our stdlib that it's hard to maintain and fixes can get dropped on the floor. If this causes us to re-prioritize what should be in the stdlib and trim it back to things we consider critical to have in all Python releases, then IMO that's as a huge win in maintainability and workload savings instead of carrying forward neglected code (or at least help people focus on modules they care about and let others know where help is truly needed).

Topic 2 ======= Independent releases of the stdlib could be done, although if we break the stdlib up into individual repos then it shifts the conversation as individual modules could simply do their own releases independent of the big stdlib release. Personally I don't see a point of doing a stdlib release separate from CPython, but I could see doing a more frequent release of CPython where the only thing that changed is the stdlib itself (but I don't know if that would even alleviate the RM workload).

For me, I'm more interested in thinking about breaking the stdlib modules into their own repos and making a CPython release more of a collection of python-dev-approved modules that are maintained under the python organization on GitHub and follow our compatibility guidelines and code quality along with the CPython interpreter. This would also make it much easier for custom distros, e.g. a cloud-targeted CPython release that ignored all GUI libraries.

-Brett

...

Cheers, Steve

Top-posted from my Windows Phone ------------------------------ From: Guido van Rossum Sent: ‎7/‎3/‎2016 7:42 To: Python-Dev Cc: Nick Coghlan Subject: Re: [Python-Dev] Request for CPython 3.5.3 release

Another thought recently occurred to me. Do releases really have to be such big productions? A recent ACM article by Tom Limoncelli[1] reminded me that we're doing releases the old-fashioned way -- infrequently, and with lots of manual labor. Maybe we could (eventually) try to strive for a lighter-weight, more automated release process? It would be less work, and it would reduce stress for authors of stdlib modules and packages -- there's always the next release. I would think this wouldn't obviate the need for carefully planned and timed "big deal" feature releases, but it could make the bug fix releases *less* of a deal, for everyone.

[1] http://cacm.acm.org/magazines/2016/7/204027-the-small-batches-principle/abst... (sadly requires login)

-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris%40chriskrycho.com

My thinking on this issue was that some/most packages from the stdlib would move into site-packages. Certainly I'd expect asyncio to be in this category, and probably typing. Even going as far as email and urllib would potentially be beneficial (to those packages, is my thinking). Obviously not every single module can do this, but there are plenty that aren't low-level dependencies for other modules that could. Depending on particular versions of these then becomes a case of adding normal package version constraints - we could even bundle version information for non-updateable packages so that installs fail on incompatible Python versions. The "Uber repository" could be a requirements.txt that pulls down wheels for the selected stable versions of each package so that we still distribute all the same code with the same stability, but users have much more ability to patch their own stdlib after install. (FWIW, we use a system similar to this at Microsoft for building Visual Studio, so I can vouch that it works on much more complicated software than Python.) Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Paul Moore" Sent: ‎7/‎3/‎2016 14:23 To: "Brett Cannon" Cc: "Guido van Rossum" ; "Nick Coghlan" ; "Python-Dev" ; "Steve Dower" Subject: Re: [Python-Dev] release cadence (was: Request for CPython 3.5.3 release) On 3 July 2016 at 22:04, Brett Cannon wrote:

This last bit is what I would advocate if we broke the stdlib out unless an emergency patch release is warranted for a specific module (e.g. like asyncio that started this discussion). Obviously backporting is its own thing.

It's also worth noting that pip has no mechanism for installing an updated stdlib module, as everything goes into site-packages, and the stdlib takes precedence over site-packages unless you get into sys.path hacking abominations like setuptools uses (or at least used to use, I don't know if it still does). So as things stand, independent patch releases of stdlib modules would need to be manually copied into place. Allowing users to override the stdlib opens up a different can of worms - not necessarily one that we couldn't resolve, but IIRC, it was always a deliberate policy that overriding the stdlib wasn't possible (that's why backports have names like unittest2...) Paul

On 03Jul2016 1556, Terry Reedy wrote:

On 7/3/2016 4:22 PM, Brett Cannon wrote:

...

So if we really wanted to go this route of breaking out the stdlib, I think we have two options. One is to have the cpython repo represent the CPython interpreter and then have a separate stdlib repo. The other option is to still have cpython represent the interpreter but then each stdlib module have their own repository.

Option 3 is something in between: groups of stdlib modules in their own repository. An obvious example: a gui group with _tkinter, tkinter, idlelib, turtle, turtledemo, and their doc files. Having 100s of repositories would not work well with with TortoiseHg.

A rough count of how I'd break up the current 3.5 Lib folder (which I happened to have handy) suggests no more than 50 repos. But there'd be no need to have all of them checked out just to build - only the ones you want to modify. And in that case, you'd probably have a stable Python to work against the separate package repo and wouldn't need to clone the core one. (I'm envisioning a build process that generates wheels from online sources and caches them. So updating the stdlib wheel cache would be part of the build process, but then the local wheels are used to install.) I personally would only have about 5 repos cloned on any of my dev machines (core, ctypes, distutils, possibly tkinter, ssl), as I rarely touch any other packages. (Having those separate from core is mostly for the versioning benefits - I doubt we could ever release Python without them, but it'd be great to be able to update distutils, ctypes or ssl in place with a simple pip/package mgr command.) Cheers, Steve

On 03.07.2016 16:39, Guido van Rossum wrote:

Another thought recently occurred to me. Do releases really have to be such big productions? A recent ACM article by Tom Limoncelli[1] reminded me that we're doing releases the old-fashioned way -- infrequently, and with lots of manual labor. Maybe we could (eventually) try to strive for a lighter-weight, more automated release process?

I can only recommend such an approach. We use it internally for years now and the workload for releasing, quality assurance and final deployment dropped significantly. We basically automated everything. The devs are pretty happy with it now and sometimes "mis-use" it for some of its side-products; but that's okay as it's very convenient to use. For some parts we use pip to install/upgrade the dependencies but CPython might need to use a different tooling for the stdlib and its C-dependencies. If you need some assistance here, let me know.

It would be less work, and it would reduce stress for authors of stdlib modules and packages -- there's always the next release. I would think this wouldn't obviate the need for carefully planned and timed "big deal" feature releases, but it could make the bug fix releases *less* of a deal, for everyone.

[1] http://cacm.acm.org/magazines/2016/7/204027-the-small-batches-principle/abst... (sadly requires login)

Best, Sven

On 03.07.2016 06:09, Nick Coghlan wrote:

On 2 July 2016 at 16:17, Ludovic Gasc wrote:

...

Hi everybody,

I fully understand that AsyncIO is a drop in the ocean of CPython, you're working to prepare the entire 3.5.3 release for December, not yet ready. However, you might create a 3.5.2.1 release with only this AsyncIO fix ?

That would be more work than just doing a 3.5.3 release, though - the problem isn't with the version number bump, it's with asking the release team to do additional work without clearly explaining the rationale for the request (more on that below). While some parts of the release process are automated, there's still a lot of steps to run through by a number of different people: https://www.python.org/dev/peps/pep-0101/.

The first key question to answer in this kind of situation is: "Is there code that will run correctly on 3.5.1 that will now fail on 3.5.2?" (i.e. it's a regression introduced by the asyncio and coroutine changes in the point release rather than something that was already broken in 3.5.0 and 3.5.1).

I don't know about 3.5.1 exactly, but at least things worked on the branch early April, which were broken by 3.5.2 final. I was trying to prepare for an update to 3.5.2 for Ubuntu 16.04 LTS, and found some regressions, documented at https://launchpad.net/bugs/1586673 (comment #6). It looks like that at least the packages nuitka, python-websockets and urwid fail to build with the 3.5.2 release. Still need to investigate. Unless I'm missing things, there is unfortunately no issue in the Python bug tracker, and there is no patch for the 3.5 branch either. My understanding is that it's not yet decided what to do about the issue. Matthias

I should quickly mention that future workflow-related stuff in regards to https://www.python.org/dev/peps/pep-0512 and the move to GitHub (e.g. CI), happens on the core-workflow mailing list. On Mon, 4 Jul 2016 at 15:35 Steve Dower wrote:

On 04Jul2016 0822, Kevin Ollivier wrote:

...

On 7/4/16, 3:32 AM, "Python-Dev on behalf of Sven R. Kunze" wrote:

...

If you need some assistance here, let me know.

I also offer my help with setting up CI and automated builds. :) I've actually done build automation for a number of the projects I've worked on in the past. In every case, doing so gave benefits that far outweighed the work needed to get it going.

It's actually not that much effort - we already have a fleet of buildbots that automatically build, test and report on Python's stability on a range of platforms. Once a build machine is configured, producing a build is typically a single command.

The benefit we get from the heavyweight release procedures is that someone trustworthy (the Release Manager) has controlled the process, reducing the rate of change and ensuring stability over the end of the process. Also that trustworthy people (the build managers) have downloaded, built and signed the code without modifying it or injecting unauthorised code.

As a result of these, people trust official releases to be correct and stable. It's very hard to put the same trust in an automated system (and it's a great way to lose signing certificates).

I don't believe the release procedures are too onerous (though Benjamin, Larry and Ned are welcome to disagree :) ), and possibly there is some more scripting that could help out, but there's really nothing in the direct process that we need to do more releases.

More frequent releases would mean more frequent feature freezes and more time in "cherry-picking" mode (where the RM has to approve and merge each individual fix), which affects all contributors. Shorter cycles make it harder to get changes reviewed, merged and tested. This is the limiting factor.

So don't worry about offering skills/effort for CI systems (unless you want to maintain a few buildbots, in which case read https://www.python.org/dev/buildbot/) - go and help review and improve some patches instead. The shorter the cycle between finding a need and committing the patch, and the more often issues are found *before* commit, the more frequently we can do releases.

Cheers, Steve

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org

On 07/04/2016 12:19 AM, Steve Dower wrote:

My thinking on this issue was that some/most packages from the stdlib would move into site-packages. Certainly I'd expect asyncio to be in this category, and probably typing. Even going as far as email and urllib would potentially be beneficial (to those packages, is my thinking).

Obviously not every single module can do this, but there are plenty that aren't low-level dependencies for other modules that could. Depending on particular versions of these then becomes a case of adding normal package version constraints - we could even bundle version information for non-updateable packages so that installs fail on incompatible Python versions.

The "Uber repository" could be a requirements.txt that pulls down wheels for the selected stable versions of each package so that we still distribute all the same code with the same stability, but users have much more ability to patch their own stdlib after install.

(FWIW, we use a system similar to this at Microsoft for building Visual Studio, so I can vouch that it works on much more complicated software than Python.)

While we're on the subject, I'd like to offer another point for consideration: not all implementations of Python can provide the full stdlib, and not everyone wants the full stdlib. For MicroPython, most of Python's batteries are too heavy. Tkinter on Android is probably not useful enough for people to port it. Weakref can't be emulated nicely in Javascript. If packages had a way to opt-out of needing the whole standard library, and instead specify the stdlib subset they need, answering questions like "will this run on my phone?" and "what piece of the stdlib do we want to port next?" would be easier. Both Debian and Fedora package some parts of the stdlib separately (tkinter, venv, tests), and have opt-in subsets of the stdlib for minimal systems (python-minimal, system-python). Tools like pyinstaller run magic heuristics to determine what parts of stdlib can be left out. It would help these projects if the "not all of stdlib is installed" case was handled more systematically at the CPython or distutils level. As I said on the Language summit, this is just talk; I don't currently have the resources to drive this effort. But if anyone is thinking of splitting the stdlib, please keep these points in mind as well. I think that, at least, if "pip install -U asyncio" becomes possible, "pip uninstall --yes-i-know-what-im-doing asyncio" should be possible as well.

From: Paul Moore mailto:p.f.moore@gmail.com Sent: ‎7/‎3/‎2016 14:23 To: Brett Cannon mailto:brett@python.org Cc: Guido van Rossum mailto:guido@python.org; Nick Coghlan mailto:ncoghlan@gmail.com; Python-Dev mailto:python-dev@python.org; Steve Dower mailto:steve.dower@python.org Subject: Re: [Python-Dev] release cadence (was: Request for CPython 3.5.3 release)

On 3 July 2016 at 22:04, Brett Cannon wrote:

...

This last bit is what I would advocate if we broke the stdlib out unless an emergency patch release is warranted for a specific module (e.g. like asyncio that started this discussion). Obviously backporting is its own thing.

It's also worth noting that pip has no mechanism for installing an updated stdlib module, as everything goes into site-packages, and the stdlib takes precedence over site-packages unless you get into sys.path hacking abominations like setuptools uses (or at least used to use, I don't know if it still does). So as things stand, independent patch releases of stdlib modules would need to be manually copied into place.

Allowing users to override the stdlib opens up a different can of worms - not necessarily one that we couldn't resolve, but IIRC, it was always a deliberate policy that overriding the stdlib wasn't possible (that's why backports have names like unittest2...)

On 07/05/2016 10:05 AM, Chris Angelico wrote:

On Tue, Jul 5, 2016 at 5:53 PM, Petr Viktorin wrote:

...

If packages had a way to opt-out of needing the whole standard library, and instead specify the stdlib subset they need, answering questions like "will this run on my phone?" and "what piece of the stdlib do we want to port next?" would be easier.

On the flip side, answering questions like "what version of Python do people need to run my program" become harder to answer, particularly if you have third-party dependencies. (The latest version of numpy might decide that it's going to 'import statistics', for instance.)

That question is already hard to answer. How do you tell if a library works on Micropython? Or Python for Android? I'm not arguing to change the default; if the next version of numpy doesn't do anything, nothing should change. However, under the status quo, "Python 3.4" means "CPython 3.4 with the full stdlib, otherwise all bets are off", and there's no good way to opt in to more granularity.

One of the arguments against splitting the stdlib was that corporate approval for software is often hard to obtain, and it's much easier to say "I need approval to use Python, exactly as distributed by python.org" than "I need approval to use Python-core plus these five Python-stdlib sections".

I'm not arguing against "Python, exactly as distributed by python.org" not including all of stdlib. I would like making stripped-down variants of CPython easier, and to make it possible to opt-in to use CPython without all of stdlib, so that major problems with stdlib availablility in other Python implementations can be caught early. Basically, instead of projects getting commits like "Add metadata for one flavor of Android packaging tool", I'd like to see "Add common metadata for Android, IPhone, PyInstaller, and minimal Linux, and make sure the CPython-based CI smoke-tests that metadata". Also, I believe corporate approval for python.org's Python is a bit of a red herring – usually you'd get approval for Python distributed by Continuum or Red Hat or Canonical or some such. As a Red Hat employee, I can say that what I'm suggesting won't make me suffer, and I see no reason it would hurt the others either.

On Jul 03, 2016, at 01:17 AM, Ludovic Gasc wrote:

If 3.5.2.1 or 3.5.3 are impossible to release before december, what are the alternative solutions for AsyncIO users ? 1. Use 3.5.1 and hope that Linux distributions won't use 3.5.2 ?

Matthias just uploaded a 3.5.2-2 to Debian unstable, which will also soon make its way to Ubuntu 16.10: https://launchpad.net/ubuntu/+source/python3.5/3.5.2-2 Ubuntu 16.04 LTS currently still has 3.5.1. Cheers, -Barry

On 5 July 2016 at 18:02, Steven D'Aprano wrote:

Yes, we're all probably sick and tired of hearing all the Chicken Little scare stories about how the GIL is killing Python, how everyone is abandoning Python for Ruby/Javascript/Go/Swift, how Python 3 is killing Python, etc. But sometimes the sky does fall. For many people, Python's single biggest advantage until now has been "batteries included", and I think that changing that is risky and shouldn't be done lightly.

+1 To be fair, I don't think anyone is looking at this "lightly", but I do think it's easy to underestimate the value of "batteries included", and the people it's *most* useful for are precisely the people who aren't involved in any of the Python mailing lists. They just want to get on with things, and "it came with the language" is a *huge* selling point. Internal changes in how we manage the stdlib modules are fine. But changing what the end user sees as "python" is a much bigger deal. Paul

On 05Jul2016 1021, Paul Moore wrote:

On 5 July 2016 at 18:02, Steven D'Aprano wrote:

...

Yes, we're all probably sick and tired of hearing all the Chicken Little scare stories about how the GIL is killing Python, how everyone is abandoning Python for Ruby/Javascript/Go/Swift, how Python 3 is killing Python, etc. But sometimes the sky does fall. For many people, Python's single biggest advantage until now has been "batteries included", and I think that changing that is risky and shouldn't be done lightly.

+1

To be fair, I don't think anyone is looking at this "lightly", but I do think it's easy to underestimate the value of "batteries included", and the people it's *most* useful for are precisely the people who aren't involved in any of the Python mailing lists. They just want to get on with things, and "it came with the language" is a *huge* selling point.

Internal changes in how we manage the stdlib modules are fine. But changing what the end user sees as "python" is a much bigger deal.

Also +1 on this - a default install of Python should continue to include everything it currently does. My interest in changing anything at all is to provide options for end-users/distributors to either reduce the footprint (which they already do), to more quickly update specific modules, and perhaps long-term to make user's code be less tied to a particular Python version (instead being tied to, for example, a specific asyncio version that can be brought into a range of supported Python versions). Batteries included is a big deal. Cheers, Steve

On 05Jul2016 1028, Barry Warsaw wrote:

On Jul 03, 2016, at 04:21 PM, Steve Dower wrote:

...

A rough count of how I'd break up the current 3.5 Lib folder (which I happened to have handy) suggests no more than 50 repos.

A concern with a highly split stdlib is local testing. I'm not worried about pull request testing, or after-the-fact buildbot testing since I'd have to assume that we'd make sure the fully integrated sumo package was tested in both environments.

But what about local testing? Let's say you change something in one module that causes a regression in a different module in a different repo. If you've only got a small subset checked out, you might never notice that before you PR'd your change. And then once the test fails, how easy will it be for you to recreate the tested environment locally so that you could debug your regression?

I'm sure it's doable, but let's not lose sight of that if this path is taken.

My hope is that it would be essentially a "pip freeze"/"pip install -r ..." (or equivalent with whatever tool is used/created for managing the stdlib). Perhaps using VCS URIs rather than version numbers? That is, the test run would dump a list of exactly which stdlib versions it's using, so that when you review the results it is possible to recreate it. But the point is well taken. I'm very hesitant about splitting out packages that are common dependencies of other parts of the stdlib, but there are plenty of leaf nodes in there too. Creating a complex dependency graph would be a disaster. Cheers, Steve

(Personally, I'm +0 on splitting out the stdlib and -1 on micro-splitting it.)

Cheers, -Barry

On Jul 05, 2016, at 10:38 AM, Steve Dower wrote:

My hope is that it would be essentially a "pip freeze"/"pip install -r ..." (or equivalent with whatever tool is used/created for managing the stdlib). Perhaps using VCS URIs rather than version numbers?

That is, the test run would dump a list of exactly which stdlib versions it's using, so that when you review the results it is possible to recreate it.

I think you'd have to have vcs checkouts though, because you will often need to fix or change something in one of those other library pieces. The other complication of course is that now you'll have two dependent PRs with reviews in two different repos.

But the point is well taken. I'm very hesitant about splitting out packages that are common dependencies of other parts of the stdlib, but there are plenty of leaf nodes in there too. Creating a complex dependency graph would be a disaster.

Yeah. <shudder> Cheers, -Barry

On 07/05/2016 10:44 AM, Barry Warsaw wrote:

On Jul 04, 2016, at 10:31 AM, Nick Coghlan wrote:

...

While we liked the "consistent calendar cadence that is some multiple of 6 months" idea, several of us thought 12 months was way too short as it makes for too many entries in third party support matrices.

18 months for a major release cadence still seems right to me. Downstreams and third-parties often have to go through *a lot* of work to ensure compatibility, and try as we might, every Python release breaks *something*. Major version releases trigger a huge cascade of other work for lots of other people, and I don't think shortening that would be for the overall community good. It just feels like we'd always be playing catch up.

+1 from me as well. Rapid major releases are just a huge headache. The nice thing about a .6 or .7 minor release is that we get closer to no bugs with each one. -- ~Ethan~

On 5 July 2016 at 19:01, Petr Viktorin wrote:

There are people who want to cut out what they don't need – to build self-contained applications (pyinstaller, Python for Android), or to eliminate unnecessary dependencies (python3-tkinter). And they will do it with CPython's blessing or without. [...] It would be much better for the ecosystem if CPython acknowledges this and sets some rules (like "here's how you can do it, but don't call the result an unqualified Python").

That doesn't sound unreasonable in principle. As a baseline, I guess the current policy is essentially: """ You can build your own Python installation with whatever parts of the stdlib omitted that you please. However, if you do this, you accept responsibility for any consequences, in terms of 3rd-party modules not working, or even stdlib breakage (for example, we don't guarantee that parts of the stdlib may not rely on other parts). """ That's pretty simple, both to state and to adhere to. And it's basically the current reality. What I'm not clear about is what *additional* guarantees people want to make, and how we'd make them. First of all, Python's packaging ecosystem has no way to express "this package won't work if pathlib isn't present in your stdlib". It seems to me that without something like that, it's pretty hard to support anything better than the current position with regard to 3rd party modules. Documenting stdlib inter-dependencies may be possible, but I wouldn't like to make "X doesn't depend on Y" a guarantee that's subject to backward compatibility rules. Maybe the suggestion is to provide better tools for people wanting to *build* such stripped down versions? That might be a possibility, I guess. I don't know much about how people build their own copies of Python to be able to comment. So I guess my question is, what is the actual proposal here? People seem to have concerns over things that aren't actually being proposed - but without knowing what *is* being proposed, it's hard to avoid that. Paul

On 05Jul2016 1404, Brett Cannon wrote:

Realizing that all of these are just proposals with no solid plan behind them, they are all predicated on moving to GitHub, and none of these are directly promoting releasing every module in the stdlib on PyPI as a stand-alone package with its own versioning, they are:

1. Break the stdlib out from CPython and have it be a stand-alone repo 2. Break the stdlib up into a bunch of independent repos that when viewed together make up the stdlib (Steve Dower did some back-of-envelope grouping and pegged the # of repos at ~50)

Actually, I was meaning to directly promote releasing each module on PyPI as a standalone package :) "Each module" here is at most the ~50 I counted (though likely many many fewer), which sounds intimidating until you realise that there are virtually no cross-dependencies between them and most would only depend on the core stdlib (which would *not* be a package on PyPI - you get most of Lib/*.py with the core install and it's fixed, while much of Lib/**/* is separately updateable). Take email as an example. It's external dependencies (found by grep for "import") are abc, base64, calendar, datetime, functools, imghdr, os, quopri, sndhdr, socket, time, urllib.parse, uu, warnings. IMHO, only urllib has the slightest chance of being non-fixed here (remembering that "non-fixed" means upgradeable from PyPI, not that it's missing). The circular references (email<->urllib) would probably need to be resolved, but I think many would see that as a good cleanliness step anyway. A quick glance suggests that both email and urllib are only using each other's public APIs, which means that any version of the other package is sufficient. An official release picks the two latest designated stable releases (this is where I'm imagining a requirements.txt-like file in the core repository) and bundles them both, and then users can update either package on its own. If email makes a change that requires a particular change to urllib, it adds a version constraint, which will force both users and the core requirements.txt file to update both together (this is probably why the circular references would need breaking...) Done with care and even incrementally (e.g. just the provisional modules at first), I don't think this is that scary a proposition. It's not strictly predicated on moving to github or to many repositories, but if we did decide to make a drastic change to the repository layout (which I think this requires at a minimum, at least for our own sanity), doing it while migrating VCS at least keeps all the disruption together. Cheers, Steve

On 6 July 2016 at 05:11, Brett Cannon wrote:

Sticking w/ 18 months is also fine, but then I would like to discuss choosing what months we try to release to get into a date-based release cadence so we know that every e.g. December and June are when releases typically happen thanks to our 6 month bug-fix release cadence. This has the nice benefit of all of us being generally aware of when a bug-fix release is coming up instead of having to check the PEP or go through our mail archive to find out what month a bug-fix is going to get cut (and also something the community to basically count on).

I don't have a strong preference on that front, as even the worst case outcome of a schedule misalignment for Fedora is what's happening for Fedora 25 & 26: F25 in November will still have Python 3.5, while Rawhide will get the 3.6 beta in September or so and then F26 will be released with 3.6 in the first half of next year. So really, I think the main criterion here is "Whatever works best for the folks directly involved in release management" However, if we did decide we wanted to take minimising "time to redistribution" for at least Ubuntu & Fedora into account, then the two main points to consider would be: - starting the upstream beta phase before the first downstream alpha freeze - publishing the upstream final release before the last downstream beta freeze Assuming 6 month distro cadences, and taking the F25 and 16.10 release cycles as representative, we get: - Ubuntu alpha 1 releases in late January & June - Fedora alpha freezes in early February & August - Ubuntu final beta freezes in late March & September - Fedora beta freezes in late March & September Further assuming we stuck with the current model of ~3 months from beta 1 -> final release, that would suggest a cadence alternating between: * December beta, February release * May beta, August release If we did that, then 3.6 -> 3.7 would be another "short" cycle (15 months from Dec 2016 to Feb 2018) before settling into a regular cadence of: * 2017-12: 3.7.0b1 * 2018-02: 3.7.0 * 2019-05: 3.8.0b1 * 2019-08: 3.8.0 * 2020-12: 3.9.0b1 * 2021-02: 3.9.0 * 2022-05: 3.10.0b1 * 2022-08: 3.10.0 * etc... The precise timing of maintenance releases isn't as big a deal (since they don't require anywhere near as much downstream coordination), but offsetting them by a month from the feature releases (so March & September in the Fedbuntu driven proposal above) would allow for the X.(Y+1).1 release to go out at the same time as the final X.Y.Z release. I'll reiterate though that we should be able to adjust to *any* consistent 18 month cycle downstream - the only difference will be the typical latency between new versions being released on python.org, and them showing up in Linux distros as the system Python installation. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 6 July 2016 at 03:47, Barry Warsaw wrote:

On Jul 05, 2016, at 10:38 AM, Steve Dower wrote:

...

My hope is that it would be essentially a "pip freeze"/"pip install -r ..." (or equivalent with whatever tool is used/created for managing the stdlib). Perhaps using VCS URIs rather than version numbers?

That is, the test run would dump a list of exactly which stdlib versions it's using, so that when you review the results it is possible to recreate it.

I think you'd have to have vcs checkouts though, because you will often need to fix or change something in one of those other library pieces. The other complication of course is that now you'll have two dependent PRs with reviews in two different repos.

I'd actually advocate for keeping a unified clone, and make any use of pip to manage pieces of the standard library purely an install-time thing (as it is for ensurepip). The main problem I see with actually making stdlib development dependent on having a venv already set up for pip to do its thing without affecting the rest of your system is that it would pose a major bootstrapping problem. It does mean we'd be introducing a greater divergence between the way devs work locally and the way the CI system worked (as in this model we'd definitely want the buildbots to be exercising both the "test in checkout" and "test an installed version" case), but working across multiple repos would be worse. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 6 July 2016 at 07:04, Brett Cannon wrote:

Realizing that all of these are just proposals with no solid plan behind them, they are all predicated on moving to GitHub, and none of these are directly promoting releasing every module in the stdlib on PyPI as a stand-alone package with its own versioning, they are:

1. Break the stdlib out from CPython and have it be a stand-alone repo 2. Break the stdlib up into a bunch of independent repos that when viewed together make up the stdlib (Steve Dower did some back-of-envelope grouping and pegged the # of repos at ~50)

3. Keep everything in the main CPython repo, but add a "Bundled" subdirectory of independently releasable multi-version compatible subprojects that we move some Lib/* components to. I think one of our goals here should be that "./configure && make && make altinstall" continues to get you a full Python installation for the relevant version. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, 5 Jul 2016 at 18:16 Nick Coghlan wrote:

On 6 July 2016 at 07:04, Brett Cannon wrote:

...

Realizing that all of these are just proposals with no solid plan behind them, they are all predicated on moving to GitHub, and none of these are directly promoting releasing every module in the stdlib on PyPI as a stand-alone package with its own versioning, they are:

1. Break the stdlib out from CPython and have it be a stand-alone repo 2. Break the stdlib up into a bunch of independent repos that when viewed together make up the stdlib (Steve Dower did some back-of-envelope grouping and pegged the # of repos at ~50)

3. Keep everything in the main CPython repo, but add a "Bundled" subdirectory of independently releasable multi-version compatible subprojects that we move some Lib/* components to.

That's basically what Steve is proposing.

I think one of our goals here should be that "./configure && make && make altinstall" continues to get you a full Python installation for the relevant version.

I don't think anyone is suggesting otherwise. You just might have to do `git clone --recursive` to get a full-fledged CPython checkout w/ stdlib.

On 07/06/2016 05:11 AM, Steven D'Aprano wrote:

On Tue, Jul 05, 2016 at 08:01:43PM +0200, Petr Viktorin wrote:

...

In the tkinter case, compiling for source is easy on a developer's computer, but doing that on a headless server brings in devel files for the entire graphical environment. Are you saying Python on servers should have a way to do turtle graphics, otherwise it's not Python?

That's a really good question.

I don't think we have an exact answer to "What counts as Python?". It's not like EMCAScript (Javascript) or C where there's a standard that defines the language and standard modules. We just have some de facto guidelines:

- CPython is definitely Python; - Jython is surely Python, even if it lacks the byte-code of CPython and some things behave slightly differently; - MicroPython is probably Python, because nobody expects to be able to run Tkinter GUI apps on an embedded device with 256K or RAM;

but it's hard to make that judgement except on a case-by-case basis.

I think though that even if there's no documented line, most people recognise that there are "core" and "non-core" standard modules. dis and tkinter are non-core: if µPython leaves out tkinter, nobody will be surprised; if Jython leaves out dis, nobody will hold it against them; but if they leave out math or htmllib that's another story.

For MicroPython, I would definitely expect htmllib to be an optional add-on – it's not useful for reading data off a thermometer saving it to an SD card. But I guess that's getting too deep into specifics.

So a headless server can probably leave out tkinter; but a desktop shouldn't.

Up till recently this wasn't possible to express in terms of RPM dependencies. Now, it's on the ever-growing TODO list... Another problem here is that you don't explicitly "install Python" on Fedora: when you install the system, you get a minimal set of packages to make everything work, and most of Python is part of that – but tkinter is not. This is in contrast to python.org releases, where you explicitly ask for (all of) Python. Technically it would now be possible to have to install Python to use it, but we run into another "batteries included" problem: Python (or, "most-of-Python") is a pretty good battery for an OS. Maybe a good short-term solution would be to make "import tkinter" raise ImportError("Run `dnf install tkinter` to install the tkinter module") if not found. This would prevent confusion while keeping the status quo. I'll look into that.

[...]

...

...

The other extreme is Javascript/Node.js, where the "just use pip" (or npm in this case) philosophy has been taken to such extremes that one developer practically brought down the entire Node.js ecosystem by withdrawing an eleven line module, left-pad, in a fit of pique.

Being open source, the damage was routed around quite quickly, but still, I think it's a good cautionary example of how a technological advance can transform a programming culture to the worse.

I don't understand the analogy. Should the eleven-line module have been in Node's stdlib? Outside of stdlib, people are doing this.

The point is that Javascript/Node.js is so lacking in batteries that the community culture has gravitated to an extreme version of "just use pip". I'm not suggesting that you, or anyone else, has proposed that Python do the same, only that there's a balance to be found between the extremes of "everything in the Python ecosystem should be part of the standard installation" and "next to nothing should be part of the standard installation".

The hard part is deciding where that balance should be :-)

I think the balance is where it needs to be for CPython, and it's also where it needs to be for Fedora. The real hard part is acknowledging that it needs to be in different places for different use cases, and making sure work to support the different use cases is coordinated. So, I guess I'm starting to form a concrete proposal: 1) Document what should happen when a stdlib module is not available. This should be an ImportError informative error message, something along the lines of 'This build of Python does not include SQLite support.' or 'MicroPython does not support turtle' or 'Use `sudo your-package-manager` install tkinter` to install this module'. 2) Document leaf modules (or "clusters") that can be removed from the stdlib, and their dependencies. Make no guarantees about cross-version compatibility of this metadata. 3) Standardize a way to query which stdlib modules are present (without side effects, i.e. trying an import doesn't count) 4) Adjust pip to ignore installed stdlib modules that are present, so distributions can depend on "statistics" and not "statistics if python_ver<3.4". (statistics is just an example, obviously this would only work for modules added after the PEP). For missing stdlib modules, pip should fail with the message from 1). (Unless the "pip upgrade asynciio" proposal goes through, in which case install the module if it's upgradable). 5) Reserve all stdlib module names on PyPI for backports or non-installable placeholders. 6) To ease smoke-testing behavior on Pythons without all of the stdlib, allow pip to remove leaf stdlib modules from a venv. Looks like it's time for a PEP.

On 6 July 2016 at 10:01, Petr Viktorin wrote:

4) Adjust pip to ignore installed stdlib modules that are present, so distributions can depend on "statistics" and not "statistics if python_ver<3.4". (statistics is just an example, obviously this would only work for modules added after the PEP). For missing stdlib modules, pip should fail with the message from 1). (Unless the "pip upgrade asynciio" proposal goes through, in which case install the module if it's upgradable).

A couple of comments here. 1. Projects may still need to depend on "statistics from Python 3.6 or later, but the one in 3.5 isn't good enough". Consider for example unittest, where projects often need the backport unittest2 to get access to features that aren't in older versions. 2. This is easy enough to do if we make stdlib modules publish version metadata. But it does raise the question of what the version of a stdlib module is - probably Python version plus a micro version for interim updates. Also, I have a recollection of pip having problems with some stdlib modules that publish version data right now (wsgiref?) - that should be checked to make sure this approach would work.

Looks like it's time for a PEP.

Probably - in principle, something like this proposal could be workable, it'll be a matter of thrashing out the details (which is something the PEP process is good at). Paul

I consider making stdlib modules "optional" like this to be completely separate from making them individually versioned - can't quite tell whether you guys do as well? To everyone: please don't conflate these two discussions. The other is about CPython workflow and this one is about community/user expectations (I have not been proposing to remove stdlib modules at any point). Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Petr Viktorin" Sent: ‎7/‎6/‎2016 2:04 To: "Steven D'Aprano" ; "Python-Dev" Subject: Re: [Python-Dev] Breaking up the stdlib (Was: release cadence) On 07/06/2016 05:11 AM, Steven D'Aprano wrote:

On Tue, Jul 05, 2016 at 08:01:43PM +0200, Petr Viktorin wrote:

...

In the tkinter case, compiling for source is easy on a developer's computer, but doing that on a headless server brings in devel files for the entire graphical environment. Are you saying Python on servers should have a way to do turtle graphics, otherwise it's not Python?

That's a really good question.

I don't think we have an exact answer to "What counts as Python?". It's not like EMCAScript (Javascript) or C where there's a standard that defines the language and standard modules. We just have some de facto guidelines:

- CPython is definitely Python; - Jython is surely Python, even if it lacks the byte-code of CPython and some things behave slightly differently; - MicroPython is probably Python, because nobody expects to be able to run Tkinter GUI apps on an embedded device with 256K or RAM;

but it's hard to make that judgement except on a case-by-case basis.

I think though that even if there's no documented line, most people recognise that there are "core" and "non-core" standard modules. dis and tkinter are non-core: if µPython leaves out tkinter, nobody will be surprised; if Jython leaves out dis, nobody will hold it against them; but if they leave out math or htmllib that's another story.

For MicroPython, I would definitely expect htmllib to be an optional add-on – it's not useful for reading data off a thermometer saving it to an SD card. But I guess that's getting too deep into specifics.

So a headless server can probably leave out tkinter; but a desktop shouldn't.

Up till recently this wasn't possible to express in terms of RPM dependencies. Now, it's on the ever-growing TODO list... Another problem here is that you don't explicitly "install Python" on Fedora: when you install the system, you get a minimal set of packages to make everything work, and most of Python is part of that – but tkinter is not. This is in contrast to python.org releases, where you explicitly ask for (all of) Python. Technically it would now be possible to have to install Python to use it, but we run into another "batteries included" problem: Python (or, "most-of-Python") is a pretty good battery for an OS. Maybe a good short-term solution would be to make "import tkinter" raise ImportError("Run `dnf install tkinter` to install the tkinter module") if not found. This would prevent confusion while keeping the status quo. I'll look into that.

[...]

...

...

The other extreme is Javascript/Node.js, where the "just use pip" (or npm in this case) philosophy has been taken to such extremes that one developer practically brought down the entire Node.js ecosystem by withdrawing an eleven line module, left-pad, in a fit of pique.

Being open source, the damage was routed around quite quickly, but still, I think it's a good cautionary example of how a technological advance can transform a programming culture to the worse.

I don't understand the analogy. Should the eleven-line module have been in Node's stdlib? Outside of stdlib, people are doing this.

The point is that Javascript/Node.js is so lacking in batteries that the community culture has gravitated to an extreme version of "just use pip". I'm not suggesting that you, or anyone else, has proposed that Python do the same, only that there's a balance to be found between the extremes of "everything in the Python ecosystem should be part of the standard installation" and "next to nothing should be part of the standard installation".

The hard part is deciding where that balance should be :-)

I think the balance is where it needs to be for CPython, and it's also where it needs to be for Fedora. The real hard part is acknowledging that it needs to be in different places for different use cases, and making sure work to support the different use cases is coordinated. So, I guess I'm starting to form a concrete proposal: 1) Document what should happen when a stdlib module is not available. This should be an ImportError informative error message, something along the lines of 'This build of Python does not include SQLite support.' or 'MicroPython does not support turtle' or 'Use `sudo your-package-manager` install tkinter` to install this module'. 2) Document leaf modules (or "clusters") that can be removed from the stdlib, and their dependencies. Make no guarantees about cross-version compatibility of this metadata. 3) Standardize a way to query which stdlib modules are present (without side effects, i.e. trying an import doesn't count) 4) Adjust pip to ignore installed stdlib modules that are present, so distributions can depend on "statistics" and not "statistics if python_ver<3.4". (statistics is just an example, obviously this would only work for modules added after the PEP). For missing stdlib modules, pip should fail with the message from 1). (Unless the "pip upgrade asynciio" proposal goes through, in which case install the module if it's upgradable). 5) Reserve all stdlib module names on PyPI for backports or non-installable placeholders. 6) To ease smoke-testing behavior on Pythons without all of the stdlib, allow pip to remove leaf stdlib modules from a venv. Looks like it's time for a PEP. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org

On 7/6/2016 10:44 PM, Nick Coghlan wrote:

The point of overlap I see is that if the stdlib starts putting some selected modules into site-packages (so "pip install --upgrade <sublibrary>" works without any changes to pip or equivalent tools), then that also solves the "How to explicitly declare dependencies on particular pieces of the standard library" problem: you use the same mechanisms we already use to declare dependencies on 3rd party packages distributed via a packaging server.

One thing to keep in mind if we do this is how it interacts with the -S command line option to not include site-packages in sys.path. I currently use -S to basically mean "give my python as it was distributed, and don't include anything that was subsequently added by adding other RPM's (or package manager of your choice)". I realize that's a rough description, and possibly an abuse of -S. If using -S were to start excluding parts of the stdlib, that would be a problem for me. Eric.

Yes, not too long ago I installed "every" Python package on Ubuntu, and Python basically would not start. Perhaps some plugin system was trying to import everything and caused a segfault in GTK. The "short sys.path" model of everything installed is importable has its limits. On Thu, Jul 7, 2016 at 9:24 AM Barry Warsaw wrote:

On Jul 07, 2016, at 08:12 AM, Eric V. Smith wrote:

...

One thing to keep in mind if we do this is how it interacts with the -S command line option to not include site-packages in sys.path. I currently use -S to basically mean "give my python as it was distributed, and don't include anything that was subsequently added by adding other RPM's (or package manager of your choice)". I realize that's a rough description, and possibly an abuse of -S. If using -S were to start excluding parts of the stdlib, that would be a problem for me.

It's an important consideration, and leads to another discussion that's recurred over the years. Operating systems often want an "isolated" Python, similar to what's given by -I, which cannot be altered by subsequent installs. It's one of the things that lead to the Debian ecosystem using dist-packages for PyPI installed packages. Without isolation, it's just too easy for some random PyPI thing to break your system, and yes, that has really happened in the past.

So if we go down the path of moving more of the stdlib to site-packages, we also need to get serious about a system-specific isolated Python.

Cheers, -Barry _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com

On 05.07.2016 21:11, Brett Cannon wrote:

On Tue, 5 Jul 2016 at 10:45 Barry Warsaw wrote:

...

On Jul 04, 2016, at 10:31 AM, Nick Coghlan wrote:

...

While we liked the "consistent calendar cadence that is some multiple of 6 months" idea, several of us thought 12 months was way too short as it makes for too many entries in third party support matrices.

18 months for a major release cadence still seems right to me. Downstreams and third-parties often have to go through *a lot* of work to ensure compatibility, and try as we might, every Python release breaks *something*. Major version releases trigger a huge cascade of other work for lots of other people, and I don't think shortening that would be for the overall community good. It just feels like we'd always be playing catch up.

Sticking w/ 18 months is also fine, but then I would like to discuss choosing what months we try to release to get into a date-based release cadence so we know that every e.g. December and June are when releases typically happen thanks to our 6 month bug-fix release cadence. This has the nice benefit of all of us being generally aware of when a bug-fix release is coming up instead of having to check the PEP or go through our mail archive to find out what month a bug-fix is going to get cut (and also something the community to basically count on).

I like the 18 months cycle, because it's a multiple of six, which fits the Ubuntu release cadence (as as I understand, the Fedora cadence as well). Sometimes it might be ambitious to update reverse dependencies in the distro within two months until the distro freeze, and two more months during the freeze leading to a distro release, but such is life, and it's then up to distro maintainers of LTS releases to prepare the distro for a new version with only four months left. My hope with time based releases is that also upstreams will start testing with new versions more early when they can anticipate the release date. Matthias