working on a PEP for the __preview__ package
Hello, I plan to take the ideas discussed in the thread on the __experimental__/__preview__ package (1), and further elaborated by Nick Coghlan in (2), and turn it into a PEP. Further, I'm willing to invest the time resources to make the package release-worthy before Alpha 1 of 3.3, with at least one module in it. This work will be done under the mentorship of Nick Coghlan. Any ideas, suggestions or objections are welcome. Eli ---- (1) http://mail.python.org/pipermail/python-ideas/2011-August/011317.html (2) http://readthedocs.org/docs/ncoghlan_devs-python-notes/en/latest/pep_ideas/p...
On Tue, Jan 3, 2012 at 1:08 PM, Eli Bendersky <eliben@gmail.com> wrote:
I think one thing that needs to be made crystal clear (both in the PEP and in the documentation for the namespace) is that anyone attempting to combine the preview namespace with persistence mechanisms that rely on module names are setting themselves up for migration problems in the future. It isn't just the case that the __preview__ namespace APIs *might* change: they're actually *guaranteed* to change, when they move to their final location (probably, but not necessarily, in the next release). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Just a suggestion: Golang uses an "exp" namespace, this would translate to a exp package for grouping like modules in a similar way that xml, http and concurrent are currently used. http://golang.org/pkg/exp/ The name is unoffensive, and somewhat more palatable than __preview__, which really isn't a protocol, or a bag for special interpreter switches like __future__. Under this scheme, the currently proposed modules would be: exp.regex exp.daemon exp.ipaddr Rather than __preview__.regex __preview__.daemon __preview__.ipaddr Either way I really look forward to something for PEP 3153 being trialed. http://www.python.org/dev/peps/pep-3153/ Matt On Tue, Jan 3, 2012 at 3:32 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
-- ಠ_ಠ
On 1/3/12 2:57 PM, Matt Joiner wrote:
The nice thing about the dunders is that no one will make a module with a conflicting name. That's really the common factor behind all uses of dunders: reserving a name for Python's use. Whether something is a protocol or bag of switches doesn't really enter into it. Like "new" and "random", "exp" or anything else that means something like "experimental" is going to trigger some problems when people use that name for *their* experimental modules. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Wed, Jan 4, 2012 at 3:29 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Dunder names also alert people that something special is going on (in this case, it helps alert them to the fact that the modules in this namespace *will* be moved in the future). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Jan 4, 2012 at 4:29 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I am afraid that dunders will become a standard thing in many Python packages, because of everybody's natural desire to use latest features that work right. That means that new people to Python will have to deal with Python magic right from the start, which will make language more complicated for them. A correct way IMO would be to introduce a versioned exp27, exp271 packages: 1. It makes it explicitly clear when imported package version was considered experimental 2. It allows to switch between multiple experimental package versions 3. It will provide future compatibility if a package released from `exp` breaks API 5. as a sum of 2+3 above - it provides a development process and extends time to stabilize API with public testing and feedback before freezing the package to die in stdlib -- anatoly t.
anatoly techtonik writes:
__preview__ is for the latest features that have not been confirmed to work right (yet). Where's the problem? Newbies certainly should not be using such in production applications, although it's an excellent way to learn everything you always wanted to know about Python internals but were afraid to ask (and more!) <wink/>
On Wed, Jan 4, 2012 at 2:29 PM, Stephen J. Turnbull <stephen@xemacs.org>wrote:
What's the point in developing code you won't use? __preview__ features don't have a Roadmap, so you may wait forever to release you package into production. With indefinite release cycle __preview__ will become useless for the most people, who will quickly lose enthusiasm to use __preview__ and continuously update their code for API breaks. -- anatoly t.
A correct way IMO would be to introduce a versioned exp27, exp271 packages:
+2. Rather than silently introducing subtle semantics changes, wouldn't it be better for older imports to fail with an ImportError? Since this is just for toying around, this shouldn't break any play, but it _should_ discourage (very strongly) people using experimental stuff in their published code. Perhaps a dunder name like __preview32__ ? __preview__._32 ?
1. It makes it explicitly clear when imported package version was considered experimental
I agree with this too. It's a good benefit.
Less sold on these. This isn't the purpose of __preview__ as I understand it, and including every revision of every __preview__ package could increase Python's size very dramatically over time. -- Devin On Wed, Jan 4, 2012 at 5:45 AM, anatoly techtonik <techtonik@gmail.com> wrote:
Surely preview should only contain whats being previewed for the next release then when its released with the next version they carry deprecation warnings and are removed in the following version e.g. v1 __preview__.magic v2 deprecate(__preview__.magic) v3 There is no longer __preview__.magic in __preview__ On Wed, Jan 4, 2012 at 11:49 AM, Devin Jeanpierre <jeanpierreda@gmail.com>wrote:
On Wed, Jan 4, 2012 at 10:03 PM, Jakob Bowyer <jkbbwr@gmail.com> wrote:
The expected progression is: 3.X __preview__.example 3.X+1 There is no longer a __preview__.example (What's New will say whether to find it in the standard library or on PyPI) I think there's a useful set of guidelines to come out of this discussion, though: 1. All __preview__ candidates must be available as supported modules on PyPI 2. Entry into __preview__ marks the start of the transition to the standard library. Issues may be reported both via project specific channels and core Python channels. 3. If insurmountable problems are encountered during the preview release (for example, buildbot instability on par with that historically exhibited by bsddb), the project will be removed from __preview__ and revert to standalone status 4. If no such problems are encountered, the module fully enters the standard library in the subsequent release, with the PyPI release remaining available for use in earlier Python versions So multi-version code (such as other packages published on PyPI) should generally depend on the PyPI version, but __preview__ becomes an option if you can't use the PyPI version for some reason. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Jan 4, 2012 at 9:49 PM, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
__preview__ isn't just for toying around: so long as you don't need to support multiple Python versions, it will be just as stable and well supported as the rest of the standard library. That means people deploying to controlled environments (such as the folks in corporate and governmental environments that are intended to be the primary beneficiaries) can use it quite happily (although keeping in mind the potential for changes when finally upgrading in the future). That's the entire point of the exercise. It's unlikely that things will be in the __preview__ namespace for more than one release - I expect that most packages that are of an adequate standard to get added in the first place will make it through their preview release period without significant changes. The process just gives us one final chance to get broader feedback before we flip the switch and lock in the API indefinitely. For folks that have an easier time of dependency management, it's likely a better bet for them to depend on a PyPI release of a package rather than using the __preview__ version, but such folks don't depend as much on the standard library in the first place. People also shouldn't read too much into the relaxation of the backwards compatibility guarantees - we aren't going to go nuts and just change things on a whim. We'll just be a little less conservative when we identify problems in the preview APIs that need to be fixed before they're added to the main part of the standard library. Deciding whether or not to rely on __preview__ modules, or to depend on packages that in turn use __preview__ modules will still be a decision developers need to make for themselves. That's no different from any other form of due diligence that professional developers need to conduct on their dependencies today. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Ah, this would remove much of the point of explicit version numbering wouldn't it? Your other paragraphs make other good points against it too. It was a good response. I am no longer +2 on explicit versioning. I still find it nice for "experimental" code to be a bit clear about which experiment you're referring to, but it isn't very important if it doesn't last long and doesn't go through that many breaking changes. In fact, the way the use case is shaping up, it isn't important at all. I suspect I misremembered or only selectively remembered the original __preview__ proposal. I suppose this is a good reason to get a PEP out, huh. -- Devin On Wed, Jan 4, 2012 at 7:09 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Tue, Jan 3, 2012 at 6:08 AM, Eli Bendersky <eliben@gmail.com> wrote:
Before attempting any library refactoring or addition, I would first think about placing a feedback mechanism in place to collect public response about satisfaction with current stdlib contents, and collecting proposals about parts that should be included into stdlib (or another bundle shipped with Python). -- anatoly t.
On Wed, Jan 4, 2012 at 1:47 AM, anatoly techtonik <techtonik@gmail.com> wrote:
Sorry anatoly, you're not going to win that one. We're well aware you don't like the use of mailing lists, the issue tracker, IRC, blogs, Reddit, Hacker News, Stack Overflow, etc for feedback and want something else that is formal and structured, but isn't the issue tracker (since you object to that for reasons I don't really understand). If you want to set up your own feedback mechanism, encourage people to use it, and act as a conduit between your new mechanism and the channels the current core devs are already paying attention to, feel free. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Jan 3, 2012 at 1:08 PM, Eli Bendersky <eliben@gmail.com> wrote:
I think one thing that needs to be made crystal clear (both in the PEP and in the documentation for the namespace) is that anyone attempting to combine the preview namespace with persistence mechanisms that rely on module names are setting themselves up for migration problems in the future. It isn't just the case that the __preview__ namespace APIs *might* change: they're actually *guaranteed* to change, when they move to their final location (probably, but not necessarily, in the next release). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Just a suggestion: Golang uses an "exp" namespace, this would translate to a exp package for grouping like modules in a similar way that xml, http and concurrent are currently used. http://golang.org/pkg/exp/ The name is unoffensive, and somewhat more palatable than __preview__, which really isn't a protocol, or a bag for special interpreter switches like __future__. Under this scheme, the currently proposed modules would be: exp.regex exp.daemon exp.ipaddr Rather than __preview__.regex __preview__.daemon __preview__.ipaddr Either way I really look forward to something for PEP 3153 being trialed. http://www.python.org/dev/peps/pep-3153/ Matt On Tue, Jan 3, 2012 at 3:32 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
-- ಠ_ಠ
On 1/3/12 2:57 PM, Matt Joiner wrote:
The nice thing about the dunders is that no one will make a module with a conflicting name. That's really the common factor behind all uses of dunders: reserving a name for Python's use. Whether something is a protocol or bag of switches doesn't really enter into it. Like "new" and "random", "exp" or anything else that means something like "experimental" is going to trigger some problems when people use that name for *their* experimental modules. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Wed, Jan 4, 2012 at 3:29 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Dunder names also alert people that something special is going on (in this case, it helps alert them to the fact that the modules in this namespace *will* be moved in the future). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Jan 4, 2012 at 4:29 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I am afraid that dunders will become a standard thing in many Python packages, because of everybody's natural desire to use latest features that work right. That means that new people to Python will have to deal with Python magic right from the start, which will make language more complicated for them. A correct way IMO would be to introduce a versioned exp27, exp271 packages: 1. It makes it explicitly clear when imported package version was considered experimental 2. It allows to switch between multiple experimental package versions 3. It will provide future compatibility if a package released from `exp` breaks API 5. as a sum of 2+3 above - it provides a development process and extends time to stabilize API with public testing and feedback before freezing the package to die in stdlib -- anatoly t.
anatoly techtonik writes:
__preview__ is for the latest features that have not been confirmed to work right (yet). Where's the problem? Newbies certainly should not be using such in production applications, although it's an excellent way to learn everything you always wanted to know about Python internals but were afraid to ask (and more!) <wink/>
On Wed, Jan 4, 2012 at 2:29 PM, Stephen J. Turnbull <stephen@xemacs.org>wrote:
What's the point in developing code you won't use? __preview__ features don't have a Roadmap, so you may wait forever to release you package into production. With indefinite release cycle __preview__ will become useless for the most people, who will quickly lose enthusiasm to use __preview__ and continuously update their code for API breaks. -- anatoly t.
A correct way IMO would be to introduce a versioned exp27, exp271 packages:
+2. Rather than silently introducing subtle semantics changes, wouldn't it be better for older imports to fail with an ImportError? Since this is just for toying around, this shouldn't break any play, but it _should_ discourage (very strongly) people using experimental stuff in their published code. Perhaps a dunder name like __preview32__ ? __preview__._32 ?
1. It makes it explicitly clear when imported package version was considered experimental
I agree with this too. It's a good benefit.
Less sold on these. This isn't the purpose of __preview__ as I understand it, and including every revision of every __preview__ package could increase Python's size very dramatically over time. -- Devin On Wed, Jan 4, 2012 at 5:45 AM, anatoly techtonik <techtonik@gmail.com> wrote:
Surely preview should only contain whats being previewed for the next release then when its released with the next version they carry deprecation warnings and are removed in the following version e.g. v1 __preview__.magic v2 deprecate(__preview__.magic) v3 There is no longer __preview__.magic in __preview__ On Wed, Jan 4, 2012 at 11:49 AM, Devin Jeanpierre <jeanpierreda@gmail.com>wrote:
On Wed, Jan 4, 2012 at 10:03 PM, Jakob Bowyer <jkbbwr@gmail.com> wrote:
The expected progression is: 3.X __preview__.example 3.X+1 There is no longer a __preview__.example (What's New will say whether to find it in the standard library or on PyPI) I think there's a useful set of guidelines to come out of this discussion, though: 1. All __preview__ candidates must be available as supported modules on PyPI 2. Entry into __preview__ marks the start of the transition to the standard library. Issues may be reported both via project specific channels and core Python channels. 3. If insurmountable problems are encountered during the preview release (for example, buildbot instability on par with that historically exhibited by bsddb), the project will be removed from __preview__ and revert to standalone status 4. If no such problems are encountered, the module fully enters the standard library in the subsequent release, with the PyPI release remaining available for use in earlier Python versions So multi-version code (such as other packages published on PyPI) should generally depend on the PyPI version, but __preview__ becomes an option if you can't use the PyPI version for some reason. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Jan 4, 2012 at 9:49 PM, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
__preview__ isn't just for toying around: so long as you don't need to support multiple Python versions, it will be just as stable and well supported as the rest of the standard library. That means people deploying to controlled environments (such as the folks in corporate and governmental environments that are intended to be the primary beneficiaries) can use it quite happily (although keeping in mind the potential for changes when finally upgrading in the future). That's the entire point of the exercise. It's unlikely that things will be in the __preview__ namespace for more than one release - I expect that most packages that are of an adequate standard to get added in the first place will make it through their preview release period without significant changes. The process just gives us one final chance to get broader feedback before we flip the switch and lock in the API indefinitely. For folks that have an easier time of dependency management, it's likely a better bet for them to depend on a PyPI release of a package rather than using the __preview__ version, but such folks don't depend as much on the standard library in the first place. People also shouldn't read too much into the relaxation of the backwards compatibility guarantees - we aren't going to go nuts and just change things on a whim. We'll just be a little less conservative when we identify problems in the preview APIs that need to be fixed before they're added to the main part of the standard library. Deciding whether or not to rely on __preview__ modules, or to depend on packages that in turn use __preview__ modules will still be a decision developers need to make for themselves. That's no different from any other form of due diligence that professional developers need to conduct on their dependencies today. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Ah, this would remove much of the point of explicit version numbering wouldn't it? Your other paragraphs make other good points against it too. It was a good response. I am no longer +2 on explicit versioning. I still find it nice for "experimental" code to be a bit clear about which experiment you're referring to, but it isn't very important if it doesn't last long and doesn't go through that many breaking changes. In fact, the way the use case is shaping up, it isn't important at all. I suspect I misremembered or only selectively remembered the original __preview__ proposal. I suppose this is a good reason to get a PEP out, huh. -- Devin On Wed, Jan 4, 2012 at 7:09 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Tue, Jan 3, 2012 at 6:08 AM, Eli Bendersky <eliben@gmail.com> wrote:
Before attempting any library refactoring or addition, I would first think about placing a feedback mechanism in place to collect public response about satisfaction with current stdlib contents, and collecting proposals about parts that should be included into stdlib (or another bundle shipped with Python). -- anatoly t.
On Wed, Jan 4, 2012 at 1:47 AM, anatoly techtonik <techtonik@gmail.com> wrote:
Sorry anatoly, you're not going to win that one. We're well aware you don't like the use of mailing lists, the issue tracker, IRC, blogs, Reddit, Hacker News, Stack Overflow, etc for feedback and want something else that is formal and structured, but isn't the issue tracker (since you object to that for reasons I don't really understand). If you want to set up your own feedback mechanism, encourage people to use it, and act as a conduit between your new mechanism and the channels the current core devs are already paying attention to, feel free. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (10)
-
anatoly techtonik -
Antoine Pitrou -
Devin Jeanpierre -
Dj Gilcrease -
Eli Bendersky -
Jakob Bowyer -
Matt Joiner -
Nick Coghlan -
Robert Kern -
Stephen J. Turnbull