Open PEPs and large-scale changes for 3.3

With 3.3a3 tagged and the beta stage currently 2 months away, I would like to draw your attention to the following list of possible features for 3.3 as specified by PEP 398: Candidate PEPs: * PEP 362: Function Signature Object * PEP 395: Qualified Names for Modules * PEP 397: Python launcher for Windows * PEP 402: Simplified Package Layout (likely a new PEP derived from it) -- I assume PEP 420 is a candidate for that? * PEP 405: Python Virtual Environments * PEP 421: Adding sys.implementation * PEP 3143: Standard daemon process library * PEP 3144: IP Address manipulation library * PEP 3154: Pickle protocol version 4 Other planned large-scale changes: * Addition of the "regex" module * Email version 6 * A standard event-loop interface (PEP by Jim Fulton pending) * Breaking out standard library and docs in separate repos? Benjamin: I'd also like to know what will become of PEP 415. If anyone feels strongly about one of these items, please get ready to finalize and implement it well before June 23 (beta 1), or we have to discuss about adding another alpha. Also, if I missed any obvious candidate PEP or change, please let me know. cheers, Georg

On 5/1/2012 7:57 AM, Georg Brandl wrote:
Also, if I missed any obvious candidate PEP or change, please let me know.
I'd like to include PEP 420, Implicit Namespace Packages. We discussed it at PyCon, and a sample implementation is available at features/pep-420. Barry Warsaw, Jason Coombs, and I are sprinting this Thursday to hopefully finish up tests and other loose ends. Then we'll ask that it be accepted. If accepted, we should be able to get it in before alpha 4. Eric.

On May 01, 2012, at 08:24 AM, Eric V. Smith wrote:
Oops, I missed your reference to PEP 402 and PEP 420. Sorry about that.
It is indeed 420 that would replace 402.
And the older PEP 382. Once 420 is accepted, we should simply reject 382 and 402. At that point, I'll update them to point to 420. -Barry

On Tue, May 1, 2012 at 9:57 PM, Georg Brandl <g.brandl@gmx.net> wrote:
A few of those are on my plate, soo...
* PEP 395: Qualified Names for Modules
I'm currently thinking I'll defer this to 3.4. With the importlib change and PEP 420, there's already going to be an awful lot of churn in that space for 3.3, plus I have other things that I consider more important that I want to get done first.
* PEP 405: Python Virtual Environments
I pinged Carl and Vinay about the remaining open issues yesterday, and indicated I'd really like to have something I can pronounce on soon so we can get it into the fourth alpha on May 26. I'm hoping we'll see the next draft of the PEP soon, but the ball is back in their court for the moment.
* PEP 3144: IP Address manipulation library
This is pretty close to approval. Peter's addressed all the substantive comments that were made regarding the draft API, and he's going to provide an update to the PEP shortly that should get it into a state where I can mark it as Approved. Integration of the library and tests shouldn't be too hard, but it would really help if a sphinx expert could take a look at my Stack Overflow question [1] about generating an initial version of the API reference docs. (I've been meaning to figure out the right mailing list to send sphinx questions to, but haven't got around to it yet). [1] http://stackoverflow.com/questions/10377576/emit-restructuredtext-from-sphin...
* Breaking out standard library and docs in separate repos?
Our current development infrastructure simply isn't set up to cope with this. With both 407 and 413 still open (and not likely to go anywhere any time soon), this simply isn't going to happen for 3.3.
Benjamin: I'd also like to know what will become of PEP 415.
I emailed Guido and Benjamin about that one the other day. I'll be PEP czar, and the most likely outcome is that I'll approve the PEP as is and we'll create a separate tracker issue to discuss the exact behaviour of the traceback display functions when they're handed exceptions with __suppress_context__ set to False and __cause__ and __context__ are both non-None (Benjamin's patch preserves the status quo of only displaying __cause__ in that case, which I don't think is ideal, but also don't think is worth holding up PEP 415 over). I'm still waiting to hear back from Benjamin though. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, May 1, 2012 at 11:34 PM, Eli Bendersky <eliben@gmail.com> wrote:
Yeah, it will. While the ipaddr heritage means we can be confident the underlying implementation is solid, there's no need to be hasty in locking down the cleaned up API. Clarifying that is one of the updates I've asked Peter to make to the PEP before I can accept it. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, May 1, 2012 at 11:43 PM, Benjamin Peterson <benjamin@python.org> wrote:
Indeed, it's a decision to be made on a case-by-case basis when a module is up for inclusion. For example, the unittest.mock API isn't provisional, since it's already been well tested on PyPI. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, May 1, 2012 at 16:43, Benjamin Peterson <benjamin@python.org> wrote:
You're right, it doesn't require it. However, since Nick's summary above mentioned a "draft API", I thought this package can be a good candidate for a PEP-411-process. Without PEP 411, once a module gets into stdlib, its API is pretty much locked. If we are wary of such lock-in with the current state ipaddr's API is in, PEP 411 seems like a reasonable way to go. Eli

On 01.05.2012 15:30, Nick Coghlan wrote:
OK, I've moved this one to the "deferred" section for now.
Yes, there also was an RFC on the distutils-sig.
I can create that initial .rst for you. It is quite trivial, but not supported by Sphinx without hacking the autodoc code a little.
Agreed, and moved to deferred.
I've added 420 to the pending list in any case. Georg

On Wed, May 2, 2012 at 2:58 AM, Éric Araujo <merwok@netwok.org> wrote:
As near as I can tell, autogen does the same thing "apidoc" does - inserts autodoc directives in the generated .rst files that loads the docstrings at build time. I don't want that - I want to load the docstrings at generation time in order to use them as a basis for the hand written docs. Instead, I'll just take Georg up on his offer to generate the initial file for us. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, May 1, 2012 at 07:57, Georg Brandl <g.brandl@gmx.net> wrote:
This is mine and I can say that the chance of me getting to this in time is near zero. If someone wants to pick it up and try to finish up the work (which involves addressing Guido's comments on the PEP and seeing if the patch someone submitted is worth looking at) then I'm fine with that. Else this PEP will become a 3.4 addition. -Brett

On 2012-05-01, at 7:57 AM, Georg Brandl wrote:
Regarding PEP 362: there are some outstanding issues with the PEP, that should be resolved. I've outlined some in this email: http://mail.python.org/pipermail/python-dev/2012-March/117540.html If Brett is tied up with the importlib integration, I'd be glad to offer my help with adjustment of the PEP and reference implementation update. - Yury

On Tue, May 1, 2012 at 10:26, Yury Selivanov <yselivanov.ml@gmail.com>wrote:
Yes I am. =)
I'd be glad to offer my help with adjustment of the PEP and reference implementation update.
That would be great! First thing is addressing Guido's concerns from http://mail.python.org/pipermail/python-dev/2012-March/117515.html and then handling any issues you found. Not sure if Larry was asking about this out of curiosity or because he too wanted to help. I think the overall trick is keeping the API simple so it's easy to use but exposes what one could reasonably need (e.g. I wouldn't try to keep the order of keyword-only arguments).

On 05/01/2012 01:12 PM, Brett Cannon wrote:
Asking, that is, off-list. So your observation was kinda out of left field for the casual observer ;-) I was asking because I was interested in helping, but I haven't looked into it too much, and I'm not sure how much of a priority it is. It's clear that Yury has spent way more time with the issue. If he'd* like my help I'll try to lend it but I bet he's got it under control. /arry * Assuming "Yury" is a he; apologies if my shot in the dark was a miss.

On 2012-05-01, at 4:12 PM, Brett Cannon wrote:
That would be great! First thing is addressing Guido's concerns from http://mail.python.org/pipermail/python-dev/2012-March/117515.html and then handling any issues you found. Not sure if Larry was asking about this out of curiosity or because he too wanted to help.
Great! I'll start looking into this on the weekend. - Yury

On Tue, 01 May 2012 13:57:50 +0200, Georg Brandl <g.brandl@gmx.net> wrote:
I guess it's time to talk about my plans for this one :) RIM/QNX is currently paying me to work on their stuff rather than email6, (but it does leave me with some time for email6). However, while QNX directly funded a big chunk of email6, as a consequence of their current priorities the whole of the email6 spec isn't going to be implemented for Python3.3. There is, however, a very useful big chunk of it that is pretty much done: the improved header parsing, header API, and header folding. I covered the primary improvements in my PyCon talk, for those who were there or have seen the video. Even that is not quite complete, but I'm currently planning to finish it before alpha 4. (There may be a couple of details that won't make it in until beta1.) At the PyCon sprints I finished the folding implementation. It's every bit as ugly as the old folding implementation that I simplified some time ago, but it gets a lot more corner cases right, and implements an important feature that the old folding algorithm got wrong more often than not: folding at "higher level syntactic breaks". So while I'd like to revisit that code and improve it, it *works*. So any further work on that can be bug-fix stage. Also at the sprints I started on a performance refactoring. It has been bothering me for a while that any program using the new code would have been doing a complete RFC5322 parse on every header in every message, even if it was processing a boatload of messages, only cared about the content of a few headers, and wanted to just pass the rest through. I was treating fixing that as a premature optimization, though I had some thoughts about how to do so. Well, to my great surprise, the most logical way of fixing it turned out to have two significant benefits: the code got simpler, and it provides a way to maintain pretty much 100% backward compatibility with Python3.2. I guess some optimizations aren't premature. The basic scheme (which I have almost completely implemented in the email6 feature repo at this point) is to continue to store the raw data from a parse in the Message just like we always have, and only do the full RFC5322 parse when either an application program asks for the header, or a generator needs to re-fold that header for some reason. By setting the policy controls appropriately and being aware of the consequences of looking at a header, an application could take advantage of the new header parsing for headers of interest with minimal performance impact compared to 3.2. Now, here's the tricky bit. The new API for headers has been out on PyPI for review for almost a year now, but hasn't seen what you would call widespread use. In particular, I haven't gotten any feedback about it. It seems to me that introducing this new API in 3.3 would be a perfect application of PEP 411...except that email is already a package in the standard library. This is where the backward-compatibility of my performance refactor comes in. The way this works is that the policy object, which has already been added to the 3.3 codebase and *has* gotten some review and feedback, controls what happens to the headers. The way the code in the 'nemail6' branch of /features/email6 currently works is that the policy used by default is named 'compat32'. (Actually it's compat5 right now in the repository, but I plan to change the name today.) That policy implements the exact same header handling that 3.2 currently uses (bugs and all). The new header handling is introduced by any *other* pre-defined policy an application may select. Thus, if code is not changed to use one of the new named policies, nothing changes and we have full backward compatibility. If a policy is specified, then the new header handling code (and the API it provides) is used. What I'm currently preparing is two patches. The first patch will refactor the policy code that was already committed so that the above scheme can be implemented, and so that compat32 is the default policy for 3.3. (This is the 'nemail6base' branch in /features/email6.) The second patch will use the policy hooks introduced by the first patch to add the new policies that use the new header parsing/folding code. My plan is that the first patch will go into 3.3 regardless (and should be ready for review/commit soon). What I'd like to do is have the second patch introduce the new policies as *provisional policies*. That is, in the spirit but not the letter of PEP 411, I'd like the new header API to be considered provisional and subject to improvement in 3.4 based on what we learn by having it actually out there in the field and getting tested. --David

On May 01, 2012, at 10:40 AM, R. David Murray wrote:
I guess it's time to talk about my plans for this one :)
Thanks for the update RDM. I really wish I had more time to contribute to email6, but I'd still really like to see this land in 3.3 if possible. I suspect you're just not going to get much practical feedback on email6 until it's available in Python's stdlib. I don't know how many Python 3 email consuming applications there are out there. The one I'm intimately familiar with <wink> still can't port to Python 3 because of its dependencies.
That seems reasonable to me. The documentation should be clear as to what's provisional and what's stable. With that, and based on your level of confidence, I'd be in favor of getting email6 into Python 3.3. Cheers, -Barry

On Tue, 01 May 2012 10:55:03 -0400, Barry Warsaw <barry@python.org> wrote:
My thought exactly.
OK, both patches are now up on the tracker. The first patch, as mentioned, does some internal refactoring that makes the policy framework cleaner and adds hooks and a 'compat32' policy implementation such that the current Python 3.2 behavior is preserved by default. That's issue 14731: http://bugs.python.org/issue14731 The second patch adds a policy implementation (marked as provisional) that adds the new header parsing and folding. As of this patch only 'Date' type and 'Address' type headers are parsed as anything other than Unstructured, but that's already worlds better than the compat32 policy. That's issue 12586: http://bugs.python.org/issue12586 I would appreciate reviews of both patches, even cursory ones. This split up should make them as easy to review as such big patches can be: the goal of 14731 is 100% backward compatibility, so a review can focus on making sure that the tests match the Python 3.2 tests (with some additions for bugs fixed). 125867 then adds a bunch of new code that can be evaluated on its own merits. Absent objection from patch reviewers, my plan is to apply these patches before the next alpha (which is scheduled for May 26th, ie: next weekend). --David

* PEP 397: Python launcher for Windows
I hope to submit a rewrite of this PEP RSN.
Also, if I missed any obvious candidate PEP or change, please let me know.
A big pending change is the switch to a new Visual Studio release. The challenge here is that we need to stop using the outdated VS 2008, but then, VS 2010 will soon be outdated as well, so it would be sad (IMO) if we switch from one outdated tool to the next. Therefore, I would really like to see Python 3.3 use VS 2012, except that this won't be released for a few more months (the release is likely along with the release for Windows 8, which likely happens "this summer"). So what specific VS release we use may depend on whether there will be another alpha release or not (but it may also be that another alpha release still won't buy enough time, so that we use VS 2008 for 2.7, VS 2010 for 3.3, and VS 2012 for 3.4). Regards, Martin P.S. There is (as of yet unconfirmed) rumor that VS 2012 won't support XP, which would clearly rule it out for Python 3.3, and likely also for 3.4. It also appears that VS 2012 might include the VS 2010 tool chain, which means that this tool chain won't be that outdated. P.P.S. this affects primarily the build files and the packaging, but then also affects distutils etc., and the buildbots - for the latter, switching the VS version likely means that all Windows buildbots will break, likely requiring several months for them to come back. P.P.P.S. People, please don't propose to drop VS in favor of gcc. That won't happen.

On 01.05.2012 17:48, "Martin v. Löwis" wrote:
* PEP 397: Python launcher for Windows
I hope to submit a rewrite of this PEP RSN.
Good to hear.
Do you know when a more detailed schedule for VS 2012 will be available (and confirmation regarding XP support)? While I agree that it would be best to use the most up-to-date toolchain, we shouldn't defer the beta stage indefinitely if there is no concrete date set.
Which is definitely not something we want to do during beta stage. Georg

Do you know when a more detailed schedule for VS 2012 will be available (and confirmation regarding XP support)?
Unfortunately, Microsoft doesn't publish any release dates. It's ready when it's ready :-( I just search again, and it appears that some roadmap has leaked: http://www.zdnet.com/blog/microsoft/microsoft-roadmap-leaks-for-office-15-ie... That says that a release is scheduled for "late 2012", which would put it after the Python 3.3 release (contrary to rumors I heard elsewhere). Regards, Martin

Georg Brandl <g.brandl@gmx.net> writes:
* PEP 3143: Standard daemon process library
Our porting work will not be done in time for Python 3.3. I will update this to target Python 3.4. -- \ “The best mind-altering drug is truth.” —Jane Wagner, via Lily | `\ Tomlin | _o__) | Ben Finney

On 05/02/2012 02:24 AM, Ben Finney wrote:
I think that http://0pointer.de/public/systemd-man/daemon.html would a good addition to the 'see also' section. It contains a detailed listing of steps to be taked during daemonization. Zbyszek

Ben Finney <ben+python@benfinney.id.au> writes:
The PEP document currently says it targets “3.x”. I'll leave it in that state until we're more confident that the current work will be on track for a particular Python release. Do I need to do anything in particular to be explicit that PEP 3143 is not coming in Python 3.3? -- \ “Human reason is snatching everything to itself, leaving | `\ nothing for faith.” —Bernard of Clairvaux, 1090–1153 CE | _o__) | Ben Finney

On 5/1/2012 7:57 AM, Georg Brandl wrote:
Also, if I missed any obvious candidate PEP or change, please let me know.
I'd like to include PEP 420, Implicit Namespace Packages. We discussed it at PyCon, and a sample implementation is available at features/pep-420. Barry Warsaw, Jason Coombs, and I are sprinting this Thursday to hopefully finish up tests and other loose ends. Then we'll ask that it be accepted. If accepted, we should be able to get it in before alpha 4. Eric.

On May 01, 2012, at 08:24 AM, Eric V. Smith wrote:
Oops, I missed your reference to PEP 402 and PEP 420. Sorry about that.
It is indeed 420 that would replace 402.
And the older PEP 382. Once 420 is accepted, we should simply reject 382 and 402. At that point, I'll update them to point to 420. -Barry

On Tue, May 1, 2012 at 9:57 PM, Georg Brandl <g.brandl@gmx.net> wrote:
A few of those are on my plate, soo...
* PEP 395: Qualified Names for Modules
I'm currently thinking I'll defer this to 3.4. With the importlib change and PEP 420, there's already going to be an awful lot of churn in that space for 3.3, plus I have other things that I consider more important that I want to get done first.
* PEP 405: Python Virtual Environments
I pinged Carl and Vinay about the remaining open issues yesterday, and indicated I'd really like to have something I can pronounce on soon so we can get it into the fourth alpha on May 26. I'm hoping we'll see the next draft of the PEP soon, but the ball is back in their court for the moment.
* PEP 3144: IP Address manipulation library
This is pretty close to approval. Peter's addressed all the substantive comments that were made regarding the draft API, and he's going to provide an update to the PEP shortly that should get it into a state where I can mark it as Approved. Integration of the library and tests shouldn't be too hard, but it would really help if a sphinx expert could take a look at my Stack Overflow question [1] about generating an initial version of the API reference docs. (I've been meaning to figure out the right mailing list to send sphinx questions to, but haven't got around to it yet). [1] http://stackoverflow.com/questions/10377576/emit-restructuredtext-from-sphin...
* Breaking out standard library and docs in separate repos?
Our current development infrastructure simply isn't set up to cope with this. With both 407 and 413 still open (and not likely to go anywhere any time soon), this simply isn't going to happen for 3.3.
Benjamin: I'd also like to know what will become of PEP 415.
I emailed Guido and Benjamin about that one the other day. I'll be PEP czar, and the most likely outcome is that I'll approve the PEP as is and we'll create a separate tracker issue to discuss the exact behaviour of the traceback display functions when they're handed exceptions with __suppress_context__ set to False and __cause__ and __context__ are both non-None (Benjamin's patch preserves the status quo of only displaying __cause__ in that case, which I don't think is ideal, but also don't think is worth holding up PEP 415 over). I'm still waiting to hear back from Benjamin though. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, May 1, 2012 at 11:34 PM, Eli Bendersky <eliben@gmail.com> wrote:
Yeah, it will. While the ipaddr heritage means we can be confident the underlying implementation is solid, there's no need to be hasty in locking down the cleaned up API. Clarifying that is one of the updates I've asked Peter to make to the PEP before I can accept it. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, May 1, 2012 at 11:43 PM, Benjamin Peterson <benjamin@python.org> wrote:
Indeed, it's a decision to be made on a case-by-case basis when a module is up for inclusion. For example, the unittest.mock API isn't provisional, since it's already been well tested on PyPI. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, May 1, 2012 at 16:43, Benjamin Peterson <benjamin@python.org> wrote:
You're right, it doesn't require it. However, since Nick's summary above mentioned a "draft API", I thought this package can be a good candidate for a PEP-411-process. Without PEP 411, once a module gets into stdlib, its API is pretty much locked. If we are wary of such lock-in with the current state ipaddr's API is in, PEP 411 seems like a reasonable way to go. Eli

On 01.05.2012 15:30, Nick Coghlan wrote:
OK, I've moved this one to the "deferred" section for now.
Yes, there also was an RFC on the distutils-sig.
I can create that initial .rst for you. It is quite trivial, but not supported by Sphinx without hacking the autodoc code a little.
Agreed, and moved to deferred.
I've added 420 to the pending list in any case. Georg

On Wed, May 2, 2012 at 2:58 AM, Éric Araujo <merwok@netwok.org> wrote:
As near as I can tell, autogen does the same thing "apidoc" does - inserts autodoc directives in the generated .rst files that loads the docstrings at build time. I don't want that - I want to load the docstrings at generation time in order to use them as a basis for the hand written docs. Instead, I'll just take Georg up on his offer to generate the initial file for us. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, May 1, 2012 at 07:57, Georg Brandl <g.brandl@gmx.net> wrote:
This is mine and I can say that the chance of me getting to this in time is near zero. If someone wants to pick it up and try to finish up the work (which involves addressing Guido's comments on the PEP and seeing if the patch someone submitted is worth looking at) then I'm fine with that. Else this PEP will become a 3.4 addition. -Brett

On 2012-05-01, at 7:57 AM, Georg Brandl wrote:
Regarding PEP 362: there are some outstanding issues with the PEP, that should be resolved. I've outlined some in this email: http://mail.python.org/pipermail/python-dev/2012-March/117540.html If Brett is tied up with the importlib integration, I'd be glad to offer my help with adjustment of the PEP and reference implementation update. - Yury

On Tue, May 1, 2012 at 10:26, Yury Selivanov <yselivanov.ml@gmail.com>wrote:
Yes I am. =)
I'd be glad to offer my help with adjustment of the PEP and reference implementation update.
That would be great! First thing is addressing Guido's concerns from http://mail.python.org/pipermail/python-dev/2012-March/117515.html and then handling any issues you found. Not sure if Larry was asking about this out of curiosity or because he too wanted to help. I think the overall trick is keeping the API simple so it's easy to use but exposes what one could reasonably need (e.g. I wouldn't try to keep the order of keyword-only arguments).

On 05/01/2012 01:12 PM, Brett Cannon wrote:
Asking, that is, off-list. So your observation was kinda out of left field for the casual observer ;-) I was asking because I was interested in helping, but I haven't looked into it too much, and I'm not sure how much of a priority it is. It's clear that Yury has spent way more time with the issue. If he'd* like my help I'll try to lend it but I bet he's got it under control. /arry * Assuming "Yury" is a he; apologies if my shot in the dark was a miss.

On 2012-05-01, at 4:12 PM, Brett Cannon wrote:
That would be great! First thing is addressing Guido's concerns from http://mail.python.org/pipermail/python-dev/2012-March/117515.html and then handling any issues you found. Not sure if Larry was asking about this out of curiosity or because he too wanted to help.
Great! I'll start looking into this on the weekend. - Yury

On Tue, 01 May 2012 13:57:50 +0200, Georg Brandl <g.brandl@gmx.net> wrote:
I guess it's time to talk about my plans for this one :) RIM/QNX is currently paying me to work on their stuff rather than email6, (but it does leave me with some time for email6). However, while QNX directly funded a big chunk of email6, as a consequence of their current priorities the whole of the email6 spec isn't going to be implemented for Python3.3. There is, however, a very useful big chunk of it that is pretty much done: the improved header parsing, header API, and header folding. I covered the primary improvements in my PyCon talk, for those who were there or have seen the video. Even that is not quite complete, but I'm currently planning to finish it before alpha 4. (There may be a couple of details that won't make it in until beta1.) At the PyCon sprints I finished the folding implementation. It's every bit as ugly as the old folding implementation that I simplified some time ago, but it gets a lot more corner cases right, and implements an important feature that the old folding algorithm got wrong more often than not: folding at "higher level syntactic breaks". So while I'd like to revisit that code and improve it, it *works*. So any further work on that can be bug-fix stage. Also at the sprints I started on a performance refactoring. It has been bothering me for a while that any program using the new code would have been doing a complete RFC5322 parse on every header in every message, even if it was processing a boatload of messages, only cared about the content of a few headers, and wanted to just pass the rest through. I was treating fixing that as a premature optimization, though I had some thoughts about how to do so. Well, to my great surprise, the most logical way of fixing it turned out to have two significant benefits: the code got simpler, and it provides a way to maintain pretty much 100% backward compatibility with Python3.2. I guess some optimizations aren't premature. The basic scheme (which I have almost completely implemented in the email6 feature repo at this point) is to continue to store the raw data from a parse in the Message just like we always have, and only do the full RFC5322 parse when either an application program asks for the header, or a generator needs to re-fold that header for some reason. By setting the policy controls appropriately and being aware of the consequences of looking at a header, an application could take advantage of the new header parsing for headers of interest with minimal performance impact compared to 3.2. Now, here's the tricky bit. The new API for headers has been out on PyPI for review for almost a year now, but hasn't seen what you would call widespread use. In particular, I haven't gotten any feedback about it. It seems to me that introducing this new API in 3.3 would be a perfect application of PEP 411...except that email is already a package in the standard library. This is where the backward-compatibility of my performance refactor comes in. The way this works is that the policy object, which has already been added to the 3.3 codebase and *has* gotten some review and feedback, controls what happens to the headers. The way the code in the 'nemail6' branch of /features/email6 currently works is that the policy used by default is named 'compat32'. (Actually it's compat5 right now in the repository, but I plan to change the name today.) That policy implements the exact same header handling that 3.2 currently uses (bugs and all). The new header handling is introduced by any *other* pre-defined policy an application may select. Thus, if code is not changed to use one of the new named policies, nothing changes and we have full backward compatibility. If a policy is specified, then the new header handling code (and the API it provides) is used. What I'm currently preparing is two patches. The first patch will refactor the policy code that was already committed so that the above scheme can be implemented, and so that compat32 is the default policy for 3.3. (This is the 'nemail6base' branch in /features/email6.) The second patch will use the policy hooks introduced by the first patch to add the new policies that use the new header parsing/folding code. My plan is that the first patch will go into 3.3 regardless (and should be ready for review/commit soon). What I'd like to do is have the second patch introduce the new policies as *provisional policies*. That is, in the spirit but not the letter of PEP 411, I'd like the new header API to be considered provisional and subject to improvement in 3.4 based on what we learn by having it actually out there in the field and getting tested. --David

On May 01, 2012, at 10:40 AM, R. David Murray wrote:
I guess it's time to talk about my plans for this one :)
Thanks for the update RDM. I really wish I had more time to contribute to email6, but I'd still really like to see this land in 3.3 if possible. I suspect you're just not going to get much practical feedback on email6 until it's available in Python's stdlib. I don't know how many Python 3 email consuming applications there are out there. The one I'm intimately familiar with <wink> still can't port to Python 3 because of its dependencies.
That seems reasonable to me. The documentation should be clear as to what's provisional and what's stable. With that, and based on your level of confidence, I'd be in favor of getting email6 into Python 3.3. Cheers, -Barry

On Tue, 01 May 2012 10:55:03 -0400, Barry Warsaw <barry@python.org> wrote:
My thought exactly.
OK, both patches are now up on the tracker. The first patch, as mentioned, does some internal refactoring that makes the policy framework cleaner and adds hooks and a 'compat32' policy implementation such that the current Python 3.2 behavior is preserved by default. That's issue 14731: http://bugs.python.org/issue14731 The second patch adds a policy implementation (marked as provisional) that adds the new header parsing and folding. As of this patch only 'Date' type and 'Address' type headers are parsed as anything other than Unstructured, but that's already worlds better than the compat32 policy. That's issue 12586: http://bugs.python.org/issue12586 I would appreciate reviews of both patches, even cursory ones. This split up should make them as easy to review as such big patches can be: the goal of 14731 is 100% backward compatibility, so a review can focus on making sure that the tests match the Python 3.2 tests (with some additions for bugs fixed). 125867 then adds a bunch of new code that can be evaluated on its own merits. Absent objection from patch reviewers, my plan is to apply these patches before the next alpha (which is scheduled for May 26th, ie: next weekend). --David

* PEP 397: Python launcher for Windows
I hope to submit a rewrite of this PEP RSN.
Also, if I missed any obvious candidate PEP or change, please let me know.
A big pending change is the switch to a new Visual Studio release. The challenge here is that we need to stop using the outdated VS 2008, but then, VS 2010 will soon be outdated as well, so it would be sad (IMO) if we switch from one outdated tool to the next. Therefore, I would really like to see Python 3.3 use VS 2012, except that this won't be released for a few more months (the release is likely along with the release for Windows 8, which likely happens "this summer"). So what specific VS release we use may depend on whether there will be another alpha release or not (but it may also be that another alpha release still won't buy enough time, so that we use VS 2008 for 2.7, VS 2010 for 3.3, and VS 2012 for 3.4). Regards, Martin P.S. There is (as of yet unconfirmed) rumor that VS 2012 won't support XP, which would clearly rule it out for Python 3.3, and likely also for 3.4. It also appears that VS 2012 might include the VS 2010 tool chain, which means that this tool chain won't be that outdated. P.P.S. this affects primarily the build files and the packaging, but then also affects distutils etc., and the buildbots - for the latter, switching the VS version likely means that all Windows buildbots will break, likely requiring several months for them to come back. P.P.P.S. People, please don't propose to drop VS in favor of gcc. That won't happen.

On 01.05.2012 17:48, "Martin v. Löwis" wrote:
* PEP 397: Python launcher for Windows
I hope to submit a rewrite of this PEP RSN.
Good to hear.
Do you know when a more detailed schedule for VS 2012 will be available (and confirmation regarding XP support)? While I agree that it would be best to use the most up-to-date toolchain, we shouldn't defer the beta stage indefinitely if there is no concrete date set.
Which is definitely not something we want to do during beta stage. Georg

Do you know when a more detailed schedule for VS 2012 will be available (and confirmation regarding XP support)?
Unfortunately, Microsoft doesn't publish any release dates. It's ready when it's ready :-( I just search again, and it appears that some roadmap has leaked: http://www.zdnet.com/blog/microsoft/microsoft-roadmap-leaks-for-office-15-ie... That says that a release is scheduled for "late 2012", which would put it after the Python 3.3 release (contrary to rumors I heard elsewhere). Regards, Martin

Georg Brandl <g.brandl@gmx.net> writes:
* PEP 3143: Standard daemon process library
Our porting work will not be done in time for Python 3.3. I will update this to target Python 3.4. -- \ “The best mind-altering drug is truth.” —Jane Wagner, via Lily | `\ Tomlin | _o__) | Ben Finney

On 05/02/2012 02:24 AM, Ben Finney wrote:
I think that http://0pointer.de/public/systemd-man/daemon.html would a good addition to the 'see also' section. It contains a detailed listing of steps to be taked during daemonization. Zbyszek

Ben Finney <ben+python@benfinney.id.au> writes:
The PEP document currently says it targets “3.x”. I'll leave it in that state until we're more confident that the current work will be on track for a particular Python release. Do I need to do anything in particular to be explicit that PEP 3143 is not coming in Python 3.3? -- \ “Human reason is snatching everything to itself, leaving | `\ nothing for faith.” —Bernard of Clairvaux, 1090–1153 CE | _o__) | Ben Finney
participants (15)
-
"Martin v. Löwis"
-
Barry Warsaw
-
Ben Finney
-
Benjamin Peterson
-
Brett Cannon
-
Eli Bendersky
-
Eric V. Smith
-
Georg Brandl
-
Larry Hastings
-
martin@v.loewis.de
-
Nick Coghlan
-
R. David Murray
-
Yury Selivanov
-
Zbigniew Jędrzejewski-Szmek
-
Éric Araujo