EuroPython Language Summit report
EuroPython 2011 Language Summit =============================== Here's a brief report on the EuroPython 2011 Language Summit, held on Sunday 19 June, 2011 in Florence. It was a fairly small meeting, with a lot of informal and loosely-focused discussion and few conclusions reached. I've outlined the topics that we discussed below. Present: Antonio Cuni Mark Dickinson Larry Hastings (chair) Marc-André Lemburg Ezio Melotti Antoine Pitrou Armin Ronacher Armin Rigo Mark Ramm Topics covered ============== Python 3 adoption ----------------- This was a long and wide-ranging discussion on the general state of Python 3 adoption. Some highlights: - Even amongst those present at the meeting, most are still using Python 2 for everyday work. - pypy: no definitive plans yet for PyPy supporting Python 3. - The web community is still very much focused on Python 2. - There's ongoing work (or perhaps just discussion?) on being able to generate Python 3 documentation using Sphinx running on Python 3. This would be particularly useful when using extensions like 'autodoc' for Python 3 code. - [Armin Ronacher] Python 3's Unicode support still has some dark areas. One example: when opening a text file for reading and writing, the default encoding used depends on the platform and on various environment variables. - Regular expression support in Python 3 needs improvement (support for Unicode character classes is a particular need). [Subtopic: what needs to be done to get the new regex module into Python? Should it replace the existing module? What about backwards compatibility issues?] - 2to3: It's still painful to upgrade a codebase from Python 2 to Python 3, and there doesn't seem to be a consensus on best practices yet. There was a long discussion that I can't hope to do justice to on whether 2to3 should be part of the standard library or not. (With side discussions on distutils and packaging in Python 3.) Armin Ronacher was one of the main participants in this discussion (and IIUC, the main opponent of having 2to3 in the standard library); Armin, do you want to summarize this? Another major topic of discussion was on how to manage Python 2 and Python 3 in plugin environments (e.g., Blender): is there some way that both Python 2 and Python 3 plugins could be used within the same system? This got a bit technical; perhaps someone else at the meeting would like to elaborate on this? State of Python 3 on the web ---------------------------- Brief discussion. Summary: it's improving; better than before. We have CherryPy and WSGI. There are issues in WSGI that are going to become more apparent as more people move to Python 3 frameworks. Discussion of open PEPS ----------------------- PEP 3151: (Reworking the OS and IO exception hierarchy.) Antoine summarized this; no real discussion here. PEP 3118: (Not open, but ...) Revising the buffer protocol. The buffer protocol implementation is still buggy and incomplete. (Mostly okay for 1d buffers, not even close for higher-dimensional work.) PEP 397: Python launcher for Windows. (This was actually discussed later, after lunch; apologies for messing with the timeline here.) There was some confusion over the purpose of this PEP. Armin Ronacher had some objections to the PEP in its current form, but it's not clear to me what those objects are, or whether they still exist. Armin, care to elaborate? PEP 0380: Syntax for delegating to a subgenerator. PEP 3150: Statement local namespaces. PEP 3152: Cofunctions. For all three of the above PEPs, there was some feeling that additional syntax changes to the language are unnecessary and make it harder to learn; where possible, we should prefer using 3rd partly libraries. Issue 12326 (Linux 3) ---------------------
From the issue report by Charles-François Natali::
Linus recently decided that the next Linux kernel version would 3.0 [1]. As a consequence, sys.platform (which is actually MACHDEP) will be 'linux3' on machines running Linux 3 kernels, and tests checking explicitely against 'linux2' will either break and won't run. There was significant discussion on how this should be solved. While there's a lot of discussion already on the tracker, the choice of solution may have significant ramifications, so it seems a good idea that this be discussed more widely on python-dev. Major questions:: - what should be done with Python 2.7? As it stands, as I understand it, sys.platform *will* become 'linux3' on Linux 3.x (it's a computed value somewhere) in Python 2.7.2, and since Python 2.7.2 is already out there we can't change that. Some of the proposed solutions to the issue would involve sys.platform changing between Python 2.7.2 and Python 2.7.3; is such a change acceptable for a bugfix release? - it's easy to fix up all uses of "== 'linux2' " in the standard library; is it okay to just let 3rd party code break here? - What should be done about Lib/plat-linux2? The rough consensus at the summit was that for Python 2.7 at least, the only realistic solution seems to be to do nothing except document the problem, and point people to the platform module. I'm not sure if this answers the Lib/plat-linux2 question. Python 2.7 ---------- How long will this be maintained? Original decision was 5 years. PyPI on EC2 ----------- There was some discussion several months ago about the possibility of hosting PyPI on EC2. Is this still something that should be considered? PyPI seems to have been much more problem free in recent times. virtualenv in Python 3.3? ------------------------- Apparently there was some discussion at the last PyCon about the possibility of virtualenv going into Python 3.3. As far as I know there's currently no open tracker item or PEP for this. Larry Hastings knows more here. -- Update from Larry: """I'm supportive of putting it in in 3.3, and I'm sitting with Raymond Hettinger right now and he supports it too. So I think if we get a working implementation it'll go in. It's under heavy discussion in c.l.p-d so I gather it's moving forward. Last I knew it was Carl Meyer pushing it forward, but Vinay Sanjip is the current standard-bearer. I understand Windows support will be a bit tricky; I don't know if they have someone who's going to handle it for them.""" Python website -------------- The Python website is painful to edit right now.
Le vendredi 24 juin 2011 à 10:52 +0200, Mark Dickinson a écrit :
- [Armin Ronacher] Python 3's Unicode support still has some dark areas.
What? Unicode support is perfect in Python 3!
One example: when opening a text file for reading and writing, the default encoding used depends on the platform and on various environment variables.
... oh, I agree. This choice is a big portability issue. Mac OS X, most Linux distro, BSD systems use UTF-8 local encoding, whereas Windows use legacy code pages like cp1252 (something like ISO-8859-1) or cp952 (shift jis). But sometimes, the locale is "C" (e.g. on our buildbots) and programs start to fail with Unicode errors... I see two options to improve the situation. (1) hard way: change open() API to make encoding a mandatory argument. Problem: it breaks compatibility with Python 3.0, 3.1 and 3.2 (ooops!); the encoding argument is the 4th argument, you have to use a keyword or choose a value for the buffering argument. I proposed to change open() API in Python 3.1 to make all arguments -except the filename and the mode- keyword-only arguments, but Guido rejected my idea: "Remember, for 3.0 we're trying to get a release out of the door, not cram in new features, no matter how small." http://bugs.python.org/issue4121 (2) soft way: add a warning if the encoding is implicit (locale encoding). I don't know what is the best warning type, and if it should be always displayed, only once, or not by default. Even if it is hidden by default, a careful developer will be able to use -Werror to fix bugs... I suspect that most tests fail if open() raises an exception if the encoding is not specified (e.g. see distutils/packaging issues about the file encoding). Victor
On 6/24/2011 7:18 AM, Victor Stinner wrote:
The third is to make utf-8 the default. I believe this *is* the proper long term solution and both options are contrary to this. I believe that this is what I want for myself even on Windows. I believe utf-8 is the default or only storage by cross-platform international programs (certainly the ones I use).
(1) hard way: change open() API to make encoding a mandatory argument.
(2) soft way: add a warning if the encoding is implicit (locale encoding).
(3) In 3.3, if the default is used and it is not utf-8, add a warning that the default will become utf-8 always in 3.4. Actually, I would like a PendingDeprecationWarning in 3.2.1 if possible. -- Terry Jan Reedy
Le vendredi 24 juin 2011 à 16:30 -0400, Terry Reedy a écrit :
Oh yes, I also prefer this option, but I suspect that some people prefer to not break backward compatibility. Or should we consider this bad design choice as a bug? The UTF-8 encoder (of Python 3) raises an error if the text contains a surrogate character. The surrogatepass error handler should be used to encode surrogages. ... The surrogateescape can be used to encode back undecodable bytes (e.g. filename decoded by Python using the surrogateescape), but it is not a good idea to write illegal byte sequences (most programs will fail to open the file).
I believe that this is what I want for myself even on Windows.
Can you open a UTF-8 files in all Windows program (and the text is displayed correctly)? I remember that notepad.exe writes an evil UTF-8 BOM, but I don't know if it requires this BOM to detect the UTF-8 encoding. Or do some program expect text files encoded to the ANSI code page? If you want to write files in the ANSI code page, you can use encoding="mbcs" (or use an explicit code page, like encoding="cp1252").
I'm not sure that the "and it is not utf-8" condition is a good idea. If you develop on Linux but your users are on Windows, you will not get the warning (even with -Werror) nor your users (users don't want to see warnings)... Or maybe an user using Windows and Linux will notice the bug without the warning :-) It doesn't mean that it is not possible to check your program: you can change your locale encoding (e.g. use LANG=C). At least, it will be possible to check test_distutils and test_packaging using LANG=C and -Werror :-) -- A fourth option is to use ASCII by default! Your program will work and be portable until you write the first non-ASCII character... Oh wait, it remembers me the Unicode nightmare of Python 2! Victor
Mark Dickinson <dickinsm <at> gmail.com> writes:
Mark, thanks for the summary. Re. a PEP for virtual environments in Python, Carl is working on the PEP. The first draft by Carl with my initial comments is at [1]. There's still some work to do before we can actually present it as a PEP we're happy with. I'm pleased to report good progress with the proof of concept implementation. There are some issues still with packaging which I'm working through with Éric Araujo, but I've gone ahead and committed changes on my branch to get things working. It's a good exercising of the packaging functionality :-) What I have at the moment is pythonv [2]: A modified Python which supports virtual environments via changes in Modules/getpath.c, PC/getpathp.c, Lib/site.py, Lib/sysconfig.py, Lib/sysconfig.cfg and Lib/distutils/sysconfig.py. These changes basically detect if you're running in a virtual environment by looking for an env.cfg, and if found, fixing it up so sys.path is set as it should be. In addition, sys.site_prefix and sys.site_exec_prefix are created - if not in a virtual env, these have the same values as sys.prefix and sys.exec_prefix. With just these changes, the basics of a virtual environment are provided (in terms of isolation from other environments and the system Python). However, in order to be useful, the packaging tools have to work with sys.site_prefix and sys.site_exec_prefix, so changes have been made to sysconfig, distutils and packaging to do this. I'm presently working on a demonstration application which integrates the above work with Doug Hellmann's virtualenvwrapper and Guillermo López' port of it to Powershell to get a nice basic tool that'll support virtual environments with packaging as an easy-to-use CLI, as well as Distribute support while people migrate over to packaging, on Windows as well as Linux. I'm expecting to cover both the Linux and Windows aspects (though I won't say no to help ;-) and working through packaging issues relating to improved Windows support. The basic functionality is there now, both in Windows and Linux - the focus of work is in the ease-of-use CLI stuff which is not envisaged to be part of core Python, but nevertheless needs to be done to make virtual envs as painless as possible. BTW Of the 398 packages on PyPI having a Py3K trove classifier, I've tested installing all of them into a pythonv virtual env with Distribute, and 310 install without errors. Of the remaining 88, some are due to missing dependencies, others due to broken packages on PyPI. BTW of the Python regression test suite passes all tests in a pythonv virtualenv, other than those for which there already are tracker issues for the default branch in hg.python.org/cpython (test_lib2to3, test_packaging, test_sysconfig). Full test results are at [3]. All in all, it's looking reasonably good, and we hope to report more progress on the PEP and the proof of concept soon! Regards, Vinay Sajip [1] http://goo.gl/6u0S0 [2] https://bitbucket.org/vinay.sajip/pythonv [3] https://gist.github.com/1022705
On Fri, 24 Jun 2011 10:52:40 +0200, Mark Dickinson <dickinsm@gmail.com> wrote:
I'm pretty sure regex has backward compatibility as a goal for just this reason (so it can replace the current module).
I disagree with this with respect to 380. Haven't looked at the others.
As I recall, the actual decision was "at *least* 5 years". It's only been one so far...was the discussion that five years was too short or too long? As the code bases continue to drift apart, and we fix the fixable bugs, the number of patches going in to 2.7 will decrease over time, so I don't think the burden of continuing to support it will be too heavy. Currently about half of the non-feature-request issues (ie: bugs) in the tracker are tagged 2.7. I expect to see that percentage continue to decrease over time. -- R. David Murray http://www.bitdance.com
On Sat, Jun 25, 2011 at 12:55 AM, R. David Murray <rdmurray@bitdance.com> wrote: [quoting VM summit notes]
Indeed, PEP 380 is *really* hard to do properly without language support. The language moratorium and lack of a Python 3 compatible patch are the only reasons it didn't go into 3.2 (and there's a patch porting the implementation to 3.3 up on bitbucket, although we've been tinkering with the compiler so it is likely out of date at this point). I'm the author PEP 3150 and *I* think it's a highly questionable and most likely terrible idea (hence the Deferred status). However, it's a good place to gather the related discussions, since proposals in that vein recur often on python-ideas. PEP 3152 definitely fits into the "let third parties experiment" bucket, though - once PEP 380 makes generators first class peers to functions in their support for modularisation, then we need to let the wider community play with the concept for a while before we embed anything more into the core language or standard library. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
At 10:46 AM 6/25/2011 +1000, Nick Coghlan wrote:
Indeed, PEP 380 is *really* hard to do properly without language support.
No, it isn't. You add a decorator, a 'from_' class, and a 'return_' function, and there you go. (See my previous code sketches here in early PEP 380 discussions.) Python frameworks have been doing variations of the same thing (with varying features and APIs) for at least 7 years now -- even on Python versions that lack decorators or the ability to return values from yield statements. So the main benefit of a PEP for this functionality would be providing a common implementation/API - and that could be initially done in the stdlib, without any added syntax support.
On Sat, 25 Jun 2011 11:31:42 -0400, "P.J. Eby" <pje@telecommunity.com> wrote:
So your proposed code would allow me, when writing a generator in my code, do something that would allow me to yield up all the values from an arbitrary generator I'm calling, over which I have no control (ie: I can't modify its code)? -- R. David Murray http://www.bitdance.com
On Sat, Jun 25, 2011 at 9:32 AM, R. David Murray <rdmurray@bitdance.com> wrote:
Let me cut this short. PEP 380 is pretty much approved. I know there are a few details worth quibbling over, but they are not going to jeopardize acceptance of the PEP. We are waiting for an implementation in Python 3.3. In fact, I wouldn't mind at this point if someone took their best effort at an implementation and checked it into the 3.3 branch, and we can do the quibbling over the details while we have a working implementation to experiment with. -- --Guido van Rossum (python.org/~guido)
At 12:32 PM 6/25/2011 -0400, R. David Murray wrote:
With a decorator on your own function, yes. See: http://mail.python.org/pipermail/python-dev/2010-July/102320.html for details. Mostly, though, that proposal was a suggestion for how the "optimized" implementation would work - i.e., a suggestion that PEP 380 be implemented that way under the hood, by implicitly turning 'yield from' into 'yield From()' and wrapping the generator itself with another From() instance. (IOW, that was a proposal for how to avoid the extra overhead of recursive yielding in a series of nested yield-from's.)
P.J. Eby wrote:
Will it handle *all* of the generator protocol correctly, including send(), exception handling, and generator closing? Also, how efficient would it be? A major benefit of a built-in implementation is that it can be almost as fast as using the sub-generator directly. -- Greg
On 24/06/2011 15:55, R. David Murray wrote:
The new regex library has some great improvements: http://bugs.python.org/issue2636 It also has users and committed maintainers, so I hope we can bring it into 3.3. It wasn't easy to tell from skimming the change notes that Unicode character classes are amongst the new features. Is that the case? Michael
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
Le mardi 28 juin 2011 à 14:41 +0100, Michael Foord a écrit :
This issue is open since April 2008, has also the longest list of attached files, and has a very long history. What is the status of the issue? I see that there is now a third party project on: http://code.google.com/p/mrab-regex-hg/ -- There is also the re2 library from Google and especially this project: http://pypi.python.org/pypi/re2/ "pyre2 is a Python extension that wraps Google's RE2 regular expression library. This version of pyre2 is similar to the one you'd find at facebook's github repository except that the stated goal of *this version is to be a drop-in replacement for the re module*.)" Victor
On 28.06.2011 16:06, Victor Stinner wrote:
This should be the same module as in the issue (and thankfully, because code management of such a big project does not belong exclusively in a tracker issue). "mrab" stands for Matthew Barnett, who is the author of regex.
Well, while it can be called drop-in, it is hardly a good replacement: """ That being said, there are features of the re module that this module may never have. For example, RE2 does not handle lookahead assertions ((?=...)). """ It falls back to old re in these cases. Georg
Michael Foord <fuzzyman@voidspace.org.uk> wrote:
According to http://effbot.org/zone/unicode-objects.htm (from 2004), the existing re module already supports Unicode character classes, so the regex module will as well. But the support has been updated, according to the change notes. Bill
On 28/06/2011 18:08, Bill Janssen wrote:
Michael
Bill
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
Le vendredi 24 juin 2011 à 10:52 +0200, Mark Dickinson a écrit :
- [Armin Ronacher] Python 3's Unicode support still has some dark areas.
What? Unicode support is perfect in Python 3!
One example: when opening a text file for reading and writing, the default encoding used depends on the platform and on various environment variables.
... oh, I agree. This choice is a big portability issue. Mac OS X, most Linux distro, BSD systems use UTF-8 local encoding, whereas Windows use legacy code pages like cp1252 (something like ISO-8859-1) or cp952 (shift jis). But sometimes, the locale is "C" (e.g. on our buildbots) and programs start to fail with Unicode errors... I see two options to improve the situation. (1) hard way: change open() API to make encoding a mandatory argument. Problem: it breaks compatibility with Python 3.0, 3.1 and 3.2 (ooops!); the encoding argument is the 4th argument, you have to use a keyword or choose a value for the buffering argument. I proposed to change open() API in Python 3.1 to make all arguments -except the filename and the mode- keyword-only arguments, but Guido rejected my idea: "Remember, for 3.0 we're trying to get a release out of the door, not cram in new features, no matter how small." http://bugs.python.org/issue4121 (2) soft way: add a warning if the encoding is implicit (locale encoding). I don't know what is the best warning type, and if it should be always displayed, only once, or not by default. Even if it is hidden by default, a careful developer will be able to use -Werror to fix bugs... I suspect that most tests fail if open() raises an exception if the encoding is not specified (e.g. see distutils/packaging issues about the file encoding). Victor
On 6/24/2011 7:18 AM, Victor Stinner wrote:
The third is to make utf-8 the default. I believe this *is* the proper long term solution and both options are contrary to this. I believe that this is what I want for myself even on Windows. I believe utf-8 is the default or only storage by cross-platform international programs (certainly the ones I use).
(1) hard way: change open() API to make encoding a mandatory argument.
(2) soft way: add a warning if the encoding is implicit (locale encoding).
(3) In 3.3, if the default is used and it is not utf-8, add a warning that the default will become utf-8 always in 3.4. Actually, I would like a PendingDeprecationWarning in 3.2.1 if possible. -- Terry Jan Reedy
Le vendredi 24 juin 2011 à 16:30 -0400, Terry Reedy a écrit :
Oh yes, I also prefer this option, but I suspect that some people prefer to not break backward compatibility. Or should we consider this bad design choice as a bug? The UTF-8 encoder (of Python 3) raises an error if the text contains a surrogate character. The surrogatepass error handler should be used to encode surrogages. ... The surrogateescape can be used to encode back undecodable bytes (e.g. filename decoded by Python using the surrogateescape), but it is not a good idea to write illegal byte sequences (most programs will fail to open the file).
I believe that this is what I want for myself even on Windows.
Can you open a UTF-8 files in all Windows program (and the text is displayed correctly)? I remember that notepad.exe writes an evil UTF-8 BOM, but I don't know if it requires this BOM to detect the UTF-8 encoding. Or do some program expect text files encoded to the ANSI code page? If you want to write files in the ANSI code page, you can use encoding="mbcs" (or use an explicit code page, like encoding="cp1252").
I'm not sure that the "and it is not utf-8" condition is a good idea. If you develop on Linux but your users are on Windows, you will not get the warning (even with -Werror) nor your users (users don't want to see warnings)... Or maybe an user using Windows and Linux will notice the bug without the warning :-) It doesn't mean that it is not possible to check your program: you can change your locale encoding (e.g. use LANG=C). At least, it will be possible to check test_distutils and test_packaging using LANG=C and -Werror :-) -- A fourth option is to use ASCII by default! Your program will work and be portable until you write the first non-ASCII character... Oh wait, it remembers me the Unicode nightmare of Python 2! Victor
Mark Dickinson <dickinsm <at> gmail.com> writes:
Mark, thanks for the summary. Re. a PEP for virtual environments in Python, Carl is working on the PEP. The first draft by Carl with my initial comments is at [1]. There's still some work to do before we can actually present it as a PEP we're happy with. I'm pleased to report good progress with the proof of concept implementation. There are some issues still with packaging which I'm working through with Éric Araujo, but I've gone ahead and committed changes on my branch to get things working. It's a good exercising of the packaging functionality :-) What I have at the moment is pythonv [2]: A modified Python which supports virtual environments via changes in Modules/getpath.c, PC/getpathp.c, Lib/site.py, Lib/sysconfig.py, Lib/sysconfig.cfg and Lib/distutils/sysconfig.py. These changes basically detect if you're running in a virtual environment by looking for an env.cfg, and if found, fixing it up so sys.path is set as it should be. In addition, sys.site_prefix and sys.site_exec_prefix are created - if not in a virtual env, these have the same values as sys.prefix and sys.exec_prefix. With just these changes, the basics of a virtual environment are provided (in terms of isolation from other environments and the system Python). However, in order to be useful, the packaging tools have to work with sys.site_prefix and sys.site_exec_prefix, so changes have been made to sysconfig, distutils and packaging to do this. I'm presently working on a demonstration application which integrates the above work with Doug Hellmann's virtualenvwrapper and Guillermo López' port of it to Powershell to get a nice basic tool that'll support virtual environments with packaging as an easy-to-use CLI, as well as Distribute support while people migrate over to packaging, on Windows as well as Linux. I'm expecting to cover both the Linux and Windows aspects (though I won't say no to help ;-) and working through packaging issues relating to improved Windows support. The basic functionality is there now, both in Windows and Linux - the focus of work is in the ease-of-use CLI stuff which is not envisaged to be part of core Python, but nevertheless needs to be done to make virtual envs as painless as possible. BTW Of the 398 packages on PyPI having a Py3K trove classifier, I've tested installing all of them into a pythonv virtual env with Distribute, and 310 install without errors. Of the remaining 88, some are due to missing dependencies, others due to broken packages on PyPI. BTW of the Python regression test suite passes all tests in a pythonv virtualenv, other than those for which there already are tracker issues for the default branch in hg.python.org/cpython (test_lib2to3, test_packaging, test_sysconfig). Full test results are at [3]. All in all, it's looking reasonably good, and we hope to report more progress on the PEP and the proof of concept soon! Regards, Vinay Sajip [1] http://goo.gl/6u0S0 [2] https://bitbucket.org/vinay.sajip/pythonv [3] https://gist.github.com/1022705
On Fri, 24 Jun 2011 10:52:40 +0200, Mark Dickinson <dickinsm@gmail.com> wrote:
I'm pretty sure regex has backward compatibility as a goal for just this reason (so it can replace the current module).
I disagree with this with respect to 380. Haven't looked at the others.
As I recall, the actual decision was "at *least* 5 years". It's only been one so far...was the discussion that five years was too short or too long? As the code bases continue to drift apart, and we fix the fixable bugs, the number of patches going in to 2.7 will decrease over time, so I don't think the burden of continuing to support it will be too heavy. Currently about half of the non-feature-request issues (ie: bugs) in the tracker are tagged 2.7. I expect to see that percentage continue to decrease over time. -- R. David Murray http://www.bitdance.com
On Sat, Jun 25, 2011 at 12:55 AM, R. David Murray <rdmurray@bitdance.com> wrote: [quoting VM summit notes]
Indeed, PEP 380 is *really* hard to do properly without language support. The language moratorium and lack of a Python 3 compatible patch are the only reasons it didn't go into 3.2 (and there's a patch porting the implementation to 3.3 up on bitbucket, although we've been tinkering with the compiler so it is likely out of date at this point). I'm the author PEP 3150 and *I* think it's a highly questionable and most likely terrible idea (hence the Deferred status). However, it's a good place to gather the related discussions, since proposals in that vein recur often on python-ideas. PEP 3152 definitely fits into the "let third parties experiment" bucket, though - once PEP 380 makes generators first class peers to functions in their support for modularisation, then we need to let the wider community play with the concept for a while before we embed anything more into the core language or standard library. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
At 10:46 AM 6/25/2011 +1000, Nick Coghlan wrote:
Indeed, PEP 380 is *really* hard to do properly without language support.
No, it isn't. You add a decorator, a 'from_' class, and a 'return_' function, and there you go. (See my previous code sketches here in early PEP 380 discussions.) Python frameworks have been doing variations of the same thing (with varying features and APIs) for at least 7 years now -- even on Python versions that lack decorators or the ability to return values from yield statements. So the main benefit of a PEP for this functionality would be providing a common implementation/API - and that could be initially done in the stdlib, without any added syntax support.
On Sat, 25 Jun 2011 11:31:42 -0400, "P.J. Eby" <pje@telecommunity.com> wrote:
So your proposed code would allow me, when writing a generator in my code, do something that would allow me to yield up all the values from an arbitrary generator I'm calling, over which I have no control (ie: I can't modify its code)? -- R. David Murray http://www.bitdance.com
On Sat, Jun 25, 2011 at 9:32 AM, R. David Murray <rdmurray@bitdance.com> wrote:
Let me cut this short. PEP 380 is pretty much approved. I know there are a few details worth quibbling over, but they are not going to jeopardize acceptance of the PEP. We are waiting for an implementation in Python 3.3. In fact, I wouldn't mind at this point if someone took their best effort at an implementation and checked it into the 3.3 branch, and we can do the quibbling over the details while we have a working implementation to experiment with. -- --Guido van Rossum (python.org/~guido)
At 12:32 PM 6/25/2011 -0400, R. David Murray wrote:
With a decorator on your own function, yes. See: http://mail.python.org/pipermail/python-dev/2010-July/102320.html for details. Mostly, though, that proposal was a suggestion for how the "optimized" implementation would work - i.e., a suggestion that PEP 380 be implemented that way under the hood, by implicitly turning 'yield from' into 'yield From()' and wrapping the generator itself with another From() instance. (IOW, that was a proposal for how to avoid the extra overhead of recursive yielding in a series of nested yield-from's.)
P.J. Eby wrote:
Will it handle *all* of the generator protocol correctly, including send(), exception handling, and generator closing? Also, how efficient would it be? A major benefit of a built-in implementation is that it can be almost as fast as using the sub-generator directly. -- Greg
On 24/06/2011 15:55, R. David Murray wrote:
The new regex library has some great improvements: http://bugs.python.org/issue2636 It also has users and committed maintainers, so I hope we can bring it into 3.3. It wasn't easy to tell from skimming the change notes that Unicode character classes are amongst the new features. Is that the case? Michael
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
Le mardi 28 juin 2011 à 14:41 +0100, Michael Foord a écrit :
This issue is open since April 2008, has also the longest list of attached files, and has a very long history. What is the status of the issue? I see that there is now a third party project on: http://code.google.com/p/mrab-regex-hg/ -- There is also the re2 library from Google and especially this project: http://pypi.python.org/pypi/re2/ "pyre2 is a Python extension that wraps Google's RE2 regular expression library. This version of pyre2 is similar to the one you'd find at facebook's github repository except that the stated goal of *this version is to be a drop-in replacement for the re module*.)" Victor
On 28.06.2011 16:06, Victor Stinner wrote:
This should be the same module as in the issue (and thankfully, because code management of such a big project does not belong exclusively in a tracker issue). "mrab" stands for Matthew Barnett, who is the author of regex.
Well, while it can be called drop-in, it is hardly a good replacement: """ That being said, there are features of the re module that this module may never have. For example, RE2 does not handle lookahead assertions ((?=...)). """ It falls back to old re in these cases. Georg
Michael Foord <fuzzyman@voidspace.org.uk> wrote:
According to http://effbot.org/zone/unicode-objects.htm (from 2004), the existing re module already supports Unicode character classes, so the regex module will as well. But the support has been updated, according to the change notes. Bill
On 28/06/2011 18:08, Bill Janssen wrote:
Michael
Bill
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
participants (13)
-
Bill Janssen -
Georg Brandl -
Glenn Linderman -
Greg Ewing -
Guido van Rossum -
Mark Dickinson -
Michael Foord -
Nick Coghlan -
P.J. Eby -
R. David Murray -
Terry Reedy -
Victor Stinner -
Vinay Sajip