Mailman 3 November 2014 - Python-Dev

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
by Isaac Schwabacher 26 Nov '14

26 Nov '14

On 11/25/14, Chris Angelico wrote: > On Wed, Nov 26, 2014 at 2:20 AM, Steven D'Aprano <steve(a)pearwood.info> wrote: > > I wouldn't interpret it like that. > > > > Calling next() on an empty iterator raises StopIteration. That's not a > > bug indicating a failure, it's the protocol working as expected. Your > > response to that may be to catch the StopIteration and ignore it, or to > > allow it to bubble up for something else to deal with it. Either way, > > next() raising StopIteration is not a bug, it is normal behaviour. > > > > (Failure to deal with any such StopIteration may be a bug.) > > > > However, if next() raises RuntimeError, that's not part of the protocol > > for iterators, so it is almost certainly a bug to be fixed. (Probably > > coming from an explicit "raise StopIteration" inside a generator > > function.) Your fix for the bug may be to refuse to fix it and just > > catch the exception and ignore it, but that's kind of nasty and hackish > > and shouldn't be considered good code. > > > > Do you agree this is a reasonable way to look at it? > > Yes. Specifically, your parenthesis in the middle is the important > bit. If you have a csv.DictReader, KeyError might be an important part > of your protocol (maybe you have an optional column in the CSV file), > but it should be caught before it crosses the boundary of "part of > your protocol". At some point, it needs to be converted into > ValueError, perhaps, or replaced with a default value, or some other > coping mechanism is used. Failure to deal with StopIteration when > calling next() is failure to cope with all of that function's > protocol, and that is most likely to be a bug. (There are times, and > some of them have been mentioned in these discussion threads, where > calling next() can never raise StopIteration, so there need be no > try/except - eg it=iter(string.split(" ")) - but that just means that > a StopIteration from that call is an error somewhere else. I'm > definitely happy for that kind of "shouldn't happen" to turn into a > RuntimeError rather than being left as an unexpectedly-short > generator.) Yield can also raise StopIteration, if it's thrown in. The current interaction of generator.throw(StopIteration) with yield from can't be emulated under the PEP's behavior, though it's not clear that that's a problem.

4 3

Re: [Python-Dev] cpython (3.4): handle errors without a reason attribute
by Antoine Pitrou 26 Nov '14

26 Nov '14

On Wed, 26 Nov 2014 00:06:06 +0000 benjamin.peterson <python-checkins(a)python.org> wrote: > https://hg.python.org/cpython/rev/e635c3ba75c8 > changeset: 93591:e635c3ba75c8 > branch: 3.4 > user: Benjamin Peterson <benjamin(a)python.org> > date: Tue Nov 25 15:43:58 2014 -0600 > summary: > handle errors without a reason attribute > > files: > Lib/test/support/__init__.py | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py > --- a/Lib/test/support/__init__.py > +++ b/Lib/test/support/__init__.py > @@ -698,7 +698,7 @@ > try: > f(*args, **kwargs) > except IOError as e: > - if e.reason == "CERTIFICATE_VERIFY_FAILED": > + if "CERTIFICATE_VERIFY_FAILED" in str(e): You should be able to keep the e.reason test if you only catch SSLError. Regards Antoine.

2 2

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
by Isaac Schwabacher 25 Nov '14

25 Nov '14

On 11/25/14, Guido van Rossum wrote: > On Tue, Nov 25, 2014 at 9:49 AM, Chris Angelico <ischwabacher(a)wisc.edu <rosuav(a)gmail.com')" target="1">rosuav(a)gmail.com> wrote: > > > On Wed, Nov 26, 2014 at 4:45 AM, Isaac Schwabacher > > <python.org/~guido(javascript:main.compose('new', 't=ischwabacher(a)wisc.edu>> wrote: > > > Yield can also raise StopIteration, if its thrown in. The current interaction of generator.throw(StopIteration) with yield from cant be emulated under the PEPs behavior, though its not clear that thats a problem. > > > > Hrm. I have *absolutely* no idea when you would use that, To close the innermost generator in a yield-from chain. No, I don't know why you'd want to do that, either. > > and how > > you'd go about reworking it to fit this proposal. Do you have any > > example code (production or synthetic) which throws StopIteration into > > a generator? No. > Sounds like a good one for the obfuscated Python contest. :-) I'm just playing with my food now. :) > Unless the generator has a try/except surrounding the yield point into which the exception is thrown, it will bubble right out, and PEP 479 will turn this into a RuntimeError. I'll clarify this in the PEP (even though it logically follows from the proposal) -- I don't think there's anything to worry about. > > > > -- > --Guido van Rossum (<a href=))

2 1

PEP 479: Change StopIteration handling inside generators
by Guido van Rossum 25 Nov '14

25 Nov '14

There's a new PEP proposing to change how to treat StopIteration bubbling up out of a generator frame (not caused by a return from the frame). The proposal is to replace such a StopIteration with a RuntimeError (chained to the original StopIteration), so that only *returning* from a generator (or falling off the end) causes the iteration to terminate. The proposal unifies the behavior of list comprehensions and generator expressions along the lines I had originally in mind when they were introduced. It renders useless/illegal certain hacks that have crept into some folks' arsenal of obfuscated Python tools. In Python 3.5 the proposed change is conditional on: from __future__ import replace_stopiteration_in_generators This would affect all generators (including generator expressions) compiled under its influence. The feature would become standard in Python 3.6 or 3.7. The PEP is here: https://www.python.org/dev/peps/pep-0479/ To avoid a lot of requests for clarification you may also want to read up on the python-ideas discussion, e.g. here: https://groups.google.com/forum/#!topic/python-ideas/yJi1gRot9yY I am leaning towards approving this PEP, but not until we've had a review here at python-dev. I would like to thank Chris Angelico for writing the original PEP draft. -- --Guido van Rossum (python.org/~guido)

15 62

PCBuild project updates
by Steve Dower 25 Nov '14

25 Nov '14

Hi all Just attracting some attention to http://bugs.python.org/issue22919 for those of us who build Python out of the PCBuild folder. More details/patches there, but the tl;dr is * Still works with VS 2010 (and now VS 2013 and VS 2015 too) * Build time should be reduced * Tools\buildbot\*.bat are also updated, so buildbots shouldn't notice any change If you're concerned/interested, I'll be watching the tracker for comments. Ideally I'd like at least a couple of people to test build with whatever VS version they have, but if nobody can do that then I'll assume that nobody will notice anything break sooner than me :) Cheers, Steve

2 1

Re: [Python-Dev] Please reconsider PEP 479.
by Isaac Schwabacher 25 Nov '14

25 Nov '14

On 11/24/14, Guido van Rossum wrote: > On Mon, Nov 24, 2014 at 1:32 PM, Isaac Schwabacher <ischwabacher(a)wisc.edu <ischwabacher(a)wisc.edu')" target="1">ischwabacher(a)wisc.edu> wrote: > > > On 11/24/14, Guido van Rossum wrote: > > > On Mon, Nov 24, 2014 at 8:14 AM, Isaac Schwabacher <python.org/~guido(javascript:main.compose('new', 't=ischwabacher(a)wisc.edu>(java_script:main.compose()> wrote: > > > > > > > On 11/23/14, Guido van Rossum wrote: > > > > > > > > > It wouldn't be so bad if we had the occasional generator author writing "raise StopIteration" instead of "return" to exit from a generator. (We could just add a recommendation against this to the style guide.) But the problem is that an unguarded next() call also raises StopIteration. Sometimes this is intentional (as in some itertools examples). But sometimes an unguarded next() call occurs deep in the bowels of some code called by the generator, and this situation is often hard to debug, since there is no stack track. > > > > > > > > I'll admit I've only skimmed the massive volume of correspondence this PEP has generated, but it seems to me that this is the main argument for this change. I can only assume that your support for this PEP is informed by your experience building Tulip, but isn't this the kind of thing that can be accomplished with a warning? Then you can get the same behavior without even needing a __future__ import to protect code bases that expect StopIteration to propagate (which seems like the more elegant and natural thing to do, even if it is more error-prone). > > > > > > Yes, this is my main reason for wanting the change -- but not just for tulip/asyncio. The issue can be just as baffling for anyone using unprotected next() calls in the context of a generator. But I'm not sure where to put the warning. Are you proposing to issue a warning under the same conditions the PEP says? > > > > Yes, I'm proposing issuing the warning at the point where the PEP raises, so that the PEP's behavior can be obtained with a warning filter (and such a filter could be installed by default around the asyncio main loop). > > > > > But then the itertools examples would issue warnings -- > > > > That's definitely problematic. They should either be fixed, or have the warning silenced with a comment about how the bubbling-up case is expected. > > So you agree with the problem that the PEP is trying to solve, you want people to fix their code in exactly the same way that the PEP is trying to get them to fix it, you want all new code that exhibits the problem to be flagged by a warning, and yet you do not support adding a __future__ statement and a a transition plan that replaces the warnings with hard failures in Python 3.7 (whose release date is going to be at least about four years in the future)? > > That sounds like the most loyal opposition I can wish for! :-) I agree with you that escaping StopIteration should be easier to notice, but with the opposition that allowing StopIteration to escape on purpose is a useful technique. But when you put it that way... > > > and I bet the advice would typically be "disable warnings" rather than "fix the code, otherwise it will break hard in Python 3.7". > > > > I don't think it's the language's responsibility to second guess a user who decides to explicitly silence such a warning. And if this *is* accomplished with a warning, then the user can just continue silencing it in 3.7. In my experience, though, python's documentation, StackOverflow presence, blogs, etc. have been absolutely stellar in terms of explaining why things are the way they are and how one should write pythonic code. I don't doubt the community's ability to educate users on this. > > Python's philosophy for (runtime) warnings is pretty clear -- a warning should never be silenced indefinitely. Warnings mean something's wrong with your code that won't get better by ignoring it, and you should fix it at some point. Until then you can silence the warning. Silencing warnings is an important mechanism for users who have no control over the code that issues the warning, and for devs who have more pressing priorities. But they should not be used to permanently enable coding in an "alternate universe" where the language has different features. I take it all back. ;D Nothing I've run across in the python ecosystem has made this point to me. (And google searches for "python warning philosophy" and the like don't find it, either.) I was interpreting the warning as indicating that there *might* be a problem in user code, so if the user knows why the system believes this and knows that the problem is a mirage, then it is reasonable to silence the warning indefinitely. In this interpretation it would make sense to simply add the warning and rely on users to determine whether their escaping StopIterations were intentional or not, without ever changing the default handling of the warning to "raise". But if warnings should never be silenced indefinitely, then this is a silly thing to do. > > I think the biggest stumbling block for this proposal is the fact that the current warning machinery doesn't appear to be up to the task of silencing a known-harmless warning in one generator without silencing meaningful warnings in generators it calls. > > You can get pretty darn specific with the warnings silencing machinery: up to the module and line number. It's intentional that you can't specify a class/method -- the latter would just encourage devs to silence a specific warning because they think they know better. And this fact makes so much more sense in context. ijs

3 2

Re: [Python-Dev] Please reconsider PEP 479.
by Isaac Schwabacher 25 Nov '14

25 Nov '14

On 11/24/14, Guido van Rossum wrote: > On Mon, Nov 24, 2014 at 8:14 AM, Isaac Schwabacher <ischwabacher@wisc.edu(javascript:main.compose()> wrote: > > > On 11/23/14, Guido van Rossum wrote: > > > > > It wouldn't be so bad if we had the occasional generator author writing "raise StopIteration" instead of "return" to exit from a generator. (We could just add a recommendation against this to the style guide.) But the problem is that an unguarded next() call also raises StopIteration. Sometimes this is intentional (as in some itertools examples). But sometimes an unguarded next() call occurs deep in the bowels of some code called by the generator, and this situation is often hard to debug, since there is no stack track. > > > > I'll admit I've only skimmed the massive volume of correspondence this PEP has generated, but it seems to me that this is the main argument for this change. I can only assume that your support for this PEP is informed by your experience building Tulip, but isn't this the kind of thing that can be accomplished with a warning? Then you can get the same behavior without even needing a __future__ import to protect code bases that expect StopIteration to propagate (which seems like the more elegant and natural thing to do, even if it is more error-prone). > > Yes, this is my main reason for wanting the change -- but not just for tulip/asyncio. The issue can be just as baffling for anyone using unprotected next() calls in the context of a generator. But I'm not sure where to put the warning. Are you proposing to issue a warning under the same conditions the PEP says? Yes, I'm proposing issuing the warning at the point where the PEP raises, so that the PEP's behavior can be obtained with a warning filter (and such a filter could be installed by default around the asyncio main loop). > But then the itertools examples would issue warnings -- That's definitely problematic. They should either be fixed, or have the warning silenced with a comment about how the bubbling-up case is expected. > and I bet the advice would typically be "disable warnings" rather than "fix the code, otherwise it will break hard in Python 3.7". I don't think it's the language's responsibility to second guess a user who decides to explicitly silence such a warning. And if this *is* accomplished with a warning, then the user can just continue silencing it in 3.7. In my experience, though, python's documentation, StackOverflow presence, blogs, etc. have been absolutely stellar in terms of explaining why things are the way they are and how one should write pythonic code. I don't doubt the community's ability to educate users on this. I think the biggest stumbling block for this proposal is the fact that the current warning machinery doesn't appear to be up to the task of silencing a known-harmless warning in one generator without silencing meaningful warnings in generators it calls. ijs

2 1

Re: [Python-Dev] Please reconsider PEP 479.
by Isaac Schwabacher 24 Nov '14

24 Nov '14

On 11/23/14, Guido van Rossum wrote: > It wouldn't be so bad if we had the occasional generator author writing "raise StopIteration" instead of "return" to exit from a generator. (We could just add a recommendation against this to the style guide.) But the problem is that an unguarded next() call also raises StopIteration. Sometimes this is intentional (as in some itertools examples). But sometimes an unguarded next() call occurs deep in the bowels of some code called by the generator, and this situation is often hard to debug, since there is no stack track. I'll admit I've only skimmed the massive volume of correspondence this PEP has generated, but it seems to me that this is the main argument for this change. I can only assume that your support for this PEP is informed by your experience building Tulip, but isn't this the kind of thing that can be accomplished with a warning? Then you can get the same behavior without even needing a __future__ import to protect code bases that expect StopIteration to propagate (which seems like the more elegant and natural thing to do, even if it is more error-prone). ijs

2 1

Support for Linux perf
by Francis Giraldeau 24 Nov '14

24 Nov '14

Hi, The PEP-418 is about performance counters, but there is no mention of performance management unit (PMU) counters, such as cache misses and instruction counts. The Linux perf tool aims at recording these samples at the system level. I ran linux perf on CPython for profiling. The resulting callstack is inside libpython.so, mostly recursive calls to PyEval_EvalFrameEx(), because the tool works at the ELF level. Here is an example with a dummy program (linux-tools on Ubuntu 14.04): $ perf record python crunch.py $ perf report --stdio # Overhead Command Shared Object Symbol # ........ ....... .................. ................................ # 32.37% python python2.7 [.] PyEval_EvalFrameEx 13.70% python libm-2.19.so [.] __sin_avx 5.25% python python2.7 [.] binary_op1.5010 4.82% python python2.7 [.] PyObject_GetAttr While this may be insightful for the interpreter developers, it it not so for the average Python developer. The report should display Python code instead. It seems obvious, still I haven't found the feature for that. When a performance counter reaches a given value, a sample is recorded. The most basic sample only records a timestamps, thread ID and the program counter (%rip). In addition, all executable memory maps of libraries are recorded. For the callstack, frame pointers are traversed, but most of the time, they are optimized on x86, so there is a fall back to unwind, which requires saving register values and a chunk of the stack. The memory space of the process is reconstructed offline. CPython seems to allocates code and frames on mmap() pages. If the data is outside about 1k from the top of stack, it is not available offline in the trace. We need some way to reconstitute this memory space of the interpreter to resolve the symbols, probably by dumping the data on disk. In Java, there is a small HotSpot agent that spits out the symbols of JIT code: https://github.com/jrudolph/perf-map-agent The problem is that CPython does not JIT code, and executed code is the ELF library itself. The executed frames are parameters of functions of the interpreter. I don't think the same approach can be used (maybe this can be applied to PyPy?). I looked at how Python frames are handled in GDB (file cpython/Tools/gdb/libpython.py). A python frame is detected in Frame(gdbframe).is_evalframeex() by a C call to PyEval_EvalFrameEx(). However, the traceback accesses PyFrameObject on the heap (at least for f->f_back = 0xa57460), which is possible in GDB when the program is paused and the whole memory space is available, but is not recorded for offline use in perf. Here is an example of callstack from GDB: #0 PyEval_EvalFrameEx (f=Frame 0x7ffff7f1b060, for file crunch.py, line 7, in bar (num=466829), throwflag=0) at ../Python/ceval.c:1039 #1 0x0000000000527877 in fast_function (func=<function at remote 0x7ffff6ec45a0>, pp_stack=0x7fffffffd280, n=1, na=1, nk=0) at ../Python/ceval.c:4106 #2 0x0000000000527582 in call_function (pp_stack=0x7fffffffd280, oparg=1) at ../Python/ceval.c:4041 We could add a kernel module that "knows" how to make samples of CPython, but it means python structures becomes sort of ABI, and kernel devs won't allow a python interpreter in kernel mode ;-). What we really want is f_code data and related objects: (gdb) print (void *)(f->f_code) $8 = (void *) 0x7ffff7e370f0 Maybe we could save these pages every time some code is loaded from the interpreter? (the memory range is about 1.7MB, but ) Anyway, I think we must change CPython to support tools such as perf. Any thoughts? Cheers, Francis

3 3

2.7.9 delayed
by Benjamin Peterson 23 Nov '14

23 Nov '14

As you may have noticed, 2.7.9rc1 wasn't tagged today. I've been much busier than I expected to be, so I haven't had the time to fix up some 2.7.9 lose ends. Hopefully, the delay won't be more than a few days.

1 0