From tomirendo at gmail.com  Sat Nov  1 15:56:12 2014
From: tomirendo at gmail.com (yotam vaknin)
Date: Sat, 1 Nov 2014 16:56:12 +0200
Subject: [Python-ideas] Change how Generator Expressions handle StopIteration
Message-ID: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>

Hi,

I would like to purpose that generator expressions will not catch
StopIteration exception, if this exception did not come from the iterated
object's __next__ function specifically. So generator expressions will be
able to raise StopIteration by calculating the current value of the
Generator.

Here is an example of a use-case :

def izip(*args):
    iters = [iter(obj) for obj in args]
    while True:
        yield tuple(next(it) for it in iters)

a = izip([1,2],[3,4])
print(next(a),next(a),next(a)) #Currently prints : (1, 3) (2, 4) ()
list(izip([1,2],[3,4])) #Currently never returns

Even thought this is the PEP described behaviour, I think this is an
unwanted behaviour.

I think Generator Expressions should work like List Comprehension in that
sense:

def iizip(*args):
    iters = [iter(obj) for obj in args]
    while True:
        yield tuple([next(it) for it in iters])

tuple(iizip([1,2],[3,4])) #Returns [(1, 3), (2, 4)]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141101/a845c53d/attachment.html>

From guido at python.org  Sat Nov  1 17:50:05 2014
From: guido at python.org (Guido van Rossum)
Date: Sat, 1 Nov 2014 09:50:05 -0700
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
Message-ID: <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>

I think you're on to something. But I think both your examples have a
problem, even though your second one "works".

If we weren't forced by backward compatibility I would have made it much
harder for StopIteration to "leak out". Currently a generator can either
return or raise StopIteration to signal it is done, but I think it would
have been better if StopIteration was treated as some kind of error in this
case. Basically I think any time a StopIteration isn't caught by a for-loop
or an explicit try/except StopIteraton, I feel there is a bug in the
program, or at least it is hard to debug.

I'm afraid that ship has sailed, though...

On Sat, Nov 1, 2014 at 7:56 AM, yotam vaknin <tomirendo at gmail.com> wrote:

> Hi,
>
> I would like to purpose that generator expressions will not catch
> StopIteration exception, if this exception did not come from the iterated
> object's __next__ function specifically. So generator expressions will be
> able to raise StopIteration by calculating the current value of the
> Generator.
>
> Here is an example of a use-case :
>
> def izip(*args):
>     iters = [iter(obj) for obj in args]
>     while True:
>         yield tuple(next(it) for it in iters)
>
> a = izip([1,2],[3,4])
> print(next(a),next(a),next(a)) #Currently prints : (1, 3) (2, 4) ()
> list(izip([1,2],[3,4])) #Currently never returns
>
> Even thought this is the PEP described behaviour, I think this is an
> unwanted behaviour.
>
> I think Generator Expressions should work like List Comprehension in that
> sense:
>
> def iizip(*args):
>     iters = [iter(obj) for obj in args]
>     while True:
>         yield tuple([next(it) for it in iters])
>
> tuple(iizip([1,2],[3,4])) #Returns [(1, 3), (2, 4)]
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141101/25f567d9/attachment.html>

From ethan at stoneleaf.us  Sat Nov  1 21:02:19 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 01 Nov 2014 13:02:19 -0700
Subject: [Python-ideas] Deprecate dunder functions from operator module
In-Reply-To: <54519687.6030708@stoneleaf.us>
References: <20141030005321.GC26866@ando.pearwood.info>
 <54519687.6030708@stoneleaf.us>
Message-ID: <54553C4B.1010404@stoneleaf.us>

On 10/29/2014 06:38 PM, Ethan Furman wrote:
> On 10/29/2014 05:53 PM, Steven D'Aprano wrote:
>>
>> I propose a few things:
>>
>> * institute a policy that, in the event of a new function being added
>>    to the operator module, only the dunderless version will be added;
>>
>> * change the documentation to make it clear that the dunderless
>>    versions should be used, rather than merely being "convenience"
>>    functions;
>>
>> * add a prominent note that the dunder versions exist for backwards
>>    compatibility only and should not be used in new code.
>
> +1

Actually, make that -1.

I'm just crafting some tests to explore how NotImplemented impacts various classes, and the dunder versions make the 
whole thing much nicer.

--
~Ethan~

From tjreedy at udel.edu  Sun Nov  2 00:07:08 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 01 Nov 2014 19:07:08 -0400
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
Message-ID: <m33p34$6k3$1@ger.gmane.org>

On 11/1/2014 12:50 PM, Guido van Rossum wrote:
> I think you're on to something. But I think both your examples have a
> problem, even though your second one "works".

Both versions are buggy in that iizip() yields () infinitely, while 
zip() yields nothing.  Fixes below.

> If we weren't forced by backward compatibility I would have made it much
> harder for StopIteration to "leak out". Currently a generator can either
> return or raise StopIteration to signal it is done, but I think it would
> have been better if StopIteration was treated as some kind of error in
> this case.

This would require some sort of additional special casing of 
StopIteration that we do not have now.  Currently, it is limited to 
'for' loops expecting and catching StopIteration as a signal to stop 
iterating.  That is rather easy to understand.

 > Basically I think any time a StopIteration isn't caught by a
> for-loop or an explicit try/except StopIteraton, I feel there is a bug
> in the program,

Outside of generator functions (and expressions), I agree as I cannot 
think of an exception when it is not.  This has come up on Python list.

 > or at least it is hard to debug.

Code within generator functions is different.  Writing "raise 
StopIteration" instead of "return" is mostly a waste of keystrokes.  As 
for next(it), StopIteration should usually propagate, as with an 
explicit raise and not be caught.  The code below that 'works' (when it 
does work), works because the StopIteration from next(it) (when there is 
at least one) propagates to the list comp, which lets it pass to the 
generator, which lets it pass to the generator user.

> I'm afraid that ship has sailed, though...
>
> On Sat, Nov 1, 2014 at 7:56 AM, yotam vaknin
> <tomirendo at gmail.com
> <mailto:tomirendo at gmail.com>> wrote:

>     I would like to purpose that generator expressions will not catch
>     StopIteration exception, if this exception did not come from the
>     iterated object's __next__ function specifically.

For the purpose of your example, all instances of StopIteration are the 
same and might as well be the same instance.

Since to my understanding generators and g.e.s already do not catch the 
StopIterations you say you want not caught, and since you need for it to 
not be caught in the code below, I do not understand exactly what you 
propose.

 >     So generator
>     expressions will be able to raise StopIteration by calculating the
>     current value of the Generator.

I cannot understand this.

>     def izip(*args):
>          iters = [iter(obj) for obj in args]
>          while True:
>              yield tuple(next(it) for it in iters)
>
>     a = izip([1,2],[3,4])
>     print(next(a),next(a),next(a)) #Currently prints : (1, 3) (2, 4) ()
>     list(izip([1,2],[3,4])) #Currently never returns

Better test code that avoid infinite looping:

a = izip([1,2],[3,4])
for i in range(3):
     print(next(a))

One the third loop, the above prints (), while the below prints a 
traceback.  With a = izip(), both print () 3 times.

The problem is that when next(it) raises, you want the StopIteration 
instance propagated (not immediately caught), so that the 
generator-using code knows that the generator is exhausted.  But the 
tuple call catches it first, so that, in combination with 'while True', 
the user never sees StopIteration  A partial solution is to provoke 
StopIteration before calling tuple, so that it does propagate.  That is 
what the list comp below does.  But if args is empty, so is iters, and 
there is no next(it) to ever raise.  For a complete solution that 
imitates zip and does not require an otherwise useless temporary list, 
replace the loop with this:

     while True:
         t = tuple(next(it) for it in iters)
         if not t:
            return
         yield t

>     Even thought this is the PEP described behaviour, I think this is an
>     unwanted behaviour.

Not if you think carefully about what you want to happen when next(it) 
raises.  I think generators and generators expressions should be left 
alone.

>     I think Generator Expressions should work like List Comprehension in
>     that sense:
>
>     def iizip(*args):
>          iters = [iter(obj) for obj in args]
>          while True:
>              yield tuple([next(it) for it in iters])

This could be fixed with 'if not iters: return' as the second line.

Replacing [genexp] with list(genexp) does not work because the latter, 
unlike the former, catches StopIteration.  This is proof that the two 
are not exactly equivalent, and the such behavior difference I know of 
(excluding introspection, such as with trace).

-- 
Terry Jan Reedy


From ncoghlan at gmail.com  Sun Nov  2 10:53:06 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 2 Nov 2014 19:53:06 +1000
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
Message-ID: <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>

On 2 November 2014 02:50, Guido van Rossum <guido at python.org> wrote:
> I think you're on to something. But I think both your examples have a
> problem, even though your second one "works".
>
> If we weren't forced by backward compatibility I would have made it much
> harder for StopIteration to "leak out". Currently a generator can either
> return or raise StopIteration to signal it is done, but I think it would
> have been better if StopIteration was treated as some kind of error in this
> case. Basically I think any time a StopIteration isn't caught by a for-loop
> or an explicit try/except StopIteraton, I feel there is a bug in the
> program, or at least it is hard to debug.
>
> I'm afraid that ship has sailed, though...

The closest existing example of this kind of generator instance
specific StopIteration handling that I can think of is the special
case handling of StopIteration in
contexlib._GeneratorContextManager.__exit__()
(https://hg.python.org/cpython/file/3.4/Lib/contextlib.py#l63). There,
the exception handling differentiates between "a specific
StopIteration instance that we just threw into the subgenerator"
(which it will allow to propagate) and "any other StopIteration
instance, which indicates that the wrapped generator iterator
terminated as expected" (which it will suppress). We had that wrong
initially - if I recall correctly, it was PJE that noticed the problem
before 2.5 was released. However, the only reason we were able to make
it work is that we knew the exact identity of the exception we were
throwing in, rather than just its type - we don't have that luxury in
the general case.

Getting back to the behaviour that prompted the thread, like a lot of
exception handling quirks, it gets back to being very careful about
the scope of exception handlers. In this case, the "next(it)" call is
inside a generator expression, and hence inside the scope of the
expression's StopIteration handling. By contrast, the comprehension
version doesn't *have* any implicit exception handling, so the
StopIteration escapes to terminate the containing generator.

In terms of changing the behaviour of generator expressions to allow
other StopIteration instances to propagate, I believe I do see one
possible way to do it that limits the degree of backwards
incompatibility.

Firstly, you'd need to add a general purpose capability to generator iterators:

    def _set_default_exception(exc):
        """Supply a specific StopIteration instance to raise when the
generator frame returns None"""
        ...

Normally, when the generator execution falls off the end of the frame
by returning, the interpreter raises StopIteration if the result is
None, or StopIteration(result) if the result is not None. With the new
method, you could set a specific instance to be raised when the
underlying result of the frame is None. (Side note: "return" and
"raise StopIteration" in a generator function aren't *exactly* the
same, as only the former relies on the return->raise conversion
supplied by the surrounding generator iterator object)

That part would be entirely backwards compatible, and would allow you
to distinguish whether calling "next", "send" or "throw" on any
generator threw StopIteration because the underlying frame returned
None (by checking if the StopIteration instance was the one you
configured to be raised on a None result), or because it either
returned a non-None value or else something running inside that frame
threw StopIteration.

The backwards incompatible part would be to then also change generator
expressions to set a specific StopIteration instance to be raised when
the underlying frame returned, and allow all other StopIteration
instances to escape, just as
contextlib._GeneratorContextManager.__exit__ allows StopIteration
instances thrown from the body of the with statement to escape.

I think the end result of such a change would definitely be less
surprising, as it would make generator expressions behave more like
the corresponding comprehensions, and eliminate a hidden infinite loop
bug. However, I'm not sure if it's *sufficiently* less surprising to
be worth changing - especially since it would mean incurring a small
amount of additional runtime overhead for each generator expression.

Regards,
Nick.

P.S. As additional background on the current difference in behaviour
between list comprehensions and generator expressions, that has its
roots in the same idiosyncrasy where putting a yield expression inside
a comprehension actually *turns it into a generator expression*.
Comprehensions are full closures, but they don't contain a yield
expression, so you get a normal function, which the interpreter then
calls.

The interpreter doesn't actually do anything particularly special to
make a generator expression instead - it just implicitly inserts a
yield expression into the closure, which then automatically makes it a
generator function instead of a normal one.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Nov  2 11:15:06 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 2 Nov 2014 20:15:06 +1000
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
Message-ID: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>

On 2 November 2014 19:53, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 2 November 2014 02:50, Guido van Rossum <guido at python.org> wrote:
>> I think you're on to something. But I think both your examples have a
>> problem, even though your second one "works".
>>
>> If we weren't forced by backward compatibility I would have made it much
>> harder for StopIteration to "leak out". Currently a generator can either
>> return or raise StopIteration to signal it is done, but I think it would
>> have been better if StopIteration was treated as some kind of error in this
>> case. Basically I think any time a StopIteration isn't caught by a for-loop
>> or an explicit try/except StopIteraton, I feel there is a bug in the
>> program, or at least it is hard to debug.
>>
>> I'm afraid that ship has sailed, though...
>
> The closest existing example of this kind of generator instance
> specific StopIteration handling that I can think of is the special
> case handling of StopIteration in
> contexlib._GeneratorContextManager.__exit__()
> (https://hg.python.org/cpython/file/3.4/Lib/contextlib.py#l63). There,
> the exception handling differentiates between "a specific
> StopIteration instance that we just threw into the subgenerator"
> (which it will allow to propagate) and "any other StopIteration
> instance, which indicates that the wrapped generator iterator
> terminated as expected" (which it will suppress). We had that wrong
> initially - if I recall correctly, it was PJE that noticed the problem
> before 2.5 was released. However, the only reason we were able to make
> it work is that we knew the exact identity of the exception we were
> throwing in, rather than just its type - we don't have that luxury in
> the general case.
>
> Getting back to the behaviour that prompted the thread, like a lot of
> exception handling quirks, it gets back to being very careful about
> the scope of exception handlers. In this case, the "next(it)" call is
> inside a generator expression, and hence inside the scope of the
> expression's StopIteration handling. By contrast, the comprehension
> version doesn't *have* any implicit exception handling, so the
> StopIteration escapes to terminate the containing generator.

Bah, I should have fully read Terry's reply before responding. He's
right, it's the tuple call that's suppressing the exception, not the
generator expression itself.

That changes the possible solution, by tweaking it to be an optional
extension to the iterator protocol, allowing iterators to make the
terminating exception configurable.

    def __setiterexc__(exc):
        """Specify the exception instance to raise when the iterator
is exhausted"""
        ...

Iterator consumers (like tuple) could then check for that method and
use it to set a specific StopIteration instance, allowing all others
to escape.

I believe actually doing this would be adding too much complexity for
too little gain, but it *is* possible.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From abarnert at yahoo.com  Sun Nov  2 20:50:01 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 2 Nov 2014 11:50:01 -0800
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
Message-ID: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com>

This is related to the fact that, although the docs imply otherwise, [COMP] isn't exactly equivalent to list(COMP), because of cases like:

    def ensure_positive(x):
        if x<=0: raise StopIteration
        return True
    eggs = list(x for x in spam if raise_on_negative(x))

    def ensure_positive(x);
        if x<=0: raise StopIteration
        return x
    eggs = list(ensure_positive(x) for x in spam)

In both cases, this acts like a "takewhile": eggs ends up as a list of the initial positive values, and the first non-positive value is consumed and discarded. But if you do the same thing with a list comprehension, the comprehension is aborted by the StopIteration, and eggs never gets set (although the same values are consumed from spam, of course).

IIRC, you asked me what the performance costs would be of changing listcomps to match, and for a trivial comp it worked out to be about 40% for the naive solution (build a genexpr, call it, call list) and about 20% with specially-optimized bytecode. So everyone agreed that even if this is a bug, that would be too much of a cost for too small of a fix.

Of course here the issue is almost the opposite. But they're clearly related; how different comprehensions "leak" exceptions differs.

Sent from a random iPhone

On Nov 2, 2014, at 2:15, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 2 November 2014 19:53, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On 2 November 2014 02:50, Guido van Rossum <guido at python.org> wrote:
>>> I think you're on to something. But I think both your examples have a
>>> problem, even though your second one "works".
>>> 
>>> If we weren't forced by backward compatibility I would have made it much
>>> harder for StopIteration to "leak out". Currently a generator can either
>>> return or raise StopIteration to signal it is done, but I think it would
>>> have been better if StopIteration was treated as some kind of error in this
>>> case. Basically I think any time a StopIteration isn't caught by a for-loop
>>> or an explicit try/except StopIteraton, I feel there is a bug in the
>>> program, or at least it is hard to debug.
>>> 
>>> I'm afraid that ship has sailed, though...
>> 
>> The closest existing example of this kind of generator instance
>> specific StopIteration handling that I can think of is the special
>> case handling of StopIteration in
>> contexlib._GeneratorContextManager.__exit__()
>> (https://hg.python.org/cpython/file/3.4/Lib/contextlib.py#l63). There,
>> the exception handling differentiates between "a specific
>> StopIteration instance that we just threw into the subgenerator"
>> (which it will allow to propagate) and "any other StopIteration
>> instance, which indicates that the wrapped generator iterator
>> terminated as expected" (which it will suppress). We had that wrong
>> initially - if I recall correctly, it was PJE that noticed the problem
>> before 2.5 was released. However, the only reason we were able to make
>> it work is that we knew the exact identity of the exception we were
>> throwing in, rather than just its type - we don't have that luxury in
>> the general case.
>> 
>> Getting back to the behaviour that prompted the thread, like a lot of
>> exception handling quirks, it gets back to being very careful about
>> the scope of exception handlers. In this case, the "next(it)" call is
>> inside a generator expression, and hence inside the scope of the
>> expression's StopIteration handling. By contrast, the comprehension
>> version doesn't *have* any implicit exception handling, so the
>> StopIteration escapes to terminate the containing generator.
> 
> Bah, I should have fully read Terry's reply before responding. He's
> right, it's the tuple call that's suppressing the exception, not the
> generator expression itself.
> 
> That changes the possible solution, by tweaking it to be an optional
> extension to the iterator protocol, allowing iterators to make the
> terminating exception configurable.
> 
>    def __setiterexc__(exc):
>        """Specify the exception instance to raise when the iterator
> is exhausted"""
>        ...
> 
> Iterator consumers (like tuple) could then check for that method and
> use it to set a specific StopIteration instance, allowing all others
> to escape.
> 
> I believe actually doing this would be adding too much complexity for
> too little gain, but it *is* possible.
> 
> Cheers,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From tjreedy at udel.edu  Sun Nov  2 22:00:46 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 02 Nov 2014 16:00:46 -0500
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com>
Message-ID: <m3661u$dc8$1@ger.gmane.org>

On 11/2/2014 2:50 PM, Andrew Barnert wrote:
> This is related to the fact that, although the docs imply otherwise,
> [COMP] isn't exactly equivalent to list(COMP),

That purported equivalence is a common meme, which I may have helped 
spread.  If it is implied in the doc, it should be changed.  Before I 
answered on this thread yesterday, I looked for such an implication in 
the Language Reference (though not the Tutorial), and only found this 
carefully written description in the expressions chapter.

"In this case, the elements of the new container are those that would be 
produced by considering each of the for or if clauses a block, nesting 
from left to right, and evaluating the expression to produce an element 
each time the innermost block is reached. Note that the comprehension is 
executed in a separate scope,"

IE, a comprehension is equivalent to the result of for and if statements 
in a separate Python function that first initializes a collection object 
(list, set, or dict) and augments the collection in the innermost scope 
with the element produced (which is a key,value pair for dicts).

Such an equivalent function would not catch any exception raised by the 
innermost element expression.  On the other hand, collection 
initializers, such as list or tuple, specifically catch StopIteration 
when fed an iterable. Hence the two cannot be equivalent when the 
element expression raises StopIteration.

-- 
Terry Jan Reedy


From abarnert at yahoo.com  Sun Nov  2 23:37:57 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 2 Nov 2014 14:37:57 -0800
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <m3661u$dc8$1@ger.gmane.org>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
Message-ID: <3196CD31-F077-4A0F-990D-606F0592EB9A@yahoo.com>

On Nov 2, 2014, at 13:00, Terry Reedy <tjreedy at udel.edu> wrote:

> On 11/2/2014 2:50 PM, Andrew Barnert wrote:
>> This is related to the fact that, although the docs imply otherwise,
>> [COMP] isn't exactly equivalent to list(COMP),
> 
> That purported equivalence is a common meme, which I may have helped spread. If it is implied in the doc, it should be changed.  Before I answered on this thread yesterday, I looked for such an implication in the Language Reference (though not the Tutorial), and only found this carefully written description in the expressions chapter.

We looked through this last year and decided nothing needed to be changed in the docs, so I doubt it's worth repeating that effort. 

IIRC, the tutorial may have been confusing in the past but wasn't as of 3.3, and the only place that might confuse anyone was the what's new in 3.0, which is generally only changed to add notes about things which were un-/re-changed (like u"" strings). But again, even if I'm remembering wrong, I don't think it matters.

All that being said, I don't really love the reference docs here. Neither 6.2.8 not anything else explicitly says what the semantics are. It's a pretty obvious guess that the syntax is interpreted the same as for comprehensions (as in 6.2.4), and that the values yielded are those that would be used to produce elements. But the docs don't actually say that.

> "In this case, the elements of the new container are those that would be produced by considering each of the for or if clauses a block, nesting from left to right, and evaluating the expression to produce an element each time the innermost block is reached. Note that the comprehension is executed in a separate scope,"
> 
> IE, a comprehension is equivalent to the result of for and if statements in a separate Python function that first initializes a collection object (list, set, or dict) and augments the collection in the innermost scope with the element produced (which is a key,value pair for dicts).
> 
> Such an equivalent function would not catch any exception raised by the innermost element expression.  On the other hand, collection initializers, such as list or tuple, specifically catch StopIteration when fed an iterable. Hence the two cannot be equivalent when the element expression raises StopIteration.
> 
> -- 
> Terry Jan Reedy
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From guido at python.org  Mon Nov  3 04:01:05 2014
From: guido at python.org (Guido van Rossum)
Date: Sun, 2 Nov 2014 19:01:05 -0800
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <m3661u$dc8$1@ger.gmane.org>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
Message-ID: <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>

On Sun, Nov 2, 2014 at 1:00 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 11/2/2014 2:50 PM, Andrew Barnert wrote:
>
>> This is related to the fact that, although the docs imply otherwise,
>> [COMP] isn't exactly equivalent to list(COMP),
>>
>
> That purported equivalence is a common meme, which I may have helped
> spread.


I may have started it. I was aware of the non-equivalence (only
mostly-equivalence) in Python 2 and I wanted to make then identical in
Python 3 -- having one construct being exactly equivalent to another reduce
the amount of explaining needed. Unfortunately, people had started to
depend on the (in my *current* opinion deplorable) behavior of generator
expressions in the face of StopIteration thrown by arbitrary parts of the
expression or condition, and the equivalence is still imperfect. At least
the variable leakage has been fixed.

I know that when we first introduced generators (not generator expressions)
I was in favor of interpreting arbitrary things that raise StopIteration in
a generator to cause the generator to terminate just as if it had decided
to stop (i.e. 'return' or falling of the end), because I thought there were
some useful patterns that could be written more compactly this way -- in
particular, the pattern where a generator iterates over another iterator by
calling next() on it, does some processing on the value thus produced, and
then yielding the processed value (or not), and where the logical response
to a StopIteration from the inner iterator is to exit the generator. For
example:

  def only_positive(it):
      while True:
          x = next(it)
          if x > 0: yield x

This *particular* example is much better written as:

  def only_positive(x):
      for x in it:
          if x > 0: yield x

but the idea was that there might be variants where being constrained by a
single for-loop would make the code less elegant if you had to catch the
StopIteration and explicitly exit the generator.

However, I don't think this idea has panned out. I haven't done a survey,
but I have a feeling that in most cases where an explicit next() call is
used (as opposed to a for-loop) there's a try/except Stopiteration around
it, and a fair amount if time is wasted debugging situations where a
StopIteration unexpectedly escapes and silently interrupts some loop over
an unrelated generator (instead of loudly bubbling up to the top and
causing a traceback, which would be more debuggable). And the use case of
raising StopIteration from a condition used in a generator expression is
iffy at best (it makes the condition function hard to use in other
contexts, and it calls to attention the difference between generators and
comprehensions).

So I will go out on a limb here and say that this was a mistake and if we
can think of easing the transitional pain it would be a good thing to fix
this eventually.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141102/934b2e4a/attachment-0001.html>

From ncoghlan at gmail.com  Mon Nov  3 05:02:25 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 3 Nov 2014 14:02:25 +1000
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com>
 <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
Message-ID: <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>

On 3 November 2014 13:01, Guido van Rossum <guido at python.org> wrote:
> On Sun, Nov 2, 2014 at 1:00 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>>
>> On 11/2/2014 2:50 PM, Andrew Barnert wrote:
>>>
>>> This is related to the fact that, although the docs imply otherwise,
>>> [COMP] isn't exactly equivalent to list(COMP),
>>
>>
>> That purported equivalence is a common meme, which I may have helped
>> spread.
>
>
> I may have started it. I was aware of the non-equivalence (only
> mostly-equivalence) in Python 2 and I wanted to make then identical in
> Python 3 -- having one construct being exactly equivalent to another reduce
> the amount of explaining needed. Unfortunately, people had started to depend
> on the (in my *current* opinion deplorable) behavior of generator
> expressions in the face of StopIteration thrown by arbitrary parts of the
> expression or condition, and the equivalence is still imperfect. At least
> the variable leakage has been fixed.

I think I'm guilty as well - when I was working on the Python 3
changes, getting the *scoping* behaviour to be identical between
comprehensions and generator expressions was one of the key
objectives, so I regularly described it as making "[x for x in seq]"
equivalent to "list(x for x in seq)".

I unfortunately didn't notice the remaining exception handling
differences at the time, or we might have been able to do something
about it for 3.0 :(

> However, I don't think this idea has panned out. I haven't done a survey,
> but I have a feeling that in most cases where an explicit next() call is
> used (as opposed to a for-loop) there's a try/except Stopiteration around
> it, and a fair amount if time is wasted debugging situations where a
> StopIteration unexpectedly escapes and silently interrupts some loop over an
> unrelated generator (instead of loudly bubbling up to the top and causing a
> traceback, which would be more debuggable). And the use case of raising
> StopIteration from a condition used in a generator expression is iffy at
> best (it makes the condition function hard to use in other contexts, and it
> calls to attention the difference between generators and comprehensions).
>
> So I will go out on a limb here and say that this was a mistake and if we
> can think of easing the transitional pain it would be a good thing to fix
> this eventually.

Having had to do the dance to work around the current behaviour in
contextlib, I'm inclined to agree - there's a significant semantic
difference between the "this iterable just terminated" StopIteration,
and the "something threw StopIteration and nobody caught it", and the
current model hides it.

However, I also see potentially significant backwards compatibility
problems when it comes to helper functions that throw StopIteration to
terminate the calling generator - there would likely need to be some
kind of thread local state and a helper "iterexit()" builtin and
"PyIter_Exit()" C API to call instead of raising StopIteration
directly.

Making such a change would involve a lot of code churn just to phase
out a relatively obscure issue that mostly just makes certain bugs
harder to diagnose (as was the case with the generator expression
based izip implementation in this thread), rather than directly
causing bugs in its own right.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From guido at python.org  Mon Nov  3 06:59:18 2014
From: guido at python.org (Guido van Rossum)
Date: Sun, 2 Nov 2014 21:59:18 -0800
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
Message-ID: <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>

On Sun, Nov 2, 2014 at 8:02 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 3 November 2014 13:01, Guido van Rossum <guido at python.org> wrote:
> > On Sun, Nov 2, 2014 at 1:00 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> >>
> >> On 11/2/2014 2:50 PM, Andrew Barnert wrote:
> >>>
> >>> This is related to the fact that, although the docs imply otherwise,
> >>> [COMP] isn't exactly equivalent to list(COMP),
> >>
> >>
> >> That purported equivalence is a common meme, which I may have helped
> >> spread.
> >
> >
> > I may have started it. I was aware of the non-equivalence (only
> > mostly-equivalence) in Python 2 and I wanted to make then identical in
> > Python 3 -- having one construct being exactly equivalent to another
> reduce
> > the amount of explaining needed. Unfortunately, people had started to
> depend
> > on the (in my *current* opinion deplorable) behavior of generator
> > expressions in the face of StopIteration thrown by arbitrary parts of the
> > expression or condition, and the equivalence is still imperfect. At least
> > the variable leakage has been fixed.
>
> I think I'm guilty as well - when I was working on the Python 3
> changes, getting the *scoping* behaviour to be identical between
> comprehensions and generator expressions was one of the key
> objectives, so I regularly described it as making "[x for x in seq]"
> equivalent to "list(x for x in seq)".
>
> I unfortunately didn't notice the remaining exception handling
> differences at the time, or we might have been able to do something
> about it for 3.0 :(
>
> > However, I don't think this idea has panned out. I haven't done a survey,
> > but I have a feeling that in most cases where an explicit next() call is
> > used (as opposed to a for-loop) there's a try/except Stopiteration around
> > it, and a fair amount if time is wasted debugging situations where a
> > StopIteration unexpectedly escapes and silently interrupts some loop
> over an
> > unrelated generator (instead of loudly bubbling up to the top and
> causing a
> > traceback, which would be more debuggable). And the use case of raising
> > StopIteration from a condition used in a generator expression is iffy at
> > best (it makes the condition function hard to use in other contexts, and
> it
> > calls to attention the difference between generators and comprehensions).
> >
> > So I will go out on a limb here and say that this was a mistake and if we
> > can think of easing the transitional pain it would be a good thing to fix
> > this eventually.
>
> Having had to do the dance to work around the current behaviour in
> contextlib, I'm inclined to agree - there's a significant semantic
> difference between the "this iterable just terminated" StopIteration,
> and the "something threw StopIteration and nobody caught it", and the
> current model hides it.
>
> However, I also see potentially significant backwards compatibility
> problems when it comes to helper functions that throw StopIteration to
> terminate the calling generator - there would likely need to be some
> kind of thread local state and a helper "iterexit()" builtin and
> "PyIter_Exit()" C API to call instead of raising StopIteration
> directly.
>

That's what I was afraid of. Can you point me to an example of code that
depends on this that isn't trivial like Andrew Barnert's ensure_positive()
example? I think that particular example, and the category it represents,
are excessive cleverness that abuse the feature under discussion -- but you
sound like you have helpers for context managers that couldn't be easily
dismissed like that.


> Making such a change would involve a lot of code churn just to phase
> out a relatively obscure issue that mostly just makes certain bugs
> harder to diagnose (as was the case with the generator expression
> based izip implementation in this thread), rather than directly
> causing bugs in its own right.
>

Maybe. But the real-life version of that bug can be *really* hard to find,
and that's usually the kind of thing we'd like to fix.

FWIW the implementation of my proposal is easy to describe (which the Zen
approves of): when a StopIteration leaves a frame, replace it with some
other exception (either a new, custom one or maybe just RuntimeError),
chaining the original StopIteration.

It's the consequences that are hard to survey and describe in this case (as
they affect subtle code depending on the current semantics).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141102/9cc8bdab/attachment.html>

From ncoghlan at gmail.com  Mon Nov  3 08:24:01 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 3 Nov 2014 17:24:01 +1000
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com>
 <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
Message-ID: <CADiSq7ctPkzQAg_BPXoCmW8t2413Twstmsv=AQ9wbeWEF6e-RQ@mail.gmail.com>

On 3 November 2014 15:59, Guido van Rossum <guido at python.org> wrote:
> On Sun, Nov 2, 2014 at 8:02 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> However, I also see potentially significant backwards compatibility
>> problems when it comes to helper functions that throw StopIteration to
>> terminate the calling generator - there would likely need to be some
>> kind of thread local state and a helper "iterexit()" builtin and
>> "PyIter_Exit()" C API to call instead of raising StopIteration
>> directly.
>
>
> That's what I was afraid of. Can you point me to an example of code that
> depends on this that isn't trivial like Andrew Barnert's ensure_positive()
> example? I think that particular example, and the category it represents,
> are excessive cleverness that abuse the feature under discussion -- but you
> sound like you have helpers for context managers that couldn't be easily
> dismissed like that.

Sorry, I didn't mean to give that impression. I'm in a similar
situation to you in that regard - any specific examples I can think of
trip my "that's too obscure to be maintainable" alarm (if there is a
reasonable use case, I'd expect it to involve full generators, rather
than generator expressions).

The code in contextlib relies on the way *generators* handle
StopIteration, and if understand your proposal correctly, that would
remain unchanged - only ordinary function calls would convert
StopIteration to a different exception type, preserving the functional
equivalence of a generator returning, raising StopIteration, or having
StopIteration thrown into a yield point (it's that last one that
contextlib relies on).

>> Making such a change would involve a lot of code churn just to phase
>> out a relatively obscure issue that mostly just makes certain bugs
>> harder to diagnose (as was the case with the generator expression
>> based izip implementation in this thread), rather than directly
>> causing bugs in its own right.
>
>
> Maybe. But the real-life version of that bug can be *really* hard to find,
> and that's usually the kind of thing we'd like to fix.
>
> FWIW the implementation of my proposal is easy to describe (which the Zen
> approves of): when a StopIteration leaves a frame, replace it with some
> other exception (either a new, custom one or maybe just RuntimeError),
> chaining the original StopIteration.

That's far more elegant than the stateful possibilities I was considering.

So generators would continue to leave StopIteration untouched
(preserving the equivalence between returning from the frame and
explicitly raising StopIteration from the generator body), and only
ordinary function invocations would gain the StopIteration ->
UnexpectedStopIteration translation (assuming we went with a new
exception type)?

> It's the consequences that are hard to survey and describe in this case (as
> they affect subtle code depending on the current semantics).

Aye. I'm reasonably OK with the notion of breaking overly clever (and
hence hard to follow) generator expressions, but I'm a little more
nervous about any change that means that factoring out "raise
StopIteration" in a full generator function would stop working.

That said, such a change would bring generator functions fully into
line with the "flow control should be locally visible" principle that
guided both the with statement design and asyncio - only a local
return or raise statement could gracefully terminate the generator,
StopIteration from a nested function call would always escape as the
new exception type. If you wanted to factor out a helper function that
terminated the generator you'd have to do "return yield from helper()"
rather than just "helper()".

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From 4kir4.1i at gmail.com  Mon Nov  3 11:41:30 2014
From: 4kir4.1i at gmail.com (Akira Li)
Date: Mon, 03 Nov 2014 13:41:30 +0300
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com>
 <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
Message-ID: <87mw88wopx.fsf@gmail.com>

Guido van Rossum <guido at python.org> writes:

> On Sun, Nov 2, 2014 at 8:02 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> However, I also see potentially significant backwards compatibility
>> problems when it comes to helper functions that throw StopIteration to
>> terminate the calling generator - there would likely need to be some
>> kind of thread local state and a helper "iterexit()" builtin and
>> "PyIter_Exit()" C API to call instead of raising StopIteration
>> directly.
>>
>
> That's what I was afraid of. Can you point me to an example of code that
> depends on this that isn't trivial like Andrew Barnert's ensure_positive()
> example? I think that particular example, and the category it represents,
> are excessive cleverness that abuse the feature under discussion -- but you
> sound like you have helpers for context managers that couldn't be easily
> dismissed like that.
>
>

The pure Python implementation of itertools.groupby() provided in its
docs [1] uses next() as the helper function that terminates the calling
generator by raising StopIteration

[1]: https://docs.python.org/3/library/itertools.html#itertools.groupby

Here's a simplified example:

  from functools import partial
  
  def groupby(iterable):
      """
      >>> ' '.join(k for k, g in groupby('AAAABBBCCDAABBB'))
      'A B C D A B'
      >>> ' '.join(''.join(g) for k, g in groupby('AAAABBBCCDAABBB'))
      'AAAA BBB CC D AA BBB'
      """
      next_value = partial(next, iter(iterable))
      def yield_same(group_value_): # generate group values
          nonlocal value
          while value == group_value_:
              yield value
              value = next_value() # exit on StopIteration
  
      group_value = value = object()
      while True:
          while value == group_value: # discard unconsumed values
              value = next_value() # exit on StopIteration
  
          group_value = value
          yield group_value, yield_same(group_value)
  
The alternative is to return a sentinel from next():

  def groupby_done(iterable):
      done = object() # sentinel
      next_value = partial(next, iter(iterable), done)
      def yield_same(group_value_): # generate group values
          nonlocal value
          while value == group_value_:
              yield value
              value = next_value()
              if value is done:
                  return
  
      group_value = value = object()
      while value is not done:
          while value == group_value: # discard unconsumed values
              value = next_value()
              if value is done:
                  return
          group_value = value
          yield group_value, yield_same(group_value)
  
The first code example exploits the fact that `next(it)` continues to
raise StopIteration on subsequent calls.  The second code example has to
propagate up the stack the termination condition manually (`while True`
is replace with `while value is not done`).


--
Akira



From abarnert at yahoo.com  Mon Nov  3 20:58:28 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 3 Nov 2014 11:58:28 -0800
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
Message-ID: <C6498475-75A1-4D8D-AD94-AA0F6092FDE0@yahoo.com>

On Nov 2, 2014, at 21:59, Guido van Rossum <guido at python.org> wrote:

> On Sun, Nov 2, 2014 at 8:02 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On 3 November 2014 13:01, Guido van Rossum <guido at python.org> wrote:
>> > On Sun, Nov 2, 2014 at 1:00 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>> >>
>> >> On 11/2/2014 2:50 PM, Andrew Barnert wrote:
>> >>>
>> >>> This is related to the fact that, although the docs imply otherwise,
>> >>> [COMP] isn't exactly equivalent to list(COMP),
>> >>
>> >>
>> >> That purported equivalence is a common meme, which I may have helped
>> >> spread.
>> >
>> >
>> > I may have started it. I was aware of the non-equivalence (only
>> > mostly-equivalence) in Python 2 and I wanted to make then identical in
>> > Python 3 -- having one construct being exactly equivalent to another reduce
>> > the amount of explaining needed. Unfortunately, people had started to depend
>> > on the (in my *current* opinion deplorable) behavior of generator
>> > expressions in the face of StopIteration thrown by arbitrary parts of the
>> > expression or condition, and the equivalence is still imperfect. At least
>> > the variable leakage has been fixed.
>> 
>> I think I'm guilty as well - when I was working on the Python 3
>> changes, getting the *scoping* behaviour to be identical between
>> comprehensions and generator expressions was one of the key
>> objectives, so I regularly described it as making "[x for x in seq]"
>> equivalent to "list(x for x in seq)".
>> 
>> I unfortunately didn't notice the remaining exception handling
>> differences at the time, or we might have been able to do something
>> about it for 3.0 :(
>> 
>> > However, I don't think this idea has panned out. I haven't done a survey,
>> > but I have a feeling that in most cases where an explicit next() call is
>> > used (as opposed to a for-loop) there's a try/except Stopiteration around
>> > it, and a fair amount if time is wasted debugging situations where a
>> > StopIteration unexpectedly escapes and silently interrupts some loop over an
>> > unrelated generator (instead of loudly bubbling up to the top and causing a
>> > traceback, which would be more debuggable). And the use case of raising
>> > StopIteration from a condition used in a generator expression is iffy at
>> > best (it makes the condition function hard to use in other contexts, and it
>> > calls to attention the difference between generators and comprehensions).
>> >
>> > So I will go out on a limb here and say that this was a mistake and if we
>> > can think of easing the transitional pain it would be a good thing to fix
>> > this eventually.
>> 
>> Having had to do the dance to work around the current behaviour in
>> contextlib, I'm inclined to agree - there's a significant semantic
>> difference between the "this iterable just terminated" StopIteration,
>> and the "something threw StopIteration and nobody caught it", and the
>> current model hides it.
>> 
>> However, I also see potentially significant backwards compatibility
>> problems when it comes to helper functions that throw StopIteration to
>> terminate the calling generator - there would likely need to be some
>> kind of thread local state and a helper "iterexit()" builtin and
>> "PyIter_Exit()" C API to call instead of raising StopIteration
>> directly.
> 
> That's what I was afraid of. Can you point me to an example of code that depends on this that isn't trivial like Andrew Barnert's ensure_positive() example? I think that particular example, and the category it represents, are excessive cleverness that abuse the feature under discussion

The category is usually represented by the even more trivial and even more abusive example:

    (prime if prime<100 else throw(StopIteration) for prime in primes)

I do see these, mostly on places like StackOverflow, where someone was shown this "cool trick" by someone else, used it without understanding it, and now has no idea how to debug his code. (However, a few people on this list suggested it as an alternative to adding some kind of "syntactic takewhile" to the language, so it's possible not everyone sees it as abuse, even though I think you and others called it abuse back then as well.)

Anyway, I agree that explicitly disallowing it would make the language simpler, eliminate more bugs than useful idioms, and possibly open the door to other improvements. But if you can't justify this abuse as being actually illegal by a reading of the docs in 3.0-3.4, and people are potentially using it in real code, wouldn't that require a period of deprecation before it can be broken?

> -- but you sound like you have helpers for context managers that couldn't be easily dismissed like that.
>  
>> Making such a change would involve a lot of code churn just to phase
>> out a relatively obscure issue that mostly just makes certain bugs
>> harder to diagnose (as was the case with the generator expression
>> based izip implementation in this thread), rather than directly
>> causing bugs in its own right.
> 
> Maybe. But the real-life version of that bug can be *really* hard to find, and that's usually the kind of thing we'd like to fix.
> 
> FWIW the implementation of my proposal is easy to describe (which the Zen approves of): when a StopIteration leaves a frame, replace it with some other exception (either a new, custom one or maybe just RuntimeError), chaining the original StopIteration.
> 
> It's the consequences that are hard to survey and describe in this case (as they affect subtle code depending on the current semantics).
> 
> -- 
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141103/5d4624b0/attachment.html>

From antony.lee at berkeley.edu  Mon Nov  3 22:38:43 2014
From: antony.lee at berkeley.edu (Antony Lee)
Date: Mon, 3 Nov 2014 13:38:43 -0800
Subject: [Python-ideas] "or raise" syntax
In-Reply-To: <m2u0vr$nos$1@ger.gmane.org>
References: <5452534D.5060002@gmail.com> <545255A6.6000600@stoneleaf.us>
 <m2u0vr$nos$1@ger.gmane.org>
Message-ID: <CAGRr6BE4Y--idYsv8uCcaoyvYZtQ6yH9QO0D7D866fGcxg1k7Q@mail.gmail.com>

If anything, a better idea would be to make "raise [... [from ...]]" an
expression (possibly requiring parentheses, ? la yield), thus allowing it
to be put in constructs such as the one suggested, but also in lambda
expressions.
Of course, the value of the expression doesn't have to be specified (the
assignment target stays unbound, causing a NameError if we somehow catch
the exception and later refer to the assignment target).

2014-10-30 11:45 GMT-07:00 Terry Reedy <tjreedy at udel.edu>:

> On 10/30/2014 11:13 AM, Ethan Furman wrote:
>
>> On 10/30/2014 08:03 AM, Javier Dehesa wrote:
>>
>>>
>>> This happens to me with some frequency:
>>>
>>>      result = f(args)
>>>      if not result:   # of "if result is None:"
>>>          raise Exception(...)
>>>
>>> What if I could just say?
>>>
>>>      result = f(args) or raise Exception(...)
>>>
>>
>> Seems like a decent idea, but you can already have most of that:
>>
>>     result = f(args) or raise_exc(ValueError, 'args must be ...')
>>
>> and then have 'raise_exc' do the exception raising work.
>>
>
> No need to pass exception class and message separately instead of an
> exception instance.
>
> def raiser(exc):
>     raise exc
>
> print(1 or raiser(ValueError('null value')))
> print(0 or raiser(ValueError('null value')))
>
> >>>
> 1
> Traceback (most recent call last):
>   File "c:\programs\python34\tem.py", line 5, in <module>
>     print(0 or raiser(ValueError('null value')))
>   File "c:\programs\python34\tem.py", line 2, in raiser
>     raise exc
> ValueError: null value
>
> It does add another line to the trackback, but this is pretty minor.
>
> --
> Terry Jan Reedy
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141103/fa1df1ce/attachment-0001.html>

From greg.ewing at canterbury.ac.nz  Mon Nov  3 22:38:49 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 04 Nov 2014 10:38:49 +1300
Subject: [Python-ideas] Change how Generator Expressions handle
 StopIteration
In-Reply-To: <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
Message-ID: <5457F5E9.8060205@canterbury.ac.nz>

Guido van Rossum wrote:

> FWIW the implementation of my proposal is easy to describe (which the 
> Zen approves of): when a StopIteration leaves a frame, replace it with 
> some other exception (either a new, custom one or maybe just 
> RuntimeError), chaining the original StopIteration.

Wouldn't that break all existing iterators whose next() method
is implemented in Python? Since the code that catches the
StopIteration is always in a different frame then.

Or are you only talking about generator frames here, not
frames in general?

-- 
Greg

From guido at python.org  Tue Nov  4 01:29:31 2014
From: guido at python.org (Guido van Rossum)
Date: Mon, 3 Nov 2014 16:29:31 -0800
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <5457F5E9.8060205@canterbury.ac.nz>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
Message-ID: <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>

On Mon, Nov 3, 2014 at 1:38 PM, Greg Ewing <greg.ewing at canterbury.ac.nz>
wrote:

> Guido van Rossum wrote:
>
>  FWIW the implementation of my proposal is easy to describe (which the Zen
>> approves of): when a StopIteration leaves a frame, replace it with some
>> other exception (either a new, custom one or maybe just RuntimeError),
>> chaining the original StopIteration.
>>
>
> Wouldn't that break all existing iterators whose next() method
> is implemented in Python? Since the code that catches the
> StopIteration is always in a different frame then.
>
> Or are you only talking about generator frames here, not
> frames in general?
>

I have to apologize, I pretty much had this backwards.

What I should have said is that a generator should always be terminated by
a return or falling off the end, and if StopIteration is raised in the
generator (either by an explicit raise or raised by something it calls) and
not caught by an except clause in the same generator, it should be turned
into something else, so it bubbles out as something that keeps getting
bubbled up rather than silently ending a for-loop. OTOH StopIteration
raised by a non-generator function should not be mutated.

I'm sorry if this invalidates Nick's endorsement of the proposal.

I definitely see this as a serious backward incompatibility: no matter how
often it leads to buggy or obfuscated code, it's how things have always
worked.

Regarding Akira Li's examples of groupby(), unfortunately I find both
versions inscrutable -- on a casual inspection I have no idea what goes on.
I would have to think about how I would write groupby() myself (but I'm
pretty sure I wouldn't us functools.partial(). :-)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141103/1040d46a/attachment.html>

From ron3200 at gmail.com  Tue Nov  4 04:17:44 2014
From: ron3200 at gmail.com (Ron Adam)
Date: Mon, 03 Nov 2014 21:17:44 -0600
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
Message-ID: <m39ggp$sto$1@ger.gmane.org>



On 11/03/2014 06:29 PM, Guido van Rossum wrote:
>
> Regarding Akira Li's examples of groupby(), unfortunately I find both
> versions inscrutable -- on a casual inspection I have no idea what goes on.
> I would have to think about how I would write groupby() myself (but I'm
> pretty sure I wouldn't us functools.partial(). :-)

I agree, and just happened to try.  After several tries, it's tricky, I 
come to the conclusion it fits the pattern of peeking at the first value of 
the iterater best.  It seems to be a fairly common pattern and hard to get 
right with iterators.


For what it's worth here is what I came up with.  :-)

class PeekIter:
     """ Create an iterator which allows you to peek() at the
         next item to be yielded. It also stores the exception
         and re-raises it on the next next() call.
     """
     def __init__(self, iterable):
         self._exn = None
         self._it = iter(iterable)
         self._peek = next(iter(iterable))

     def peek(self):
         return self._peek

     def ok(self):
         #True if no exception occured when setting self._peek
         return self._exn == None

     def __iter__(self):
         return iter(self)

     def __next__(self):
         if self._exn != None:
             raise self._exn
         t = self._peek
         try:
             self._peek = next(self._it)
         except Exception as exn:
             self._exn = exn
             self._peek = None
         return t


def group_by(iterable):
     """ Yield (key, (group))'s of like values
         from an iterator.
     """
     itr = PeekIter(iterable)

     def group(it):
         # Yield only like values.
         k = it.peek()
         while it.peek() == k:
             yield next(it)

     while itr.ok():
         yield itr.peek(), [x for x in group(itr)]


print(' '.join(''.join(k) for k, v in group_by('AAAABBBCCDAABBB')))
# 'A B C D A B'

print(' '.join(''.join(g) for k, g in group_by('AAAABBBCCDAABBB')))
# 'AAAA BBB CC D AA BBB'



From ron3200 at gmail.com  Tue Nov  4 04:50:38 2014
From: ron3200 at gmail.com (Ron Adam)
Date: Mon, 03 Nov 2014 21:50:38 -0600
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <m39ggp$sto$1@ger.gmane.org>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <m39ggp$sto$1@ger.gmane.org>
Message-ID: <m39ief$nv8$1@ger.gmane.org>

Just a very minor correction, list(group(itr) instead of list comp. ...


On 11/03/2014 09:17 PM, Ron Adam wrote:
> def group_by(iterable):
>      """ Yield (key, (group))'s of like values
>          from an iterator.
>      """
>      itr = PeekIter(iterable)
>
>      def group(it):
>          # Yield only like values.
>          k = it.peek()
>          while it.peek() == k:
>              yield next(it)
>
>      while itr.ok():
>          yield itr.peek(), [x for x in group(itr)]


def group_by(iterable):
     """ Yield (key, (group))'s of like values
         from an iterator.
     """
     def group(it):
         # Yield only like values.
         k = it.peek()
         while it.peek() == k:
             yield next(it)

     itr = PeekIter(iterable)
     while itr.ok():
         yield itr.peek(), list(group(itr))

Cheers,
    Ron


From steve at pearwood.info  Tue Nov  4 04:57:04 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 4 Nov 2014 14:57:04 +1100
Subject: [Python-ideas] Deprecate dunder functions from operator module
In-Reply-To: <54553C4B.1010404@stoneleaf.us>
References: <20141030005321.GC26866@ando.pearwood.info>
 <54519687.6030708@stoneleaf.us> <54553C4B.1010404@stoneleaf.us>
Message-ID: <20141104035704.GD17635@ando.pearwood.info>

On Sat, Nov 01, 2014 at 01:02:19PM -0700, Ethan Furman wrote:
> On 10/29/2014 06:38 PM, Ethan Furman wrote:
> >On 10/29/2014 05:53 PM, Steven D'Aprano wrote:
> >>
> >>I propose a few things:
> >>
> >>* institute a policy that, in the event of a new function being added
> >>   to the operator module, only the dunderless version will be added;
> >>
> >>* change the documentation to make it clear that the dunderless
> >>   versions should be used, rather than merely being "convenience"
> >>   functions;
> >>
> >>* add a prominent note that the dunder versions exist for backwards
> >>   compatibility only and should not be used in new code.
> >
> >+1
> 
> Actually, make that -1.
> 
> I'm just crafting some tests to explore how NotImplemented impacts various 
> classes, and the dunder versions make the whole thing much nicer.

I'm surprised. I can't imagine dunder names making anything look 
nicer. It sounds like you are writing "operator.__add__" instead 
of "operator.add", which doesn't look nicer to me.

Can you show me your code?

Note that the dunder versions won't be going away soon, if at all, so 
if your code is relying on them, they will still be there. They just 
won't be prominently advertised.



-- 
Steven

From ethan at stoneleaf.us  Tue Nov  4 05:04:10 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 03 Nov 2014 20:04:10 -0800
Subject: [Python-ideas] Deprecate dunder functions from operator module
In-Reply-To: <20141104035704.GD17635@ando.pearwood.info>
References: <20141030005321.GC26866@ando.pearwood.info>
 <54519687.6030708@stoneleaf.us> <54553C4B.1010404@stoneleaf.us>
 <20141104035704.GD17635@ando.pearwood.info>
Message-ID: <5458503A.70500@stoneleaf.us>

On 11/03/2014 07:57 PM, Steven D'Aprano wrote:
> On Sat, Nov 01, 2014 at 01:02:19PM -0700, Ethan Furman wrote:
>> On 10/29/2014 06:38 PM, Ethan Furman wrote:
>>> On 10/29/2014 05:53 PM, Steven D'Aprano wrote:
>>>>
>>>> I propose a few things:
>>>>
>>>> * institute a policy that, in the event of a new function being added
>>>>    to the operator module, only the dunderless version will be added;
>>>>
>>>> * change the documentation to make it clear that the dunderless
>>>>    versions should be used, rather than merely being "convenience"
>>>>    functions;
>>>>
>>>> * add a prominent note that the dunder versions exist for backwards
>>>>    compatibility only and should not be used in new code.
>>>
>>> +1
>>
>> Actually, make that -1.
>>
>> I'm just crafting some tests to explore how NotImplemented impacts various
>> classes, and the dunder versions make the whole thing much nicer.
>
> I'm surprised. I can't imagine dunder names making anything look
> nicer. It sounds like you are writing "operator.__add__" instead
> of "operator.add", which doesn't look nicer to me.

The dunder versions are being used to check if a particular method is in a type's dictionary, to see how the subclass 
should be tailored for a test.  Without the dunder versions I would have had to create my own mapping between dunder and 
dunderless names.


> Can you show me your code?

Check out the "Role of NotImplemented" over on pydev.  The script is the last half of the first post.

--
~Ethan~

From joshua at landau.ws  Tue Nov  4 05:10:54 2014
From: joshua at landau.ws (Joshua Landau)
Date: Tue, 4 Nov 2014 04:10:54 +0000
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
Message-ID: <CAN1F8qXjkig0EeOm=T5R4sNsu8v8awa+NVJ6C7ZFo_oBr5yiMg@mail.gmail.com>

On 4 November 2014 00:29, Guido van Rossum <guido at python.org> wrote:
>
> Regarding Akira Li's examples of groupby(), unfortunately I find both
> versions inscrutable -- on a casual inspection I have no idea what goes on.
> I would have to think about how I would write groupby() myself (but I'm
> pretty sure I wouldn't us functools.partial(). :-)

It's worth noting that the functools.partial doesn't really do
anything. Just changing

    next_value = partial(next, iter(iterable))

to

    iterator = iter(iterable)

and "next_value()" to "next(iterator)" gets rid of it.

I would have tackled it by letting the inner iterator set a "finished"
flag and I would have exhausted the iterable by iterating over it:

    def groupby(iterable):
        # Make sure this is one-pass
        iterator = iter(iterable)
        finished = False

        # Yields a group
        def yield_group():
            nonlocal finished, group_key

            # This was taken off the iterator
            # by the previous group
            yield group_key

            for item in iterator:
                if item != group_key:
                    # Set up next group
                    group_key = item
                    return

                yield item

            # No more items in iterator
            finished = True

        # This is always the head of the next
        # or current group
        group_key = next(iterator)

        while not finished:
            group = yield_group()
            yield group_key, group

            # Make sure the iterator is exhausted
            for _ in group:
                pass

            # group_key will now be the head of the next group

This does have a key difference. Whereas with groupby you have the
confusing property that

    from itertools import groupby

    grps = groupby("|||---|||")
    a = next(grps)
    b = next(grps)
    c = next(grps)
    list(a[1])
    #>>> ['|', '|', '|']

with mine this does not happen:

    grps = groupby("|||---|||")

    grps = groupby("|||---|||")
    a = next(grps)
    b = next(grps)
    c = next(grps)
    list(a[1])
    #>>> []

Was this an oversight in the original design or is this actually
desired? I would guess it's an oversight.

From guido at python.org  Tue Nov  4 05:49:45 2014
From: guido at python.org (Guido van Rossum)
Date: Mon, 3 Nov 2014 20:49:45 -0800
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CAN1F8qXjkig0EeOm=T5R4sNsu8v8awa+NVJ6C7ZFo_oBr5yiMg@mail.gmail.com>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CAN1F8qXjkig0EeOm=T5R4sNsu8v8awa+NVJ6C7ZFo_oBr5yiMg@mail.gmail.com>
Message-ID: <CAP7+vJL1FNA9w1HW-vffdq0NUweNNet0n3UJupv3miaD7+5+hg@mail.gmail.com>

I guess implementing groupby() would be a good interview question. :-)

Can we get back to the issue at hand, which is whether and how we can
change the behavior of StopIteraton to be less error-prone?


On Mon, Nov 3, 2014 at 8:10 PM, Joshua Landau <joshua at landau.ws> wrote:

> On 4 November 2014 00:29, Guido van Rossum <guido at python.org> wrote:
> >
> > Regarding Akira Li's examples of groupby(), unfortunately I find both
> > versions inscrutable -- on a casual inspection I have no idea what goes
> on.
> > I would have to think about how I would write groupby() myself (but I'm
> > pretty sure I wouldn't us functools.partial(). :-)
>
> It's worth noting that the functools.partial doesn't really do
> anything. Just changing
>
>     next_value = partial(next, iter(iterable))
>
> to
>
>     iterator = iter(iterable)
>
> and "next_value()" to "next(iterator)" gets rid of it.
>
> I would have tackled it by letting the inner iterator set a "finished"
> flag and I would have exhausted the iterable by iterating over it:
>
>     def groupby(iterable):
>         # Make sure this is one-pass
>         iterator = iter(iterable)
>         finished = False
>
>         # Yields a group
>         def yield_group():
>             nonlocal finished, group_key
>
>             # This was taken off the iterator
>             # by the previous group
>             yield group_key
>
>             for item in iterator:
>                 if item != group_key:
>                     # Set up next group
>                     group_key = item
>                     return
>
>                 yield item
>
>             # No more items in iterator
>             finished = True
>
>         # This is always the head of the next
>         # or current group
>         group_key = next(iterator)
>
>         while not finished:
>             group = yield_group()
>             yield group_key, group
>
>             # Make sure the iterator is exhausted
>             for _ in group:
>                 pass
>
>             # group_key will now be the head of the next group
>
> This does have a key difference. Whereas with groupby you have the
> confusing property that
>
>     from itertools import groupby
>
>     grps = groupby("|||---|||")
>     a = next(grps)
>     b = next(grps)
>     c = next(grps)
>     list(a[1])
>     #>>> ['|', '|', '|']
>
> with mine this does not happen:
>
>     grps = groupby("|||---|||")
>
>     grps = groupby("|||---|||")
>     a = next(grps)
>     b = next(grps)
>     c = next(grps)
>     list(a[1])
>     #>>> []
>
> Was this an oversight in the original design or is this actually
> desired? I would guess it's an oversight.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141103/3839e8fe/attachment.html>

From steve at pearwood.info  Tue Nov  4 06:00:51 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 4 Nov 2014 16:00:51 +1100
Subject: [Python-ideas] Deprecate dunder functions from operator module
In-Reply-To: <5458503A.70500@stoneleaf.us>
References: <20141030005321.GC26866@ando.pearwood.info>
 <54519687.6030708@stoneleaf.us> <54553C4B.1010404@stoneleaf.us>
 <20141104035704.GD17635@ando.pearwood.info> <5458503A.70500@stoneleaf.us>
Message-ID: <20141104050051.GE17635@ando.pearwood.info>

On Mon, Nov 03, 2014 at 08:04:10PM -0800, Ethan Furman wrote:
> On 11/03/2014 07:57 PM, Steven D'Aprano wrote:

> >>I'm just crafting some tests to explore how NotImplemented impacts various
> >>classes, and the dunder versions make the whole thing much nicer.
> >
> >I'm surprised. I can't imagine dunder names making anything look
> >nicer. It sounds like you are writing "operator.__add__" instead
> >of "operator.add", which doesn't look nicer to me.
> 
> The dunder versions are being used to check if a particular method is in a 
> type's dictionary, to see how the subclass should be tailored for a test.  
> Without the dunder versions I would have had to create my own mapping 
> between dunder and dunderless names.

The dunder *methods* aren't going away. The dunder *functions* in 
operator won't be going away either. But even if they did, it is trivial 
to change:

    getattr(operator, op)  # op looks like '__add__', etc.

to

    getattr(operator, op.strip('_'))


except of course for full generality you might need a mapping to cover 
cases where the operator function isn't the same as the dunder method. 
E.g. operator.truth, operator.contains. (I don't think they are relevant 
to your case though.)

In any case, unless I'm missing something, I don't think that even 
full-blown removal of the dunder functions would affect your code in any 
significant way, and I'm not arguing for removal.


> >Can you show me your code?
> 
> Check out the "Role of NotImplemented" over on pydev.  The script is the 
> last half of the first post.

https://mail.python.org/pipermail/python-dev/2014-November/136875.html



-- 
Steven

From ron3200 at gmail.com  Tue Nov  4 08:07:29 2014
From: ron3200 at gmail.com (Ron Adam)
Date: Tue, 04 Nov 2014 01:07:29 -0600
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <m39ggp$sto$1@ger.gmane.org>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <m39ggp$sto$1@ger.gmane.org>
Message-ID: <m39tvi$7p1$1@ger.gmane.org>


A bad edit got me...  Don't mean to side track the Stopiteration 
discussion, but also don't want anyone having to debug my mistakes.

In the PeekIter __init__ class...

On 11/03/2014 09:17 PM, Ron Adam wrote:
>          self._peek = next(iter(iterable))

    to...   self._peek = next(self._it)

It was yielding an extra value on the first group.



As for the handling of the StopIteration, Maybe adding a StopGenerator
exception could ease the way.  It could be guaranteed to not to escape the 
generator it's raised in, while a StopIteration could propagate out until 
it caught by a loop.

Also StopGenerator(value) could be equivalent to return value.  The two for 
now at least overlap.  StopIterator(value) would be caught and converted. 
While a bare StopIterator would propogate out.  The new exception would 
make for clearor and easier to understand code in coroutines I think.

Another aspect of StopGenerator is it would escape inner loops of the 
generator it's raised in. Like return does.  And so you throw a 
StopGenerator exception into generator who's yield is nested in multiple 
loops to stop it.

Of course all bets are off until it's actually tried I think. Just because 
it sound good here, (or to me at the moment), doesn't mean it will work. 
Does it.  :-)

Cheers,
    Ron






From tomirendo at gmail.com  Wed Nov  5 00:12:05 2014
From: tomirendo at gmail.com (Yotam Vaknin)
Date: Wed, 5 Nov 2014 01:12:05 +0200
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CAP7+vJL1FNA9w1HW-vffdq0NUweNNet0n3UJupv3miaD7+5+hg@mail.gmail.com>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CAN1F8qXjkig0EeOm=T5R4sNsu8v8awa+NVJ6C7ZFo_oBr5yiMg@mail.gmail.com>
 <CAP7+vJL1FNA9w1HW-vffdq0NUweNNet0n3UJupv3miaD7+5+hg@mail.gmail.com>
Message-ID: <DCD6963A-4332-4152-99EE-F0094DB2D495@gmail.com>

Firstly, sorry for my buggy code that began all this mess.

But what about adding a named parameter to the next function, specifying an exception to be raised on StopIteration, allowing it to propagate?

?-4 ???? 2014, ???? 06:49, Guido van Rossum <guido at python.org> ???/?:

> I guess implementing groupby() would be a good interview question. :-)
> 
> Can we get back to the issue at hand, which is whether and how we can change the behavior of StopIteraton to be less error-prone?
> 
> 
> On Mon, Nov 3, 2014 at 8:10 PM, Joshua Landau <joshua at landau.ws> wrote:
>> On 4 November 2014 00:29, Guido van Rossum <guido at python.org> wrote:
>> >
>> > Regarding Akira Li's examples of groupby(), unfortunately I find both
>> > versions inscrutable -- on a casual inspection I have no idea what goes on.
>> > I would have to think about how I would write groupby() myself (but I'm
>> > pretty sure I wouldn't us functools.partial(). :-)
>> 
>> It's worth noting that the functools.partial doesn't really do
>> anything. Just changing
>> 
>>     next_value = partial(next, iter(iterable))
>> 
>> to
>> 
>>     iterator = iter(iterable)
>> 
>> and "next_value()" to "next(iterator)" gets rid of it.
>> 
>> I would have tackled it by letting the inner iterator set a "finished"
>> flag and I would have exhausted the iterable by iterating over it:
>> 
>>     def groupby(iterable):
>>         # Make sure this is one-pass
>>         iterator = iter(iterable)
>>         finished = False
>> 
>>         # Yields a group
>>         def yield_group():
>>             nonlocal finished, group_key
>> 
>>             # This was taken off the iterator
>>             # by the previous group
>>             yield group_key
>> 
>>             for item in iterator:
>>                 if item != group_key:
>>                     # Set up next group
>>                     group_key = item
>>                     return
>> 
>>                 yield item
>> 
>>             # No more items in iterator
>>             finished = True
>> 
>>         # This is always the head of the next
>>         # or current group
>>         group_key = next(iterator)
>> 
>>         while not finished:
>>             group = yield_group()
>>             yield group_key, group
>> 
>>             # Make sure the iterator is exhausted
>>             for _ in group:
>>                 pass
>> 
>>             # group_key will now be the head of the next group
>> 
>> This does have a key difference. Whereas with groupby you have the
>> confusing property that
>> 
>>     from itertools import groupby
>> 
>>     grps = groupby("|||---|||")
>>     a = next(grps)
>>     b = next(grps)
>>     c = next(grps)
>>     list(a[1])
>>     #>>> ['|', '|', '|']
>> 
>> with mine this does not happen:
>> 
>>     grps = groupby("|||---|||")
>> 
>>     grps = groupby("|||---|||")
>>     a = next(grps)
>>     b = next(grps)
>>     c = next(grps)
>>     list(a[1])
>>     #>>> []
>> 
>> Was this an oversight in the original design or is this actually
>> desired? I would guess it's an oversight.
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141105/621e121d/attachment.html>

From tjreedy at udel.edu  Wed Nov  5 01:55:51 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 04 Nov 2014 19:55:51 -0500
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <DCD6963A-4332-4152-99EE-F0094DB2D495@gmail.com>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CAN1F8qXjkig0EeOm=T5R4sNsu8v8awa+NVJ6C7ZFo_oBr5yiMg@mail.gmail.com>
 <CAP7+vJL1FNA9w1HW-vffdq0NUweNNet0n3UJupv3miaD7+5+hg@mail.gmail.com>
 <DCD6963A-4332-4152-99EE-F0094DB2D495@gmail.com>
Message-ID: <m3bsio$g8f$1@ger.gmane.org>

On 11/4/2014 6:12 PM, Yotam Vaknin wrote:
> Firstly, sorry for my buggy code that began all this mess.
>
> But what about adding a named parameter to the next function, specifying
> an exception to be raised on StopIteration, allowing it to propagate?

There is already a 'default' option, so the two would have to be 
mutually exclusive.

"next(iterator[, default])

     Retrieve the next item from the iterator by calling its __next__() 
method. If default is given, it is returned if the iterator is 
exhausted, otherwise StopIteration is raised."

I believe either adding an alternate exception, or raising the default 
if it is an exception, has been discussed before on this list.  In 
cannot remember details other than the obvious fact that nothing was 
changed.

-- 
Terry Jan Reedy


From greg.ewing at canterbury.ac.nz  Wed Nov  5 02:18:39 2014
From: greg.ewing at canterbury.ac.nz (Greg)
Date: Wed, 05 Nov 2014 14:18:39 +1300
Subject: [Python-ideas] Change how Generator Expressions handle
 StopIteration
In-Reply-To: <m3bsio$g8f$1@ger.gmane.org>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CAN1F8qXjkig0EeOm=T5R4sNsu8v8awa+NVJ6C7ZFo_oBr5yiMg@mail.gmail.com>
 <CAP7+vJL1FNA9w1HW-vffdq0NUweNNet0n3UJupv3miaD7+5+hg@mail.gmail.com>
 <DCD6963A-4332-4152-99EE-F0094DB2D495@gmail.com> <m3bsio$g8f$1@ger.gmane.org>
Message-ID: <54597AEF.5090106@canterbury.ac.nz>

On 5/11/2014 1:55 p.m., Terry Reedy wrote:

> There is already a 'default' option, so the two would have to be
> mutually exclusive.
>
> "next(iterator[, default])

If it were a keyword-only argument, there wouldn't be any
conflict.

But I'm not sure I like the idea of adding more complexities
to the iterator protocol.

Anything that changes the signature of __next__ or __iter__
would be a bad idea, since all existing iterables
would need to be updated to take it into account.

If this is is to be done, it might be better to add a new
optional dunder method, so that existing iterables would
continue to work.

-- 
Greg


From ncoghlan at gmail.com  Wed Nov  5 15:09:19 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 6 Nov 2014 00:09:19 +1000
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com>
 <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
Message-ID: <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>

On 4 Nov 2014 10:31, "Guido van Rossum" <guido at python.org> wrote:
>
> On Mon, Nov 3, 2014 at 1:38 PM, Greg Ewing <greg.ewing at canterbury.ac.nz>
wrote:
>>
>> Guido van Rossum wrote:
>>
>>> FWIW the implementation of my proposal is easy to describe (which the
Zen approves of): when a StopIteration leaves a frame, replace it with some
other exception (either a new, custom one or maybe just RuntimeError),
chaining the original StopIteration.
>>
>>
>> Wouldn't that break all existing iterators whose next() method
>> is implemented in Python? Since the code that catches the
>> StopIteration is always in a different frame then.
>>
>> Or are you only talking about generator frames here, not
>> frames in general?
>
>
> I have to apologize, I pretty much had this backwards.
>
> What I should have said is that a generator should always be terminated
by a return or falling off the end, and if StopIteration is raised in the
generator (either by an explicit raise or raised by something it calls) and
not caught by an except clause in the same generator, it should be turned
into something else, so it bubbles out as something that keeps getting
bubbled up rather than silently ending a for-loop. OTOH StopIteration
raised by a non-generator function should not be mutated.
>
> I'm sorry if this invalidates Nick's endorsement of the proposal.

I actually thought this was what you meant originally, and while it *would*
require changes to contextlib, they're fairly minor: the current check for
StopIteration would change to catch "UnexpectedStopIteration" instead, and
the exception identity check would look at __cause__ rather than directly
at the caught exception.

> I definitely see this as a serious backward incompatibility: no matter
how often it leads to buggy or obfuscated code, it's how things have always
worked.

Aye, breaking the equivalence between "return" and "raise StopIteration" is
pretty major.

I'm not even sure a plausible transition plan is possible, as at least
contextlib would trigger any warning we might issue.

Regards,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141106/5b71e632/attachment.html>

From ncoghlan at gmail.com  Wed Nov  5 22:47:09 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 6 Nov 2014 07:47:09 +1000
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com>
 <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
Message-ID: <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>

On 6 Nov 2014 00:09, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
> On 4 Nov 2014 10:31, "Guido van Rossum" <guido at python.org> wrote:
> >
> >
> > I have to apologize, I pretty much had this backwards.
> >
> > What I should have said is that a generator should always be terminated
by a return or falling off the end, and if StopIteration is raised in the
generator (either by an explicit raise or raised by something it calls) and
not caught by an except clause in the same generator, it should be turned
into something else, so it bubbles out as something that keeps getting
bubbled up rather than silently ending a for-loop. OTOH StopIteration
raised by a non-generator function should not be mutated.
> >
> > I'm sorry if this invalidates Nick's endorsement of the proposal.
>
> I actually thought this was what you meant originally, and while it
*would* require changes to contextlib, they're fairly minor: the current
check for StopIteration would change to catch "UnexpectedStopIteration"
instead, and the exception identity check would look at __cause__ rather
than directly at the caught exception.
>
> > I definitely see this as a serious backward incompatibility: no matter
how often it leads to buggy or obfuscated code, it's how things have always
worked.
>
> Aye, breaking the equivalence between "return" and "raise StopIteration"
is pretty major.
>
> I'm not even sure a plausible transition plan is possible, as at least
contextlib would trigger any warning we might issue.

And having said that... what if we introduced UnexpectedStopIteration but
initially made it a subclass of StopIteration?

We could issue a deprecation warning whenever we triggered the
StopIteration -> UnexpectedStopIteration conversion, pointing out that at
some point in the future (3.6? 3.7?), UnexpectedStopIteration will no
longer be a subclass of StopIteration (perhaps becoming a subclass of
RuntimeError instead?).

contextlib could avoid the warning by preconstructing a suitable
UnexpectedStopIteration instance and throwing *that* into the generator,
rather than throwing in a StopIteration raised from the with statement body.

Regards,
Nick.

>
> Regards,
> Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141106/9f3406df/attachment.html>

From apalala at gmail.com  Thu Nov  6 01:35:56 2014
From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=)
Date: Wed, 5 Nov 2014 20:05:56 -0430
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
Message-ID: <CAN1YFWtgiJw9uNXCOYzpyCOcvmKwQF7E-sGA15Tv2q9pW1202A@mail.gmail.com>

On Wed, Nov 5, 2014 at 5:17 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> And having said that... what if we introduced UnexpectedStopIteration but
> initially made it a subclass of StopIteration?


Exactly!


-- 
Juancarlo *A?ez*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141105/8733c1a6/attachment.html>

From ron3200 at gmail.com  Thu Nov  6 04:34:05 2014
From: ron3200 at gmail.com (Ron Adam)
Date: Wed, 05 Nov 2014 21:34:05 -0600
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
References: <CANMcLn17DCtSXzRikTL0H97YFJis8nZGa50rkatjfe4xGnzDqw@mail.gmail.com>
 <CAP7+vJJ2YHTemTyRaHO40iBUAy0Gp5WAHOf_O5aWwtkrAO25Xg@mail.gmail.com>
 <CADiSq7fgd-un-DFG1-Vkq0MUj1w1c+grRce2QEL=MfGCmM6Y4g@mail.gmail.com>
 <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
Message-ID: <m3eq7f$dfn$1@ger.gmane.org>



On 11/05/2014 03:47 PM, Nick Coghlan wrote:
>  > > I'm sorry if this invalidates Nick's endorsement of the proposal.
>  >
>  > I actually thought this was what you meant originally, and while it
> *would* require changes to contextlib, they're fairly minor: the current
> check for StopIteration would change to catch "UnexpectedStopIteration"
> instead, and the exception identity check would look at __cause__ rather
> than directly at the caught exception.
>  >
>  > > I definitely see this as a serious backward incompatibility: no matter
> how often it leads to buggy or obfuscated code, it's how things have always
> worked.
>  >
>  > Aye, breaking the equivalence between "return" and "raise StopIteration"
> is pretty major.
>  >
>  > I'm not even sure a plausible transition plan is possible, as at least
> contextlib would trigger any warning we might issue.
>
> And having said that... what if we introduced UnexpectedStopIteration but
> initially made it a subclass of StopIteration?
>
> We could issue a deprecation warning whenever we triggered the
> StopIteration -> UnexpectedStopIteration conversion, pointing out that at
> some point in the future (3.6? 3.7?), UnexpectedStopIteration will no
> longer be a subclass of StopIteration (perhaps becoming a subclass of
> RuntimeError instead?).
>
> contextlib could avoid the warning by preconstructing a suitable
> UnexpectedStopIteration instance and throwing *that* into the generator,
> rather than throwing in a StopIteration raised from the with statement body.

I really don't like the name Unexpected anything.  It makes me think of 
blue screens and page faults.  :-/

I'm also not sure how it's supposed to work or where it should come from.


As near as I can tell these two examples below are equivalent.  I think the 
thing that needs to be avoided is the case of the endless loop.  It would 
be better to let the Exception be noisy.

How would the second example be changed in order to do that?  Or is there 
some other thing that needs to be fixed?

Cheers,
    Ron



def izip(*args):
     iters = [iter(obj) for obj in args]
     while True:
         yield list(next(it) for it in iters)  #StopIteration suppressed
                                               #by list comprehension.
                                               #resulting in empty lists.
     #While Loop never exits.
     print("never printed")

a = izip([1,2],[3,4])
print(next(a),next(a),next(a))
# (1, 3) (2, 4) ()

#list(izip([1,2],[3,4])) #Currently never returns




def izip2(*args):
     iters = [iter(obj) for obj in args]
     while True:
         L = []
         for it in iters:
             try:
               obj = next(it)       # StopIteration suppressed here.
             except StopIteration:
               break                # exit for-loop
                                    # "return" instead of "break"?
             L.append(obj)
         yield L
     # While Loop never exits.
     print("never printed")

a = izip2([5,6],[7,8])
print(next(a),next(a),next(a))
# (5, 7) (6, 8) ()

list(izip2([5,6],[7,8])) #Currently never returns

# Not only doesn't it exit, but it's building
# an endless list of empty lists!




From steve at pearwood.info  Thu Nov  6 11:15:26 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 6 Nov 2014 21:15:26 +1100
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
References: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
Message-ID: <20141106101525.GC3597@ando.pearwood.info>

On Thu, Nov 06, 2014 at 07:47:09AM +1000, Nick Coghlan wrote:

> And having said that... what if we introduced UnexpectedStopIteration but
> initially made it a subclass of StopIteration?
> 
> We could issue a deprecation warning whenever we triggered the
> StopIteration -> UnexpectedStopIteration conversion, pointing out that at
> some point in the future (3.6? 3.7?), UnexpectedStopIteration will no
> longer be a subclass of StopIteration (perhaps becoming a subclass of
> RuntimeError instead?).

I'm sorry, I have been trying to follow this thread, but there have 
been too many wrong turns and side-tracks for me to keep it straight. 
What is the problem this is supposed to solve?

Is it just that list (and set and dict) comprehensions treat 
StopIteration differently than generator expressions? That is, that 

    [expr for x in iterator]

    list(expr for x in iterator)

are not exactly equivalent, if expr raises StopIteration.

If so, it seems to me that you're adding a lot of conceptual baggage and 
complication for very little benefit, and this will probably confuse 
people far more than the current situation does. The different treatment 
of StopIteration in generator expressions and list comprehensions does 
not seem to be a problem for people in practice, judging by the 
python-list and tutor mailing lists.

The current situation is simple to learn and understand:

(1) Generator expressions *emit* StopIteration when they are done:

py> next(iter([]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration


(2) Functions such as tuple, list, set, dict *absorb* StopIteration:

py> list(iter([]))
[]
py> it = iter([])
py> list(next(it) for y in range(1000))
[]

For-loops do the same, if StopIteration is raised in the "for x in 
iterable" header. That's how it knows the loop is done. The "for" part 
of a comprehension is the same.


(3) But raising StopIteration in the expression part (or if part) of a 
comprehension does not absord the exception, it is treated like any 
other exception:

py> [next(iter([])) for y in range(1000)]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <listcomp>
StopIteration

If that is surprising to anyone, I suggest it is because they haven't 
considered what happens when you raise StopIteration in the body of a 
for-loop:

py> for y in range(1000):
...     next(iter([]))
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
StopIteration


To me, the status quo is consistent, understandable, and predictable.

In contrast, you have:

- a solution to something which I'm not sure is even a problem
  that needs solving;

- but if it does, the solution seems quite magical, complicated,
  and hard to understand;

- it is unclear (to me) under what circumstances StopIteration 
  will be automatically converted to UnexpectedStopIteration;

- and it seems to me that it will lead to surprising behaviour
  when people deliberately raise StopIteration only to have it
  mysteriously turn into a different exception, but only 
  sometimes.


It seems to me that if the difference between comprehensions and 
generator expressions really is a problem that needs solving, that the 
best way to proceed is using the __future__ mechanism. 3.5 could 
introduce

    from __future__ comprehensions_absorb_stopiteration

and then 3.6 or 3.7 could make it the default behaviour.

We're still breaking backwards compatibility, but at least we're doing 
it cleanly, without magic (well, except the __future__ magic, but that's 
well-known and acceptible magic). There will be a transition period 
during which people can choose to keep the old behaviour or the new, and 
then we transition to the new behaviour. This automatic transformation 
of some StopIterations into something else seems like it will be worse 
than the problem it is trying to fix.

For what it is worth, I'm a strong -1 on changing the behaviour of 
comprehensions at all, but if we must change it in a backwards 
incompatible way, +1 on __future__ and -1 on changing the exceptions to 
a different exception.



-- 
Steven

From ron3200 at gmail.com  Thu Nov  6 15:24:13 2014
From: ron3200 at gmail.com (Ron Adam)
Date: Thu, 06 Nov 2014 08:24:13 -0600
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <20141106101525.GC3597@ando.pearwood.info>
References: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
Message-ID: <m3g0ae$mlf$1@ger.gmane.org>



On 11/06/2014 04:15 AM, Steven D'Aprano wrote:
> The current situation is simple to learn and understand:
>
> (1) Generator expressions*emit*  StopIteration when they are done:
>
> py> next(iter([]))
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> StopIteration

It's the "when they are done" part that's having the issue.  In some cases, 
they are never completely done because the StopIteration Error is handled 
incorrectly.  The reason this doesn't show up more is that most iterators 
only have one loop, which is exited and the generator ends causing a 
different StopIteration to be emitted from the one that ends the loop.

> (2) Functions such as tuple, list, set, dict*absorb*  StopIteration:
>
> py> list(iter([]))
> []
> py> it = iter([])
> py> list(next(it) for y in range(1000))
> []

Right, sometimes this doesn't happen.


> For-loops do the same, if StopIteration is raised in the "for x in
> iterable" header. That's how it knows the loop is done. The "for" part
> of a comprehension is the same.

I think this part is working as it should.

> (3) But raising StopIteration in the expression part (or if part) of a
> comprehension does not absord the exception, it is treated like any
> other exception:
>
> py> [next(iter([])) for y in range(1000)]
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
>    File "<stdin>", line 1, in <listcomp>
> StopIteration

This is not always working.  When a StopIteration is raised from the next 
call in the example that started the thread, it's getting replaced with the 
equivalent of a break, so it's never exiting the generator completely.  I 
think it needs to be replaced with the equivalent of return, which will end 
the generator, and cause a StopIteration to be emitted.  (As you describe 
here.)

It looks like it's a bug in the C code for generator expressions.  Since it 
only effects generator expressions that have iterators with nested loops, 
it may be fixible.

Cheers,
    Ron


From tjreedy at udel.edu  Thu Nov  6 18:02:57 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 06 Nov 2014 12:02:57 -0500
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <20141106101525.GC3597@ando.pearwood.info>
References: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
Message-ID: <m3g9k4$vhe$1@ger.gmane.org>

On 11/6/2014 5:15 AM, Steven D'Aprano wrote:
> On Thu, Nov 06, 2014 at 07:47:09AM +1000, Nick Coghlan wrote:
>
>> And having said that... what if we introduced UnexpectedStopIteration but
>> initially made it a subclass of StopIteration?
>>
>> We could issue a deprecation warning whenever we triggered the
>> StopIteration -> UnexpectedStopIteration conversion, pointing out that at
>> some point in the future (3.6? 3.7?), UnexpectedStopIteration will no
>> longer be a subclass of StopIteration (perhaps becoming a subclass of
>> RuntimeError instead?).
>
> I'm sorry, I have been trying to follow this thread, but there have
> been too many wrong turns and side-tracks for me to keep it straight.
> What is the problem this is supposed to solve?
>
> Is it just that list (and set and dict) comprehensions treat
> StopIteration differently than generator expressions? That is, that
>
>      [expr for x in iterator]
>
>      list(expr for x in iterator)
>
> are not exactly equivalent, if expr raises StopIteration.
>
> If so, it seems to me that you're adding a lot of conceptual baggage and
> complication for very little benefit, and this will probably confuse
> people far more than the current situation does. The different treatment
> of StopIteration in generator expressions and list comprehensions does
> not seem to be a problem for people in practice, judging by the
> python-list and tutor mailing lists.
>
> The current situation is simple to learn and understand:
>
> (1) Generator expressions *emit* StopIteration when they are done:
>
> py> next(iter([]))
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> StopIteration

'iter([])' is a list_iterator, not a generator expression.  Here is the 
example I think you wanted.

>>> next(i for i in ())
Traceback (most recent call last):
   File "<pyshell#0>", line 1, in <module>
     next(i for i in ())
StopIteration

> (2) Functions such as tuple, list, set, dict *absorb* StopIteration:
>
> py> list(iter([]))
> []
> py> it = iter([])
> py> list(next(it) for y in range(1000))
> []
>
> For-loops do the same, if StopIteration is raised in the "for x in
> iterable" header. That's how it knows the loop is done. The "for" part
> of a comprehension is the same.
>
>
> (3) But raising StopIteration in the expression part (or if part) of a
> comprehension does not absord the exception, it is treated like any
> other exception:
>
> py> [next(iter([])) for y in range(1000)]
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
>    File "<stdin>", line 1, in <listcomp>
> StopIteration
>
> If that is surprising to anyone, I suggest it is because they haven't
> considered what happens when you raise StopIteration in the body of a
> for-loop:

Which is the Python translation of the comprehension.

> py> for y in range(1000):
> ...     next(iter([]))
> ...
> Traceback (most recent call last):
>    File "<stdin>", line 2, in <module>
> StopIteration
>
>
> To me, the status quo is consistent, understandable, and predictable.
...

I agree.

-- 
Terry Jan Reedy


From ethan at stoneleaf.us  Thu Nov  6 18:12:00 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 06 Nov 2014 09:12:00 -0800
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <m3g9k4$vhe$1@ger.gmane.org>
References: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info> <m3g9k4$vhe$1@ger.gmane.org>
Message-ID: <545BABE0.5030809@stoneleaf.us>

I believe the issue is that, in some situations, the generator is absorbing StopIteration when it shouldn't.

--
~Ethan~

From guido at python.org  Thu Nov  6 19:54:51 2014
From: guido at python.org (Guido van Rossum)
Date: Thu, 6 Nov 2014 10:54:51 -0800
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <20141106101525.GC3597@ando.pearwood.info>
References: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
Message-ID: <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>

TL;DR :-(

The crux of the issue is that sometimes the StopIteration raised by an
explicit next() call is "just" an exception, but sometimes it terminates
some controlling loop. Most precedents clearly establish it as "just" an
exception, e.g. (these are all nonsense)

it = iter([1, 2, 3, -1])
for x in it:
    if x < 0:
        next(it)

it = iter([1, 2, 3])
[x for x in it if next(it) < 0]

Both these raise StopIteration and report a traceback.

But generators and generator expressions are different. In a generator, any
StopIteration that isn't caught by an exception handler in the body of the
generator implicitly terminates the iteration, just like "return" or
falling off the end. This was an explicitly designed feature, but I don't
think has worked out very well, given that more often than not, a
StopIteration that "escapes" from some expression is a bug.

And because generator expressions are implemented using generators (duh),
the following returns [] instead of raising StopIteration:

it = iter([1, 2, 3])
list(x for x in it if next(it) < 0)

This is confusing because it breaks the (intended) equivalence between
list(<genexp>) and [<genexp>] (even though we refer to the latter as a
comprehension, the syntax inside the [] is the same as a generator
expression.

If I had had the right foresight, I would have made it an error to
terminate a generator with a StopIteration, probably by raising another
exception chained to the StopIteration (so the traceback shows the place
where the StopIteration escaped).

The question at hand is if we can fix this post-hoc, using clever tricks
and (of course) a deprecation period.

--Guido


On Thu, Nov 6, 2014 at 2:15 AM, Steven D'Aprano <steve at pearwood.info> wrote:

> On Thu, Nov 06, 2014 at 07:47:09AM +1000, Nick Coghlan wrote:
>
> > And having said that... what if we introduced UnexpectedStopIteration but
> > initially made it a subclass of StopIteration?
> >
> > We could issue a deprecation warning whenever we triggered the
> > StopIteration -> UnexpectedStopIteration conversion, pointing out that at
> > some point in the future (3.6? 3.7?), UnexpectedStopIteration will no
> > longer be a subclass of StopIteration (perhaps becoming a subclass of
> > RuntimeError instead?).
>
> I'm sorry, I have been trying to follow this thread, but there have
> been too many wrong turns and side-tracks for me to keep it straight.
> What is the problem this is supposed to solve?
>
> Is it just that list (and set and dict) comprehensions treat
> StopIteration differently than generator expressions? That is, that
>
>     [expr for x in iterator]
>
>     list(expr for x in iterator)
>
> are not exactly equivalent, if expr raises StopIteration.
>
> If so, it seems to me that you're adding a lot of conceptual baggage and
> complication for very little benefit, and this will probably confuse
> people far more than the current situation does. The different treatment
> of StopIteration in generator expressions and list comprehensions does
> not seem to be a problem for people in practice, judging by the
> python-list and tutor mailing lists.
>
> The current situation is simple to learn and understand:
>
> (1) Generator expressions *emit* StopIteration when they are done:
>
> py> next(iter([]))
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> StopIteration
>
>
> (2) Functions such as tuple, list, set, dict *absorb* StopIteration:
>
> py> list(iter([]))
> []
> py> it = iter([])
> py> list(next(it) for y in range(1000))
> []
>
> For-loops do the same, if StopIteration is raised in the "for x in
> iterable" header. That's how it knows the loop is done. The "for" part
> of a comprehension is the same.
>
>
> (3) But raising StopIteration in the expression part (or if part) of a
> comprehension does not absord the exception, it is treated like any
> other exception:
>
> py> [next(iter([])) for y in range(1000)]
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "<stdin>", line 1, in <listcomp>
> StopIteration
>
> If that is surprising to anyone, I suggest it is because they haven't
> considered what happens when you raise StopIteration in the body of a
> for-loop:
>
> py> for y in range(1000):
> ...     next(iter([]))
> ...
> Traceback (most recent call last):
>   File "<stdin>", line 2, in <module>
> StopIteration
>
>
> To me, the status quo is consistent, understandable, and predictable.
>
> In contrast, you have:
>
> - a solution to something which I'm not sure is even a problem
>   that needs solving;
>
> - but if it does, the solution seems quite magical, complicated,
>   and hard to understand;
>
> - it is unclear (to me) under what circumstances StopIteration
>   will be automatically converted to UnexpectedStopIteration;
>
> - and it seems to me that it will lead to surprising behaviour
>   when people deliberately raise StopIteration only to have it
>   mysteriously turn into a different exception, but only
>   sometimes.
>
>
> It seems to me that if the difference between comprehensions and
> generator expressions really is a problem that needs solving, that the
> best way to proceed is using the __future__ mechanism. 3.5 could
> introduce
>
>     from __future__ comprehensions_absorb_stopiteration
>
> and then 3.6 or 3.7 could make it the default behaviour.
>
> We're still breaking backwards compatibility, but at least we're doing
> it cleanly, without magic (well, except the __future__ magic, but that's
> well-known and acceptible magic). There will be a transition period
> during which people can choose to keep the old behaviour or the new, and
> then we transition to the new behaviour. This automatic transformation
> of some StopIterations into something else seems like it will be worse
> than the problem it is trying to fix.
>
> For what it is worth, I'm a strong -1 on changing the behaviour of
> comprehensions at all, but if we must change it in a backwards
> incompatible way, +1 on __future__ and -1 on changing the exceptions to
> a different exception.
>
>
>
> --
> Steven
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141106/9a0bce05/attachment-0001.html>

From g.rodola at gmail.com  Thu Nov  6 19:59:31 2014
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Thu, 6 Nov 2014 19:59:31 +0100
Subject: [Python-ideas] including psutil in the standard library?
In-Reply-To: <59DEA399-5915-4562-8042-1372CF9D4353@yahoo.com>
References: <542319B1.2090606@ferrara.linux.it>
 <CAMpsgwbwLhy-WE6Jg0aSaLOCNCX4A4hx0gVC5VSw_Tqw9yoB6Q@mail.gmail.com>
 <59DEA399-5915-4562-8042-1372CF9D4353@yahoo.com>
Message-ID: <CAFYqXL_0p-wtzVm0coVfRfEzb-yE3fUbXvjxwr9V6=me5aUuNw@mail.gmail.com>

On Tue, Oct 14, 2014 at 12:49 AM, Andrew Barnert <
abarnert at yahoo.com.dmarc.invalid> wrote:

> On Oct 13, 2014, at 14:40, Victor Stinner <victor.stinner at gmail.com>
> wrote:
>
> > 2014-09-24 21:21 GMT+02:00 Stefano Borini <
> stefano.borini at ferrara.linux.it>:
> >> I am wondering if it would be possible to include psutil
> >> (https://pypi.python.org/pypi/psutil ) in the standard library, and if
> not,
> >> what would be needed.
> >
> > The problem is that Python only releases a new major version every 18
> > months (or something like that), so adding a single new function will
> > take so much time.
> >
> > Supporting a new platform, architecture, major version of an OS, etc.
> > may also need to wait a new Python major release.
> >
> > Operating systems are evolving quite fast. For example, there is the
> > new architecture AArch64, containers are more and more popular,
> > systemd can also be used to provide features similar to psutil and it
> > supports containers (running systemd, it even supports recursive
> > containers....). Network becomes more virtual with Software Defined
> > Networks (SDN, NFV, etc.). The Linux kernel is also extended at each
> > release. Example of "recent" addition: /proc/pid/fdinfo/ directory.
> > Does psutil provide NUMA information? NUMA also becomes more
> > information nowadays for performances.
> >
> > The psutil also still evolves. For example, I see that the version 2.1
> > released at April, 2014 adds "netstat-like functionalities". The
> > version 2.0 was only released one month before (March, 2014).
> >
> > The API of psutil changed a lot between 1.x and 2.0. The version 2.0
> > was only released a few months ago. Is it enough to consider that the
> > API is now stable enough (was enough tested)? Giampaolo (author of the
> > module) doesn't look to be confident in its own API ;-) He wrote "I
> > still want to work on a couple of new features first (...) and be 100%
> > sure that I'm happy with the API as it is right now".
> >
> > Maybe I'm wrong and the psutil is stable and can be "frozen" in the
> > standard library. Maybe it's possible to extract a subpart of the
> > psutil to keep the most stable part?
> >
> > I'm not strongly opposed to the integration of the psutil module into
> > the stdlib.
> >
> > Is it hard to install the psutil module today? I see wheel packages
> > and .exe installers for Windows. They are Linux packages (Debian,
> > Ubuntu, Fedora, ArchLinux, etc.)
>
> This might be a good idea.
>
> It seems to me that there are two common uses for psutil.
>
> Some people want just a little bit more than Python makes available, and
> they want it in a cross-platform (at least gnu vs. bsd vs. sysv, if not
> unix vs. windows) way. That doesn't change that much over time, and adding
> that much from psutil to the stdlib might make sense.
>
> Other people want all kinds of detailed process information, in a
> platform-specific way, without having to grub through low-level APIs. That
> changes frequently, and for those people, pip install psutil should
> probably remain the right answer.+
>
> (Of course there are also people who use psutil to do the exact same
> things subprocess and os can already do better; for those people, the
> answer is to stop doing that...)
>
> I guess the real question is, what are the use cases for the "stable
> core"? Once we know that, we can identify whether it's stable enough and
> simple enough to extract.


I think this might make sense. Skimming through
http://pythonhosted.org/psutil/ should give an idea of what's more
"necessary" and what not.
What's probably "less" necessary is:

- psutil.disk_partitions()
- psutil.net_connections()
- psutil.users()
- psutil.Process.open_files()  (it turned out this is not reliable on
Windows and FreeBSD)
- psutil.Process.connections()
- psutil.Process.memory_percent() ('cause it's a very simple utility method)

APIs which are currently duplicated in Python stdlib:

- psutil.disk_usage()  (shutil.disk_usage, see
http://bugs.python.org/issue12442)
- psutil.Process.nice()  (os.get/setpriority(), see:
http://bugs.python.org/issue10784)
- psutil.Process.cpu_affinity() (os.sched_get/setaffinity, see
https://docs.python.org/3.5/library/os.html#os.sched_setaffinity)
- psutil.cpu_count()  (available as multiprocessing.cpu_count() but psutil
can also distinguish between logical and physical cores)

The rest is probably worth it for inclusion.
As for my "doubts" regarding the stability of the API they basically comes
down to one issue only: https://github.com/giampaolo/psutil/issues/428.
In order to properly fix that issue I'll probably have to introduce a new
"ZombieProcess" exception class other than "NoSuchProcess".
That means users will have to change their code from this:

for p in psutil.process_iter():
     try:
          p.something()
    except psutil.NoSuchProcess:
         pass

...to this:

for p in psutil.process_iter():
     try:
          p.something()
    except (psutil.NoSuchProcess, psutil.ZombieProcess):
         pass

I would consider the rest of the API quite stable.
Please note that psutil currently supports:
- Linux >= 2.6 (perhaps also 2.4)
- Windows => XP
- OSX => 10.4 (not sure about older versions)
- FreeBSD => 9.0, perhaps also 8.0
- OpenSolaris (tested on OpenSolaris 10, not sure about other versions)

With that said I expect that the moment psutil is included in the buildbot
system a lot of currently undetected issues will be discovered as psutil
has only been actively developed and tested on the platforms listed above
(which are the ones I have access to).

> Is it hard to install the psutil module today? I see wheel packages
> and .exe installers for Windows. They are Linux packages (Debian,
> Ubuntu, Fedora, ArchLinux, etc.)

I would say it is reasonably easy, especially now that I'm providing wheels
for Windows.

-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141106/212a4937/attachment.html>

From solipsis at pitrou.net  Thu Nov  6 22:45:37 2014
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 6 Nov 2014 22:45:37 +0100
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
References: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com>
 <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
Message-ID: <20141106224537.4fc55146@fsol>

On Thu, 6 Nov 2014 10:54:51 -0800
Guido van Rossum <guido at python.org> wrote:
> 
> If I had had the right foresight, I would have made it an error to
> terminate a generator with a StopIteration, probably by raising another
> exception chained to the StopIteration (so the traceback shows the place
> where the StopIteration escaped).
> 
> The question at hand is if we can fix this post-hoc, using clever tricks
> and (of course) a deprecation period.

Is there any point in fixing it? Who relies on such borderline cases?

Regards

Antoine.



From apalala at gmail.com  Fri Nov  7 01:31:32 2014
From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=)
Date: Thu, 6 Nov 2014 20:01:32 -0430
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <20141106224537.4fc55146@fsol>
References: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
 <20141106224537.4fc55146@fsol>
Message-ID: <CAN1YFWvv9CeJ=vx7+RFHmHRNj188timQXe_XGkYkKGVXPPvJOg@mail.gmail.com>

On Thu, Nov 6, 2014 at 5:15 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Is there any point in fixing it? Who relies on such borderline cases?


import this?


-- 
Juancarlo *A?ez*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141106/0a2024fd/attachment-0001.html>

From steve at pearwood.info  Fri Nov  7 10:43:12 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 7 Nov 2014 20:43:12 +1100
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CAN1YFWvv9CeJ=vx7+RFHmHRNj188timQXe_XGkYkKGVXPPvJOg@mail.gmail.com>
References: <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
 <20141106224537.4fc55146@fsol>
 <CAN1YFWvv9CeJ=vx7+RFHmHRNj188timQXe_XGkYkKGVXPPvJOg@mail.gmail.com>
Message-ID: <20141107094311.GD3597@ando.pearwood.info>

On Thu, Nov 06, 2014 at 08:01:32PM -0430, Juancarlo A?ez wrote:
> On Thu, Nov 6, 2014 at 5:15 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> 
> > Is there any point in fixing it? Who relies on such borderline cases?
> 
> 
> import this?

What about it? There are 19 lines in the Zen, and I don't think any of 
them are particularly relevent here. Which line, or lines, were you 
thinking of?

-- 
Steven

From steve at pearwood.info  Fri Nov  7 12:20:30 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 7 Nov 2014 22:20:30 +1100
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
References: <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
Message-ID: <20141107112030.GA7768@ando.pearwood.info>

On Thu, Nov 06, 2014 at 10:54:51AM -0800, Guido van Rossum wrote:
> TL;DR :-(

That's how I feel about this whole thread ;-)


[...]
> If I had had the right foresight, I would have made it an error to
> terminate a generator with a StopIteration, probably by raising another
> exception chained to the StopIteration (so the traceback shows the place
> where the StopIteration escaped).
> 
> The question at hand is if we can fix this post-hoc, using clever tricks
> and (of course) a deprecation period.

Do we need "clever tricks"? In my earlier email, I suggested that if 
this needs to be fixed, the best way to introduce a change in 
behaviour is with the __future__ mechanism. 3.5 could introduce

    from __future__ stopiteration_is_an_error

(in my earlier post, I managed to get the suggested behaviour completely 
backwards) and then 3.6 could raise a warning and 3.7 could make it 
the default behaviour. 

We're still breaking backwards compatibility, but at least we're doing 
it cleanly, without clever and/or ugly hacks. There will be a transition 
period during which people can choose to keep the old behaviour or the 
new, and then we transition to the new behaviour.



-- 
Steven

From apalala at gmail.com  Fri Nov  7 13:57:59 2014
From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=)
Date: Fri, 7 Nov 2014 08:27:59 -0430
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <20141107094311.GD3597@ando.pearwood.info>
References: <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
 <20141106224537.4fc55146@fsol>
 <CAN1YFWvv9CeJ=vx7+RFHmHRNj188timQXe_XGkYkKGVXPPvJOg@mail.gmail.com>
 <20141107094311.GD3597@ando.pearwood.info>
Message-ID: <CAN1YFWu24v9OxFhHNV=zhGqJ3MHvmoXShsqnsk4RzOHn6Y-cSQ@mail.gmail.com>

On Fri, Nov 7, 2014 at 5:13 AM, Steven D'Aprano <steve at pearwood.info> wrote:

> What about it? There are 19 lines in the Zen, and I don't think any of
> them are particularly relevent here. Which line, or lines, were you
> thinking of?
>

You asked what was the point in fixing the current behavior regarding
StopIteration:

   -
   - Beautiful is better than ugly.
   - Explicit is better than implicit.
   - Simple is better than complex.
   - Flat is better than nested.
   - Readability counts.
   - Special cases aren't special enough to break the rules.
   - Errors should never pass silently.
   - In the face of ambiguity, refuse the temptation to guess.
   - There should be one-- and preferably only one --obvious way to do it.
   - Now is better than never.
   - If the implementation is hard to explain, it's a bad idea.

Basically, that some of the behaviours Guido mentioned are unexpected
consequences of a lack of foresight in the implementation of generator
expressions.

It is not in the Zen of Python to leave it as is just because some existing
code may be relying on the odd behavior.

Just to be clear, that StopIteration will cancel more than one iterator is
an unintended behavior that is something difficult to explain, is of
questionable usefulness, and is the source of difficult to catch bugs.

Cheers,

-- 
Juancarlo *A?ez*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141107/3e70a929/attachment.html>

From rosuav at gmail.com  Fri Nov  7 14:04:04 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 8 Nov 2014 00:04:04 +1100
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CAN1YFWu24v9OxFhHNV=zhGqJ3MHvmoXShsqnsk4RzOHn6Y-cSQ@mail.gmail.com>
References: <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
 <20141106224537.4fc55146@fsol>
 <CAN1YFWvv9CeJ=vx7+RFHmHRNj188timQXe_XGkYkKGVXPPvJOg@mail.gmail.com>
 <20141107094311.GD3597@ando.pearwood.info>
 <CAN1YFWu24v9OxFhHNV=zhGqJ3MHvmoXShsqnsk4RzOHn6Y-cSQ@mail.gmail.com>
Message-ID: <CAPTjJmrPetYnYmcaWUbmupqcWvKwHsZV-zZOXLs_K2czbVMvkA@mail.gmail.com>

On Fri, Nov 7, 2014 at 11:57 PM, Juancarlo A?ez <apalala at gmail.com> wrote:
> It is not in the Zen of Python to leave it as is just because some existing
> code may be relying on the odd behavior.

Actually, it is. Practicality beats purity. :)

ChrisA

From solipsis at pitrou.net  Fri Nov  7 14:15:37 2014
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 7 Nov 2014 14:15:37 +0100
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
References: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com>
 <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
 <20141106224537.4fc55146@fsol>
 <CAN1YFWvv9CeJ=vx7+RFHmHRNj188timQXe_XGkYkKGVXPPvJOg@mail.gmail.com>
Message-ID: <20141107141537.32310a1d@fsol>

On Thu, 6 Nov 2014 20:01:32 -0430
Juancarlo A?ez <apalala at gmail.com> wrote:
> On Thu, Nov 6, 2014 at 5:15 PM, Antoine Pitrou <solipsis-xNDA5Wrcr86sTnJN9+BGXg at public.gmane.org> wrote:
> 
> > Is there any point in fixing it? Who relies on such borderline cases?
> 
> 
> import this?

I was more interested in an answer to my second question, actually ;-)
i.e. "Who relies on such borderline cases?"

Regards

Antoine.



From abarnert at yahoo.com  Fri Nov  7 16:59:17 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 7 Nov 2014 07:59:17 -0800
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
References: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
Message-ID: <91F449C7-C6CB-4F1A-A9A4-D361B8B3A9AC@yahoo.com>

On Nov 6, 2014, at 10:54, Guido van Rossum <guido at python.org> wrote:

> This is confusing because it breaks the (intended) equivalence between list(<genexp>) and [<genexp>] (even though we refer to the latter as a comprehension, the syntax inside the [] is the same as a generator expression.

If this change (I mean the proposed clever workaround, not the "no terminating generators with StopIteration" change that's too radical) would be sufficient to make that equivalence actually true instead of just pretty close, I think that's reason enough to fix it on its own. Especially since that would make it easy to fix the genexpr docs. (Read 6.2.8 and tell me what it says the semantics of a genexpr are, and what values it yields. Now try to think of a way to fix that without repeating most of the text from 6.2.4, which nobody wants to do. If the docs could just define the semantics of genexprs, then define listcomps by saying that [<genexpr>] is equivalent to list(<genexpr>), that would be a lot simpler and clearer.)

From guido at python.org  Fri Nov  7 18:10:42 2014
From: guido at python.org (Guido van Rossum)
Date: Fri, 7 Nov 2014 09:10:42 -0800
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <20141107112030.GA7768@ando.pearwood.info>
References: <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
 <20141107112030.GA7768@ando.pearwood.info>
Message-ID: <CAP7+vJKpgpRGmgK3xF3fKpwSiPtpHt7HM6kffXCOpA9Hmn_6gw@mail.gmail.com>

Trying to keep the thread on focus...

On Fri, Nov 7, 2014 at 3:20 AM, Steven D'Aprano <steve at pearwood.info> wrote:

> On Thu, Nov 06, 2014 at 10:54:51AM -0800, Guido van Rossum wrote:
> > If I had had the right foresight, I would have made it an error to
> > terminate a generator with a StopIteration, probably by raising another
> > exception chained to the StopIteration (so the traceback shows the place
> > where the StopIteration escaped).
> >
> > The question at hand is if we can fix this post-hoc, using clever tricks
> > and (of course) a deprecation period.
>
> Do we need "clever tricks"? In my earlier email, I suggested that if
> this needs to be fixed, the best way to introduce a change in
> behaviour is with the __future__ mechanism. 3.5 could introduce
>
>     from __future__ stopiteration_is_an_error
>
> (in my earlier post, I managed to get the suggested behaviour completely
> backwards) and then 3.6 could raise a warning and 3.7 could make it
> the default behaviour.
>

Sure. (Though IMO __future__ itself is a clever hack.)


> We're still breaking backwards compatibility, but at least we're doing
> it cleanly, without clever and/or ugly hacks. There will be a transition
> period during which people can choose to keep the old behaviour or the
> new, and then we transition to the new behaviour.
>

We'd still need to specify the eventual behavior. I propose the following
as a strawman (after the deprecation period is finished and the __future__
import is no longer needed):

- If a StopIteration is about to bubble out of a generator frame, it is
replaced with some other exception (maybe RuntimeError, maybe a new custom
Exception subclass, but *not* deriving from StopIteration) which causes the
next() call (which invoked the generator) to fail, passing that exception
out. From then on it's just like any old exception.

During the transition, we check if the generator was defined in the scope
of the __future__, and if so, we do the the same thing; otherwise, we issue
a warning and let the StopIteration bubble out, eventually terminating some
loop or generator expression.

It would be nice if, when the warning is made fatal (e.g. through the -W
flag), the exception raised was the same one mentioned above (i.e.
RuntimeError or a custom subclass -- I don't care much about this detail).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141107/5756644c/attachment-0001.html>

From tjreedy at udel.edu  Sat Nov  8 01:53:04 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 07 Nov 2014 19:53:04 -0500
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CAP7+vJKpgpRGmgK3xF3fKpwSiPtpHt7HM6kffXCOpA9Hmn_6gw@mail.gmail.com>
References: <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
 <20141107112030.GA7768@ando.pearwood.info>
 <CAP7+vJKpgpRGmgK3xF3fKpwSiPtpHt7HM6kffXCOpA9Hmn_6gw@mail.gmail.com>
Message-ID: <m3jphk$rga$1@ger.gmane.org>

On 11/7/2014 12:10 PM, Guido van Rossum wrote:

> We'd still need to specify the eventual behavior. I propose the
> following as a strawman (after the deprecation period is finished and
> the __future__ import is no longer needed):

Double meanings are a constant problem.  The ambiguity of StopIteration 
(unintended bug indicator? intended stop signal?) is not needed within 
generator functions*.  An explicit 'raise StopIteration' is more easily 
written 'return'.  The rare. clever, 'yield expression' that means 
'either yield the value of expression or raise StopIteration' can be 
re-written

try:
    tem = expression
except StopIteration:
    return
yield tem

An anonymous generator expression with a 'yield or raise' expresssion 
can be re-written as a named generator function. In this context, 
allowing 'yield or raise' is too clever in that it breaks the desired 
(intended?) equivalence between 'list(genexp)' and '[genexp]'.  I agree 
that this is a bad tradeoff.

Such alternative could be documented.

* StopIteration raised within a __next__ method could be a bug, but a) 
this should be rare, b) the possibility cannot be eliminated, and c) it 
would be too expensive to not take StopIteration at face value.

> - If a StopIteration is about to bubble out of a generator frame, it is
> replaced with some other exception (maybe RuntimeError,

I support this particular replacement.  To me:
1. The proposal amounts to defining StopIteration escaping a running 
generator frame as a runtime error.

 > maybe a new custom Exception subclass,
 > but *not* deriving from StopIteration)

2. There are already more Exception classes than I can remember.
3. This error should be very rare after the transition.
4. A good error message will be at least as important as the class name.
5. A new, specific exception class is an invitation for people to write 
code that raises StopIteration so that its replacement can be caught in 
an except clause in the calling code.  If the existing builtin 
exceptions other than StopIteration are not enough to choose from, one 
can define a custom class.

 > which  causes the next() call (which invoked the generator) to fail,
 > passing that exception out.
> From then on it's just like any old exception.

Any replacement will be ordinary as soon as it is made, before it 
reaches next(), so I would move this last line up a couple of lines.

-- 
Terry Jan Reedy


From masklinn at masklinn.net  Sat Nov  8 14:05:07 2014
From: masklinn at masklinn.net (Masklinn)
Date: Sat, 8 Nov 2014 14:05:07 +0100
Subject: [Python-ideas] Store shared/locked state inside of the lock object
Message-ID: <884DB226-A3EC-4A89-8668-8C2B835FBC4A@masklinn.net>

Since the introduction of context managers, using the lock object itself
has become easier and safer (in that there's very little chance of
forgetting to release a lock anymore). A big annoyance remains though:
relation between lock and lockee remains informal and there is no 
structured way to indicate:

1. whether a state-set is protected by a lock
2. which lock protects the state-set
3. which state-set is protected by a lock

Some languages (e.g. Java) have tried to solve 2 and 3 using intrinsic
locks, however

* that does not solve 1 (and it becomes impossible to informally look
  for lock objects lying around and try to find corresponding 
  state-sets)
* it does not help much when state isn't coalesced in a single object,
  and for state hierarchies there is no way to express whether the whole
  hierarchy should be protected under the same lock (the root's) or each
  leaf should be locked individually. AFAIK intrinsic locks are not
  hierarchical themselves
* things get very awkward when using alternate concurrency-management 
  strategies such as explicit locks for security reason[0], the non-use
  of intrinsic locks has to be again documented informally

A fairly small technical change I've been considering to improve this
situation is to store the state-set inside the lock object, and only
yield it through the context manager: that the state-set is protected
by a lock is made obvious, and so is the relation between lock and
state-set. I was delighted to discover that Rust's sync::Mutex[1] and
sync::RWLock[2] follow a very similar strategy of owning the state-set

It's not a panacea, it doesn't fix issues of lock acquisition ordering
for instance, but I think it would go a fairly long way towards making
correct use of locks easier in Python.

The basic changes would be:
* threading.Lock and threading.RLock would now take an optional
  `data` parameter
* the parameter would be stored internally and not directly accessible
  by users of the lock
* that parameter would be returned by __enter__ and provided to the
  current "owner" of the lock

These should cause no forward-compatibility issues, Lock() currently
takes no arguments, and its __enter__ returns no value.

Possible improvements/questions/issues I can see:
* with Lock, the locked state would not be available unless using as
  a context manager. RLock could allow getting the protected state 
  only while locked by the current thread
* as-is, the scheme requires mutable state as it's not possible to
  swap the internal state entirely. RLock could allow 
  state-replacement when locked
* because Python has no ownership concept, it would be possible for
  a consumer to keep a reference to the locked state and manipulate
  it without locking

I don't consider the third issue to be huge, it could be mitigated by
yielding a proxy to the internal state only valid for the current lock
span. However I do not know if it's possible to create completely
transparent proxies in Python.

The first two issues are slightly more troubling and could be mitigated
by yielding not the state-set alone but a proxy object living only for
the current lock span (or both the state-set and a proxy object) that
proxy would allow getting and setting the state-set, and would error-out
after unlocking.

Lock.acquire() could be altered to return the same proxy (or (state-set,
proxy) pair) however it's currently defined as returning either True or
False so that'd be a backwards- incompatible change. An alternative
would be to add a new acquisition method or a new flag parameter
changing the return value from True to these on acquisition.

A drawback of this additional change is that it would require the lock
object to keep track of the current live proxy(/proxies for rlock?), and
invalidate it(/them) on unlocking, increasing its complexity much more
than just adding a new attribute.

Thoughts?

[0] https://www.securecoding.cert.org/confluence/display/java/LCK00-J.+Use+private+final+lock+objects+to+synchronize+classes+that+may+interact+with+untrusted+code
[1] http://doc.rust-lang.org/sync/struct.Mutex.html
[2] http://doc.rust-lang.org/sync/struct.RWLock.html

From phd at phdru.name  Sat Nov  8 14:38:56 2014
From: phd at phdru.name (Oleg Broytman)
Date: Sat, 8 Nov 2014 14:38:56 +0100
Subject: [Python-ideas] Store shared/locked state inside of the lock
 object
In-Reply-To: <884DB226-A3EC-4A89-8668-8C2B835FBC4A@masklinn.net>
References: <884DB226-A3EC-4A89-8668-8C2B835FBC4A@masklinn.net>
Message-ID: <20141108133856.GA1706@phdru.name>

Hi!

On Sat, Nov 08, 2014 at 02:05:07PM +0100, Masklinn <masklinn at masklinn.net> wrote:
> ... a proxy to the internal state only valid for the current lock
> span. However I do not know if it's possible to create completely
> transparent proxies in Python.

   I think it's possible -- in a C extension. See
http://www.egenix.com/products/python/mxBase/mxProxy/ for an example.

   (I don't have any opinion on the proposal.)

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From solipsis at pitrou.net  Sat Nov  8 16:04:25 2014
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 8 Nov 2014 16:04:25 +0100
Subject: [Python-ideas] Store shared/locked state inside of the lock
	object
References: <884DB226-A3EC-4A89-8668-8C2B835FBC4A@masklinn.net>
Message-ID: <20141108160425.1ffbd1f2@fsol>

On Sat, 8 Nov 2014 14:05:07 +0100
Masklinn <masklinn at masklinn.net> wrote:
> 
> A fairly small technical change I've been considering to improve this
> situation is to store the state-set inside the lock object, and only
> yield it through the context manager: that the state-set is protected
> by a lock is made obvious, and so is the relation between lock and
> state-set. I was delighted to discover that Rust's sync::Mutex[1] and
> sync::RWLock[2] follow a very similar strategy of owning the state-set
> 
> It's not a panacea, it doesn't fix issues of lock acquisition ordering
> for instance, but I think it would go a fairly long way towards making
> correct use of locks easier in Python.
> 
> The basic changes would be:
> * threading.Lock and threading.RLock would now take an optional
>   `data` parameter
> * the parameter would be stored internally and not directly accessible
>   by users of the lock
> * that parameter would be returned by __enter__ and provided to the
>   current "owner" of the lock

For clarity it should probably be a separate class (or set of classes),
e.g. DataLock.

Regards

Antoine.



From masklinn at masklinn.net  Sat Nov  8 19:42:50 2014
From: masklinn at masklinn.net (Masklinn)
Date: Sat, 8 Nov 2014 19:42:50 +0100
Subject: [Python-ideas] Store shared/locked state inside of the lock
	object
In-Reply-To: <20141108160425.1ffbd1f2@fsol>
References: <884DB226-A3EC-4A89-8668-8C2B835FBC4A@masklinn.net>
 <20141108160425.1ffbd1f2@fsol>
Message-ID: <A613EE59-FCEF-4E43-8F68-91D024F73C61@masklinn.net>

On 2014-11-08, at 16:04 , Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sat, 8 Nov 2014 14:05:07 +0100
> Masklinn <masklinn at masklinn.net> wrote:
>> 
>> A fairly small technical change I've been considering to improve this
>> situation is to store the state-set inside the lock object, and only
>> yield it through the context manager: that the state-set is protected
>> by a lock is made obvious, and so is the relation between lock and
>> state-set. I was delighted to discover that Rust's sync::Mutex[1] and
>> sync::RWLock[2] follow a very similar strategy of owning the state-set
>> 
>> It's not a panacea, it doesn't fix issues of lock acquisition ordering
>> for instance, but I think it would go a fairly long way towards making
>> correct use of locks easier in Python.
>> 
>> The basic changes would be:
>> * threading.Lock and threading.RLock would now take an optional
>> `data` parameter
>> * the parameter would be stored internally and not directly accessible
>> by users of the lock
>> * that parameter would be returned by __enter__ and provided to the
>> current "owner" of the lock
> 
> For clarity it should probably be a separate class (or set of classes),
> e.g. DataLock.

On the one hand this'd allow completely ignoring backwards-compatibility
issues wrt acquire() which is nice, on the other hand it would double
the number of lock types and introduce redundancy as DataLock would be
pretty much a strict superset of Lock, which is why I thought extending
Lock made sense.


From ethan at stoneleaf.us  Sat Nov  8 21:01:43 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 08 Nov 2014 12:01:43 -0800
Subject: [Python-ideas] Store shared/locked state inside of the lock
	object
In-Reply-To: <A613EE59-FCEF-4E43-8F68-91D024F73C61@masklinn.net>
References: <884DB226-A3EC-4A89-8668-8C2B835FBC4A@masklinn.net>
 <20141108160425.1ffbd1f2@fsol>
 <A613EE59-FCEF-4E43-8F68-91D024F73C61@masklinn.net>
Message-ID: <545E76A7.8060506@stoneleaf.us>

On 11/08/2014 10:42 AM, Masklinn wrote:
> On 2014-11-08, at 16:04 , Antoine Pitrou wrote:
>> On Sat, 8 Nov 2014 14:05:07 +0100 Masklinn wrote:
>>>
>>> A fairly small technical change I've been considering to improve this
>>> situation is to store the state-set inside the lock object, and only
>>> yield it through the context manager: that the state-set is protected
>>> by a lock is made obvious, and so is the relation between lock and
>>> state-set. I was delighted to discover that Rust's sync::Mutex and
>>> sync::RWLock follow a very similar strategy of owning the state-set
>>
>> For clarity it should probably be a separate class (or set of classes),
>> e.g. DataLock.
>
> On the one hand this'd allow completely ignoring backwards-compatibility
> issues wrt acquire() which is nice, on the other hand it would double
> the number of lock types and introduce redundancy as DataLock would be
> pretty much a strict superset of Lock, which is why I thought extending
> Lock made sense.

How does transforming existing locks into this kind of lock benefit existing code?  If existing code has to change to 
take advantage of the new features, said code could just as easily change the name of the lock it was using.

--
~Ethan~

From masklinn at masklinn.net  Sat Nov  8 23:09:57 2014
From: masklinn at masklinn.net (Masklinn)
Date: Sat, 8 Nov 2014 23:09:57 +0100
Subject: [Python-ideas] Store shared/locked state inside of the lock
	object
In-Reply-To: <545E76A7.8060506@stoneleaf.us>
References: <884DB226-A3EC-4A89-8668-8C2B835FBC4A@masklinn.net>
 <20141108160425.1ffbd1f2@fsol>
 <A613EE59-FCEF-4E43-8F68-91D024F73C61@masklinn.net>
 <545E76A7.8060506@stoneleaf.us>
Message-ID: <34263F5C-A623-418F-A24C-322EBA957470@masklinn.net>

On 2014-11-08, at 21:01 , Ethan Furman <ethan at stoneleaf.us> wrote:
> On 11/08/2014 10:42 AM, Masklinn wrote:
>> 
>> On the one hand this'd allow completely ignoring backwards-compatibility
>> issues wrt acquire() which is nice, on the other hand it would double
>> the number of lock types and introduce redundancy as DataLock would be
>> pretty much a strict superset of Lock, which is why I thought extending
>> Lock made sense.
> 
> How does transforming existing locks into this kind of lock benefit existing code?

I don't think I claimed that anywhere, as far as I think it makes
absolutely no difference to existing code.

>  If existing code has to change to take advantage of the new features, said code could just as easily change the name of the lock it was using.

Yes?

From ncoghlan at gmail.com  Tue Nov 11 13:21:12 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 11 Nov 2014 22:21:12 +1000
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <20141106224537.4fc55146@fsol>
References: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com>
 <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
 <20141106224537.4fc55146@fsol>
Message-ID: <CADiSq7dHmHOP+z-G-3jjzSgBVpKPek9mp3mkpN+HZ3hfz9izig@mail.gmail.com>

On 7 November 2014 07:45, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Thu, 6 Nov 2014 10:54:51 -0800
> Guido van Rossum <guido at python.org> wrote:
> >
> > If I had had the right foresight, I would have made it an error to
> > terminate a generator with a StopIteration, probably by raising another
> > exception chained to the StopIteration (so the traceback shows the place
> > where the StopIteration escaped).
> >
> > The question at hand is if we can fix this post-hoc, using clever tricks
> > and (of course) a deprecation period.
>
> Is there any point in fixing it? Who relies on such borderline cases?
>

It's not about people relying on the current behaviour (it can't be, since
we're talking about *changing* that behaviour), it's about "Errors should
never pass silently". That is, the problematic cases that (at least
arguably) may be worth fixing are those where:

1. StopIteration escapes from an expression (Error!)
2. Instead of causing a traceback, it terminates a containing generator
(Passing silently!)

As asyncio coroutines become more popular, I predict some serious head
scratching from StopIteration escaping an asynchronous operation and
getting thrown into a coroutine, which then terminates with a "return None"
rather than propagating the exception as you might otherwise expect.

The problem with this particular style of bug is that the only trace it
leaves is a generator iterator that terminates earlier than expected -
there's no traceback, log message, or any other indication of where
something strange may be happening.

Consider the following, from the original post in the thread:

    def izip(*args):
        iters = [iter(obj) for obj in args]
        while True:
            yield tuple([next(it) for it in iters])

The current behaviour of that construct is that, as soon as one of the
iterators is empty:

1. next(it) throws StopIteration
2. the list comprehension unwinds the frame, and allows the exception to
propagate
3. the generator iterator unwinds the frame, and allows the exception to
propagate
4. the code invoking the iterator sees StopIteration and assumes iteration
is complete

If you switch to the generator expression version instead, the flow control
becomes:

1. next(it) throws StopIteration
2. the generator expression unwinds the frame, and allows the exception to
propagate
3. the iteration inside the tuple constructor sees StopIteration and halts
4. the generator iterator never terminates

In that code, "next(it)" is a flow control operation akin to break (it
terminates the nearest enclosing generator iterator, just as break
terminates the nearest enclosing loop), but it's incredibly unclear that
this is the case - there's no local indication that it may raise
StopIteration, you need to "just know" that raising StopIteration is a
possibility.

Guido's suggestion is to consider looking for a viable way to break the
equivalence between "return" and "raise StopIteration" in generator
iterators - that way, the only way for the above code to work would be
through a more explicit version that clearly tracks the flow control.

Option 1 would be to assume we use a new exception, and are OK with folks
catching it explicitly

    from __future__ import explicit_generator_return
    def izip(*args):
        iters = [iter(obj) for obj in args]
        while True:
            try:
                t = tuple(next(it) for it in iters)
            except UncaughtStopIteration:
                return # One of the iterators has been exhausted
            yield t

Option 2 would be to assume the new exception is something generic like
RuntimeError, requiring the inner loop to be converted to statement form:


    def izip(*args):
        iters = [iter(obj) for obj in args]
        while True:
            entry = []
            for it in iters:
                try:
                    item = next(it)
                except StopIteration:
                    return # One of the iterators has been exhausted
                entry.append(item)
            yield tuple(entry)

With option 2, you can also still rely on the fact that list comprehensions
don't create a generator frame:

    def izip(*args):
        iters = [iter(obj) for obj in args]
        while True:
            try:
                entry = [next(it) for it in iters]
            except StopIteration:
                return # One of the iterators has been exhausted
            yield tuple(entry)

The upside of the option 2 spellings is they'll work on all currently
supported versions of Python, while the downside is the extra object
construction they have to do if you want to yield something other than a
list.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141111/7abf385e/attachment.html>

From ncoghlan at gmail.com  Tue Nov 11 13:32:29 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 11 Nov 2014 22:32:29 +1000
Subject: [Python-ideas] Store shared/locked state inside of the lock
	object
In-Reply-To: <A613EE59-FCEF-4E43-8F68-91D024F73C61@masklinn.net>
References: <884DB226-A3EC-4A89-8668-8C2B835FBC4A@masklinn.net>
 <20141108160425.1ffbd1f2@fsol>
 <A613EE59-FCEF-4E43-8F68-91D024F73C61@masklinn.net>
Message-ID: <CADiSq7cGVSnidTUn3gthpzE23HaM7M6u792-ZjYufAoZLd_j0Q@mail.gmail.com>

On 9 November 2014 04:42, Masklinn <masklinn at masklinn.net> wrote:

> On 2014-11-08, at 16:04 , Antoine Pitrou <solipsis at pitrou.net> wrote:
> > On Sat, 8 Nov 2014 14:05:07 +0100
> > Masklinn <masklinn at masklinn.net> wrote:
> >>
> >> A fairly small technical change I've been considering to improve this
> >> situation is to store the state-set inside the lock object, and only
> >> yield it through the context manager: that the state-set is protected
> >> by a lock is made obvious, and so is the relation between lock and
> >> state-set. I was delighted to discover that Rust's sync::Mutex[1] and
> >> sync::RWLock[2] follow a very similar strategy of owning the state-set
> >>
> >> It's not a panacea, it doesn't fix issues of lock acquisition ordering
> >> for instance, but I think it would go a fairly long way towards making
> >> correct use of locks easier in Python.
> >>
> >> The basic changes would be:
> >> * threading.Lock and threading.RLock would now take an optional
> >> `data` parameter
> >> * the parameter would be stored internally and not directly accessible
> >> by users of the lock
> >> * that parameter would be returned by __enter__ and provided to the
> >> current "owner" of the lock
> >
> > For clarity it should probably be a separate class (or set of classes),
> > e.g. DataLock.
>
> On the one hand this'd allow completely ignoring backwards-compatibility
> issues wrt acquire() which is nice, on the other hand it would double
> the number of lock types and introduce redundancy as DataLock would be
> pretty much a strict superset of Lock, which is why I thought extending
> Lock made sense.
>

Merging it into Lock would make Lock itself harder to learn and use, so the
separate DataLock notion sounds better to me - it keeps the documentation
separate, so folks that just want a basic Lock or RLock don't need to care
that DataLock exists.

It's also worth considering just always making DataLock recursive, and not
worrying about the non-recursive variant.

If you'd like to experiment with this as a 3rd party module, Graham
Dumpleton's wrapt library makes it possible to write almost completely
transparent proxies in pure Python:
http://wrapt.readthedocs.org/en/latest/wrappers.html

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141111/c0d17b33/attachment.html>

From ron3200 at gmail.com  Tue Nov 11 17:38:24 2014
From: ron3200 at gmail.com (Ron Adam)
Date: Tue, 11 Nov 2014 10:38:24 -0600
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CADiSq7dHmHOP+z-G-3jjzSgBVpKPek9mp3mkpN+HZ3hfz9izig@mail.gmail.com>
References: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
 <20141106224537.4fc55146@fsol>
 <CADiSq7dHmHOP+z-G-3jjzSgBVpKPek9mp3mkpN+HZ3hfz9izig@mail.gmail.com>
Message-ID: <m3te21$11t$1@ger.gmane.org>



On 11/11/2014 06:21 AM, Nick Coghlan wrote:
>
> Option 2 would be to assume the new exception is something generic like
> RuntimeError, requiring the inner loop to be converted to statement form:
>
>
>      def izip(*args):
>          iters = [iter(obj) for obj in args]
>          while True:


>              entry = []
>              for it in iters:
>                  try:
>                      item = next(it)
>                  except StopIteration:
>                      return # One of the iterators has been exhausted
>                  entry.append(item)
>              yield tuple(entry)

When I was experimenting with this earlier, I needed the try-except to 
catch the StopIteration exception in order to do a "break".  Which gave me 
the current behaviour of the generator expression being discussed.

Replacing break with return, as above, gave the expected behaviour, but 
also just removing the try-except and letting the StopIteration propagate 
out, worked as well.  That is, StopIteration(None) is equivalent to "return 
None" in the context above.

Can you point me to the source file that implements generator expression 
byte code or C code?  I wanted to look at that to see what was actually 
going on, but it looks like it may be a combination of a regular generator 
with a condition at some point to handle it slightly different.

Cheers,
    Ron













From greg.ewing at canterbury.ac.nz  Tue Nov 11 21:49:00 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 12 Nov 2014 09:49:00 +1300
Subject: [Python-ideas] Change how Generator Expressions handle
 StopIteration
In-Reply-To: <m3te21$11t$1@ger.gmane.org>
References: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
 <20141106224537.4fc55146@fsol>
 <CADiSq7dHmHOP+z-G-3jjzSgBVpKPek9mp3mkpN+HZ3hfz9izig@mail.gmail.com>
 <m3te21$11t$1@ger.gmane.org>
Message-ID: <5462763C.9030706@canterbury.ac.nz>

Ron Adam wrote:

> Can you point me to the source file that implements generator expression 
> byte code or C code?  I wanted to look at that to see what was actually 
> going on, but it looks like it may be a combination of a regular 
> generator with a condition at some point to handle it slightly different.

There's not much difference between a generator expression and
an ordinary generator. A generator expression gets compiled
into a call of an anonymous generator function.

 >>> from dis import dis
 >>> def f():
...  return list(x for x in (1, 2, 3))
...
   2           0 LOAD_GLOBAL              0 (list)
               3 LOAD_CONST               1 (<code object <genexpr> at 0x2a1d58, 
file "<stdin>", line 2>)
               6 MAKE_FUNCTION            0
               9 LOAD_CONST               5 ((1, 2, 3))
              12 GET_ITER
              13 CALL_FUNCTION            1
              16 CALL_FUNCTION            1
              19 RETURN_VALUE
 >>> dis(f.__code__.co_consts[1])
   2           0 LOAD_FAST                0 (.0)
         >>    3 FOR_ITER                11 (to 17)
               6 STORE_FAST               1 (x)
               9 LOAD_FAST                1 (x)
              12 YIELD_VALUE
              13 POP_TOP
              14 JUMP_ABSOLUTE            3
         >>   17 LOAD_CONST               0 (None)
              20 RETURN_VALUE

-- 
Greg


From stefano.borini at ferrara.linux.it  Tue Nov 11 22:51:24 2014
From: stefano.borini at ferrara.linux.it (Stefano Borini)
Date: Tue, 11 Nov 2014 22:51:24 +0100
Subject: [Python-ideas] hardware stdlib package (Was: Re: including psutil
 in the standard library?)
In-Reply-To: <CAFYqXL_0p-wtzVm0coVfRfEzb-yE3fUbXvjxwr9V6=me5aUuNw@mail.gmail.com>
References: <542319B1.2090606@ferrara.linux.it>
 <CAMpsgwbwLhy-WE6Jg0aSaLOCNCX4A4hx0gVC5VSw_Tqw9yoB6Q@mail.gmail.com>
 <59DEA399-5915-4562-8042-1372CF9D4353@yahoo.com>
 <CAFYqXL_0p-wtzVm0coVfRfEzb-yE3fUbXvjxwr9V6=me5aUuNw@mail.gmail.com>
Message-ID: <546284DC.1050104@ferrara.linux.it>



On 11/6/14 7:59 PM, Giampaolo Rodola' wrote:
> - psutil.disk_partitions()
> - psutil.net_connections()

Actually, I was thinking that some of these should be part of a standard 
library "hardware" module, where you can inquire about the underlying 
hardware in a platform independent way. I checked around and I was 
unable to find a definitive package for this, which is a pity, because I 
consider it an important functionality that should definitely be a 
battery included. What I have found is, however, plenty of sparse code 
and packages trying to address this very need.

To throw an idea, maybe it would be a good idea to aggregate all this 
code in a "hardware" package, stabilize it, and then include it as part 
of the standard library.

I created a tentative github

https://github.com/stefanoborini/hardware

I'll scout around to see how to attack the problem, but maybe the ML has 
some important feedback I am definitely interested in.

Stefano

From ncoghlan at gmail.com  Wed Nov 12 02:53:29 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 12 Nov 2014 11:53:29 +1000
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <m3te21$11t$1@ger.gmane.org>
References: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com>
 <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
 <20141106224537.4fc55146@fsol>
 <CADiSq7dHmHOP+z-G-3jjzSgBVpKPek9mp3mkpN+HZ3hfz9izig@mail.gmail.com>
 <m3te21$11t$1@ger.gmane.org>
Message-ID: <CADiSq7eHEn7t9rXE5ZgT4BHZXqMz_GFmFT+V64n7UQMmmb7enQ@mail.gmail.com>

On 12 November 2014 02:38, Ron Adam <ron3200 at gmail.com> wrote:

>
>
> Can you point me to the source file that implements generator expression
> byte code or C code?  I wanted to look at that to see what was actually
> going on, but it looks like it may be a combination of a regular generator
> with a condition at some point to handle it slightly different.
>

When a StopIteration escapes from a generator frame, it's currently just
propagated without modification. When the generator frame *returns*, the
generator object converts that to raising StopIteration:
https://hg.python.org/cpython/file/30a6c74ad87f/Objects/genobject.c#l117

An implementation of Guido's __future__ import idea would likely involve
setting a flag on generator iterators when they're instantiated to say
whether or not to intercept and convert StopIteration instances that escape
from the generator frame.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141112/e061593c/attachment.html>

From chris.barker at noaa.gov  Thu Nov 13 19:18:42 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 13 Nov 2014 10:18:42 -0800
Subject: [Python-ideas] A way to tell the gc it's OK to delete an object
	with a __del__
Message-ID: <CALGmxEJAaxKYwaBvUxSJpw2ze7--+pYfG22NNEpd1u45E7J6CQ@mail.gmail.com>

I've been poking into reference counting, circular references, etc, trying
to understand how the garbage collector works.

In practice, I'm trying to figure out a way to get a couple modules I work
with not to leak memory: one is an extension to wxPython that I wrote, that
creates a circular reference involving wx objects -- these, as is often the
case with wrappers, have a custom __del__ method.

The other is the python netcdf4 lib -- which has a circular reference built
in, so that Datasets (which map to a file on disk) know about Variables
(which map to data arrays within the file) and vice versa. We tried using
weak references here, but it turns out that some users like to work with
Variables without a reference to the Dataset, so that didn't work.

Anyway, in my poking around, I think I understand that the gc won't delete
"unreachable" objects if the have a custom __del__ method, because __del__
methods often handle resources that may depend on other objects, and bad
(really bad) things can happen if they are deleted out of order, and the gc
has no idea what order to delete.

However -- some __del__ methods are perfectly safe regardless of delete
order.

So it would be nice to be able to somehow let the gc know that a particular
object is safe to delete at any time.

Is there a technical reason this can't be done? It seems something as
simple as a __delete_in_any_order__ attribute would do it. Yes, this is
dangerous, but authors of such extension objects need to know what they are
doing anyway.

Is there a technical complication I'm not thinking of?

It seems if we could pull this off, we could eliminate one lasting ugly
wart in python memory management.

-Chris










-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141113/013c9ddf/attachment.html>

From amauryfa at gmail.com  Thu Nov 13 19:48:15 2014
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 13 Nov 2014 19:48:15 +0100
Subject: [Python-ideas] A way to tell the gc it's OK to delete an object
 with a __del__
In-Reply-To: <CALGmxEJAaxKYwaBvUxSJpw2ze7--+pYfG22NNEpd1u45E7J6CQ@mail.gmail.com>
References: <CALGmxEJAaxKYwaBvUxSJpw2ze7--+pYfG22NNEpd1u45E7J6CQ@mail.gmail.com>
Message-ID: <CAGmFidbnCrtLLvrTbQ=ct46zdRC-K3+Y_bVXpDZg7HcyEDJvsw@mail.gmail.com>

2014-11-13 19:18 GMT+01:00 Chris Barker <chris.barker at noaa.gov>:

> I've been poking into reference counting, circular references, etc, trying
> to understand how the garbage collector works.
>
> In practice, I'm trying to figure out a way to get a couple modules I work
> with not to leak memory: one is an extension to wxPython that I wrote, that
> creates a circular reference involving wx objects -- these, as is often the
> case with wrappers, have a custom __del__ method.
>
> The other is the python netcdf4 lib -- which has a circular reference
> built in, so that Datasets (which map to a file on disk) know about
> Variables (which map to data arrays within the file) and vice versa. We
> tried using weak references here, but it turns out that some users like to
> work with Variables without a reference to the Dataset, so that didn't work.
>
> Anyway, in my poking around, I think I understand that the gc won't delete
> "unreachable" objects if the have a custom __del__ method, because __del__
> methods often handle resources that may depend on other objects, and bad
> (really bad) things can happen if they are deleted out of order, and the gc
> has no idea what order to delete.
>
> However -- some __del__ methods are perfectly safe regardless of delete
> order.
>
> So it would be nice to be able to somehow let the gc know that a
> particular object is safe to delete at any time.
>
> Is there a technical reason this can't be done? It seems something as
> simple as a __delete_in_any_order__ attribute would do it. Yes, this is
> dangerous, but authors of such extension objects need to know what they are
> doing anyway.
>
> Is there a technical complication I'm not thinking of?
>
> It seems if we could pull this off, we could eliminate one lasting ugly
> wart in python memory management.
>

Did you try Python 3.4?
https://docs.python.org/3/whatsnew/3.4.html#pep-442-safe-object-finalization


-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141113/28942296/attachment.html>

From python at mrabarnett.plus.com  Thu Nov 13 19:50:13 2014
From: python at mrabarnett.plus.com (MRAB)
Date: Thu, 13 Nov 2014 18:50:13 +0000
Subject: [Python-ideas] A way to tell the gc it's OK to delete an object
 with a __del__
In-Reply-To: <CALGmxEJAaxKYwaBvUxSJpw2ze7--+pYfG22NNEpd1u45E7J6CQ@mail.gmail.com>
References: <CALGmxEJAaxKYwaBvUxSJpw2ze7--+pYfG22NNEpd1u45E7J6CQ@mail.gmail.com>
Message-ID: <5464FD65.9070507@mrabarnett.plus.com>

On 2014-11-13 18:18, Chris Barker wrote:
> I've been poking into reference counting, circular references, etc,
> trying to understand how the garbage collector works.
>
> In practice, I'm trying to figure out a way to get a couple modules I
> work with not to leak memory: one is an extension to wxPython that I
> wrote, that creates a circular reference involving wx objects -- these,
> as is often the case with wrappers, have a custom __del__ method.
>
> The other is the python netcdf4 lib -- which has a circular reference
> built in, so that Datasets (which map to a file on disk) know about
> Variables (which map to data arrays within the file) and vice versa. We
> tried using weak references here, but it turns out that some users like
> to work with Variables without a reference to the Dataset, so that
> didn't work.
>
> Anyway, in my poking around, I think I understand that the gc won't
> delete "unreachable" objects if the have a custom __del__ method,
> because __del__ methods often handle resources that may depend on other
> objects, and bad (really bad) things can happen if they are deleted out
> of order, and the gc has no idea what order to delete.
>
> However -- some __del__ methods are perfectly safe regardless of delete
> order.
>
> So it would be nice to be able to somehow let the gc know that a
> particular object is safe to delete at any time.
>
> Is there a technical reason this can't be done? It seems something as
> simple as a __delete_in_any_order__ attribute would do it. Yes, this is
> dangerous, but authors of such extension objects need to know what they
> are doing anyway.
>
> Is there a technical complication I'm not thinking of?
>
> It seems if we could pull this off, we could eliminate one lasting ugly
> wart in python memory management.
>
There's some code in this that you might find useful:

http://permalink.gmane.org/gmane.comp.python.ideas/20334

From guido at python.org  Thu Nov 13 21:04:52 2014
From: guido at python.org (Guido van Rossum)
Date: Thu, 13 Nov 2014 12:04:52 -0800
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CADiSq7dHmHOP+z-G-3jjzSgBVpKPek9mp3mkpN+HZ3hfz9izig@mail.gmail.com>
References: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
 <20141106224537.4fc55146@fsol>
 <CADiSq7dHmHOP+z-G-3jjzSgBVpKPek9mp3mkpN+HZ3hfz9izig@mail.gmail.com>
Message-ID: <CAP7+vJ+huTKoGHFpW04gVDdL5BZ64GWm2jNKA=duoN_FMHw2vA@mail.gmail.com>

Sorry for my radio silence in this thread -- it was a combination of a
conference and getting sick afterwards.

IIUC Nick prefers reusing an existing exception over inventing a new one
(to mean that a StopIteration was about to escape from a generator frame).
The motivation is that this encourages catching the StopIteration at the
source rather than relying on the new exception, thereby encouraging code
that also works with previous versions. I like this. Nobody has suggested
anything other than RuntimeError so let's go with that. I also like the
idea of a __future__ import to request the new behavior in Python 3.5, and
a warning if the error condition is detected while the new behavior isn't
requested. I assume the __future__ import applies to generator function
definitions, not calls. (But someone should reason this through before we
commit.)

Do we need a PEP for this or can we just go ahead? I haven't heard any
pushback on the basic premise that this is a problem we'd like to fix.

PS. If we decide not to go ahead with this, there's a small change to the
semantics of "return" in a generator that might allow asyncio to
distinguish between an intended return statement in the generator and an
accidentally escaping StopIteration -- the return case should use a newly
defined subclass of StopIteration. asyncio's _step() function can then tell
the two situations apart easily.

On Tue, Nov 11, 2014 at 4:21 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 7 November 2014 07:45, Antoine Pitrou <solipsis at pitrou.net> wrote:
>
>> On Thu, 6 Nov 2014 10:54:51 -0800
>> Guido van Rossum <guido at python.org> wrote:
>> >
>> > If I had had the right foresight, I would have made it an error to
>> > terminate a generator with a StopIteration, probably by raising another
>> > exception chained to the StopIteration (so the traceback shows the place
>> > where the StopIteration escaped).
>> >
>> > The question at hand is if we can fix this post-hoc, using clever tricks
>> > and (of course) a deprecation period.
>>
>> Is there any point in fixing it? Who relies on such borderline cases?
>>
>
> It's not about people relying on the current behaviour (it can't be, since
> we're talking about *changing* that behaviour), it's about "Errors should
> never pass silently". That is, the problematic cases that (at least
> arguably) may be worth fixing are those where:
>
> 1. StopIteration escapes from an expression (Error!)
> 2. Instead of causing a traceback, it terminates a containing generator
> (Passing silently!)
>
> As asyncio coroutines become more popular, I predict some serious head
> scratching from StopIteration escaping an asynchronous operation and
> getting thrown into a coroutine, which then terminates with a "return None"
> rather than propagating the exception as you might otherwise expect.
>
> The problem with this particular style of bug is that the only trace it
> leaves is a generator iterator that terminates earlier than expected -
> there's no traceback, log message, or any other indication of where
> something strange may be happening.
>
> Consider the following, from the original post in the thread:
>
>     def izip(*args):
>         iters = [iter(obj) for obj in args]
>         while True:
>             yield tuple([next(it) for it in iters])
>
> The current behaviour of that construct is that, as soon as one of the
> iterators is empty:
>
> 1. next(it) throws StopIteration
> 2. the list comprehension unwinds the frame, and allows the exception to
> propagate
> 3. the generator iterator unwinds the frame, and allows the exception to
> propagate
> 4. the code invoking the iterator sees StopIteration and assumes iteration
> is complete
>
> If you switch to the generator expression version instead, the flow
> control becomes:
>
> 1. next(it) throws StopIteration
> 2. the generator expression unwinds the frame, and allows the exception to
> propagate
> 3. the iteration inside the tuple constructor sees StopIteration and halts
> 4. the generator iterator never terminates
>
> In that code, "next(it)" is a flow control operation akin to break (it
> terminates the nearest enclosing generator iterator, just as break
> terminates the nearest enclosing loop), but it's incredibly unclear that
> this is the case - there's no local indication that it may raise
> StopIteration, you need to "just know" that raising StopIteration is a
> possibility.
>
> Guido's suggestion is to consider looking for a viable way to break the
> equivalence between "return" and "raise StopIteration" in generator
> iterators - that way, the only way for the above code to work would be
> through a more explicit version that clearly tracks the flow control.
>
> Option 1 would be to assume we use a new exception, and are OK with folks
> catching it explicitly
>
>     from __future__ import explicit_generator_return
>     def izip(*args):
>         iters = [iter(obj) for obj in args]
>         while True:
>             try:
>                 t = tuple(next(it) for it in iters)
>             except UncaughtStopIteration:
>                 return # One of the iterators has been exhausted
>             yield t
>
> Option 2 would be to assume the new exception is something generic like
> RuntimeError, requiring the inner loop to be converted to statement form:
>
>
>     def izip(*args):
>         iters = [iter(obj) for obj in args]
>         while True:
>             entry = []
>             for it in iters:
>                 try:
>                     item = next(it)
>                 except StopIteration:
>                     return # One of the iterators has been exhausted
>                 entry.append(item)
>             yield tuple(entry)
>
> With option 2, you can also still rely on the fact that list
> comprehensions don't create a generator frame:
>
>     def izip(*args):
>         iters = [iter(obj) for obj in args]
>         while True:
>             try:
>                 entry = [next(it) for it in iters]
>             except StopIteration:
>                 return # One of the iterators has been exhausted
>             yield tuple(entry)
>
> The upside of the option 2 spellings is they'll work on all currently
> supported versions of Python, while the downside is the extra object
> construction they have to do if you want to yield something other than a
> list.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141113/150ea9af/attachment-0001.html>

From timothy.c.delaney at gmail.com  Thu Nov 13 22:31:04 2014
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Fri, 14 Nov 2014 08:31:04 +1100
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CAP7+vJ+huTKoGHFpW04gVDdL5BZ64GWm2jNKA=duoN_FMHw2vA@mail.gmail.com>
References: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com>
 <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
 <20141106224537.4fc55146@fsol>
 <CADiSq7dHmHOP+z-G-3jjzSgBVpKPek9mp3mkpN+HZ3hfz9izig@mail.gmail.com>
 <CAP7+vJ+huTKoGHFpW04gVDdL5BZ64GWm2jNKA=duoN_FMHw2vA@mail.gmail.com>
Message-ID: <CAN8CLgkOH4szevxDBL9k0kveK3_-vWonZVxmnqg_4xu93XDgYA@mail.gmail.com>

On 14 November 2014 07:04, Guido van Rossum <guido at python.org> wrote:

>
> PS. If we decide not to go ahead with this, there's a small change to the
> semantics of "return" in a generator that might allow asyncio to
> distinguish between an intended return statement in the generator and an
> accidentally escaping StopIteration -- the return case should use a newly
> defined subclass of StopIteration. asyncio's _step() function can then tell
> the two situations apart easily.
>

I've only been peripherally following this, but that sounds like a good
idea whether or not the rest is implemented.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141114/168de948/attachment.html>

From ethan at stoneleaf.us  Thu Nov 13 23:20:16 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 13 Nov 2014 14:20:16 -0800
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CAP7+vJ+huTKoGHFpW04gVDdL5BZ64GWm2jNKA=duoN_FMHw2vA@mail.gmail.com>
References: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
 <20141106224537.4fc55146@fsol>
 <CADiSq7dHmHOP+z-G-3jjzSgBVpKPek9mp3mkpN+HZ3hfz9izig@mail.gmail.com>
 <CAP7+vJ+huTKoGHFpW04gVDdL5BZ64GWm2jNKA=duoN_FMHw2vA@mail.gmail.com>
Message-ID: <54652EA0.20204@stoneleaf.us>

On 11/13/2014 12:04 PM, Guido van Rossum wrote:
> 
> Do we need a PEP for this or can we just go ahead? I haven't heard any pushback on the basic premise that this is a
> problem we'd like to fix.

Given the confusion about what the problem is, and the possible fixes, I think a short PEP would be in order just so
everyone is on the same page.

I don't fully understand everything discussed so far (not a big user of generators), so I'll only throw my hat in as a
backup volunteer to write the PEP if no one one else is able to take the time.

--
~Ethan~

From chris.barker at noaa.gov  Fri Nov 14 00:18:57 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 13 Nov 2014 15:18:57 -0800
Subject: [Python-ideas] A way to tell the gc it's OK to delete an object
 with a __del__
In-Reply-To: <CAGmFidbnCrtLLvrTbQ=ct46zdRC-K3+Y_bVXpDZg7HcyEDJvsw@mail.gmail.com>
References: <CALGmxEJAaxKYwaBvUxSJpw2ze7--+pYfG22NNEpd1u45E7J6CQ@mail.gmail.com>
 <CAGmFidbnCrtLLvrTbQ=ct46zdRC-K3+Y_bVXpDZg7HcyEDJvsw@mail.gmail.com>
Message-ID: <CALGmxELo6-_CHi31SFc6YS6ROybPQFfuPqAeTg2i4Cvqmtdxfg@mail.gmail.com>

On Thu, Nov 13, 2014 at 10:48 AM, Amaury Forgeot d'Arc <amauryfa at gmail.com>
wrote:

> So it would be nice to be able to somehow let the gc know that a
>> particular object is safe to delete at any time.
>>
>> Did you try Python 3.4?
>
> https://docs.python.org/3/whatsnew/3.4.html#pep-442-safe-object-finalization
>
>
> no, I haven't -- very cool!

This may be the final push to get me to py3! (though not for wxPython quite
yet, sadly...)

I hadn't noticed that in any of the py3.4 announcements (nor had I looked
hard), but defiantly didn't find in in any of my googling about this issue.

Thanks,
  -Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141113/66e635ff/attachment.html>

From rosuav at gmail.com  Fri Nov 14 00:52:21 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 14 Nov 2014 10:52:21 +1100
Subject: [Python-ideas] A way to tell the gc it's OK to delete an object
 with a __del__
In-Reply-To: <CALGmxELo6-_CHi31SFc6YS6ROybPQFfuPqAeTg2i4Cvqmtdxfg@mail.gmail.com>
References: <CALGmxEJAaxKYwaBvUxSJpw2ze7--+pYfG22NNEpd1u45E7J6CQ@mail.gmail.com>
 <CAGmFidbnCrtLLvrTbQ=ct46zdRC-K3+Y_bVXpDZg7HcyEDJvsw@mail.gmail.com>
 <CALGmxELo6-_CHi31SFc6YS6ROybPQFfuPqAeTg2i4Cvqmtdxfg@mail.gmail.com>
Message-ID: <CAPTjJmq4KtsUHPJm=GC1oKfNSrgfuVuxKD4y5xHFNu7-K_Ly_A@mail.gmail.com>

On Fri, Nov 14, 2014 at 10:18 AM, Chris Barker <chris.barker at noaa.gov> wrote:
> This may be the final push to get me to py3! (though not for wxPython quite
> yet, sadly...)

Well, if you're coming here with language improvement suggestions,
you'll definitely want to be on Py3. There won't be any features added
to Py2. :)

ChrisA

From chris.barker at noaa.gov  Fri Nov 14 01:58:52 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 13 Nov 2014 16:58:52 -0800
Subject: [Python-ideas] A way to tell the gc it's OK to delete an object
 with a __del__
In-Reply-To: <CAPTjJmq4KtsUHPJm=GC1oKfNSrgfuVuxKD4y5xHFNu7-K_Ly_A@mail.gmail.com>
References: <CALGmxEJAaxKYwaBvUxSJpw2ze7--+pYfG22NNEpd1u45E7J6CQ@mail.gmail.com>
 <CAGmFidbnCrtLLvrTbQ=ct46zdRC-K3+Y_bVXpDZg7HcyEDJvsw@mail.gmail.com>
 <CALGmxELo6-_CHi31SFc6YS6ROybPQFfuPqAeTg2i4Cvqmtdxfg@mail.gmail.com>
 <CAPTjJmq4KtsUHPJm=GC1oKfNSrgfuVuxKD4y5xHFNu7-K_Ly_A@mail.gmail.com>
Message-ID: <CALGmxEKLcky66938R9CCfgnULFAO8aas=-US3Mq_OGrbFXPn4Q@mail.gmail.com>

On Thu, Nov 13, 2014 at 3:52 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Fri, Nov 14, 2014 at 10:18 AM, Chris Barker <chris.barker at noaa.gov>
> wrote:
> > This may be the final push to get me to py3! (though not for wxPython
> quite
> > yet, sadly...)
>
> Well, if you're coming here with language improvement suggestions,
> you'll definitely want to be on Py3. There won't be any features added
> to Py2. :)
>

yes, I'm fully aware of that -- still the the real kick to get me there --
maybe this is it.

I figure a language improvement will, at best get into the next minor
version -- then it will be years? before you can count on most people using
that version -- so proposing ideas is a whole different ball of wax than
what I do operationally.

-CHB



> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141113/ebc575e1/attachment-0001.html>

From chris.barker at noaa.gov  Fri Nov 14 00:20:46 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 13 Nov 2014 15:20:46 -0800
Subject: [Python-ideas] A way to tell the gc it's OK to delete an object
 with a __del__
In-Reply-To: <5464FD65.9070507@mrabarnett.plus.com>
References: <CALGmxEJAaxKYwaBvUxSJpw2ze7--+pYfG22NNEpd1u45E7J6CQ@mail.gmail.com>
 <5464FD65.9070507@mrabarnett.plus.com>
Message-ID: <CALGmxEK=g_0WjuJL+eUDEmMC3VFAS+47BXwb09TMN3BPbpbEEw@mail.gmail.com>

On Thu, Nov 13, 2014 at 10:50 AM, MRAB <python at mrabarnett.plus.com> wrote:

> There's some code in this that you might find useful:
>
> http://permalink.gmane.org/gmane.comp.python.ideas/20334


thanks -- that does look kind of like what I was thinking I'd need to do --
but in the long run what the gc to do it for me (which maybe it does in
py3.4)

But maybe I'll patch the netCDF lib with something like that -- folks will
be using py < 3.4 for a long time yet :-(

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141113/e158a477/attachment.html>

From rosuav at gmail.com  Fri Nov 14 02:25:36 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 14 Nov 2014 12:25:36 +1100
Subject: [Python-ideas] A way to tell the gc it's OK to delete an object
 with a __del__
In-Reply-To: <CALGmxEKLcky66938R9CCfgnULFAO8aas=-US3Mq_OGrbFXPn4Q@mail.gmail.com>
References: <CALGmxEJAaxKYwaBvUxSJpw2ze7--+pYfG22NNEpd1u45E7J6CQ@mail.gmail.com>
 <CAGmFidbnCrtLLvrTbQ=ct46zdRC-K3+Y_bVXpDZg7HcyEDJvsw@mail.gmail.com>
 <CALGmxELo6-_CHi31SFc6YS6ROybPQFfuPqAeTg2i4Cvqmtdxfg@mail.gmail.com>
 <CAPTjJmq4KtsUHPJm=GC1oKfNSrgfuVuxKD4y5xHFNu7-K_Ly_A@mail.gmail.com>
 <CALGmxEKLcky66938R9CCfgnULFAO8aas=-US3Mq_OGrbFXPn4Q@mail.gmail.com>
Message-ID: <CAPTjJmoiD-w2abWBEyQPG3EZQL7mCpyj3=oD5=tBB7yrt1ewmw@mail.gmail.com>

On Fri, Nov 14, 2014 at 11:58 AM, Chris Barker <chris.barker at noaa.gov> wrote:
> I figure a language improvement will, at best get into the next minor
> version -- then it will be years? before you can count on most people using
> that version -- so proposing ideas is a whole different ball of wax than
> what I do operationally.

The next minor version of Py3 is 3.5, which is currently scheduled for
a first alpha in February. If the feature misses that, the next minor
release will be I think about 18 months later, give or take.

The next minor version of Py2 is 2.8 and will not be happening.
http://legacy.python.org/dev/peps/pep-0404/

ChrisA

From tjreedy at udel.edu  Fri Nov 14 06:23:07 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 14 Nov 2014 00:23:07 -0500
Subject: [Python-ideas] A way to tell the gc it's OK to delete an object
	with a __del__
In-Reply-To: <CAPTjJmoiD-w2abWBEyQPG3EZQL7mCpyj3=oD5=tBB7yrt1ewmw@mail.gmail.com>
References: <CALGmxEJAaxKYwaBvUxSJpw2ze7--+pYfG22NNEpd1u45E7J6CQ@mail.gmail.com>
 <CAGmFidbnCrtLLvrTbQ=ct46zdRC-K3+Y_bVXpDZg7HcyEDJvsw@mail.gmail.com>
 <CALGmxELo6-_CHi31SFc6YS6ROybPQFfuPqAeTg2i4Cvqmtdxfg@mail.gmail.com>
 <CAPTjJmq4KtsUHPJm=GC1oKfNSrgfuVuxKD4y5xHFNu7-K_Ly_A@mail.gmail.com>
 <CALGmxEKLcky66938R9CCfgnULFAO8aas=-US3Mq_OGrbFXPn4Q@mail.gmail.com>
 <CAPTjJmoiD-w2abWBEyQPG3EZQL7mCpyj3=oD5=tBB7yrt1ewmw@mail.gmail.com>
Message-ID: <m443k5$6mt$2@ger.gmane.org>

On 11/13/2014 8:25 PM, Chris Angelico wrote:
> On Fri, Nov 14, 2014 at 11:58 AM, Chris Barker <chris.barker at noaa.gov> wrote:
>> I figure a language improvement will, at best get into the next minor
>> version -- then it will be years? before you can count on most people using
>> that version -- so proposing ideas is a whole different ball of wax than
>> what I do operationally.
>
> The next minor version of Py3 is 3.5, which is currently scheduled for
> a first alpha in February. If the feature misses that, the next minor
> release will be I think about 18 months later, give or take.

Beta1, scheduled for late May, is the new feature deadline, though 
sooner is better.

-- 
Terry Jan Reedy


From ncoghlan at gmail.com  Fri Nov 14 08:41:32 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 14 Nov 2014 17:41:32 +1000
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <54652EA0.20204@stoneleaf.us>
References: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com>
 <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
 <20141106224537.4fc55146@fsol>
 <CADiSq7dHmHOP+z-G-3jjzSgBVpKPek9mp3mkpN+HZ3hfz9izig@mail.gmail.com>
 <CAP7+vJ+huTKoGHFpW04gVDdL5BZ64GWm2jNKA=duoN_FMHw2vA@mail.gmail.com>
 <54652EA0.20204@stoneleaf.us>
Message-ID: <CADiSq7f-wux41ify3ccYHLnJ2ggosysyd7an5QZA0HotnJ9MzA@mail.gmail.com>

On 14 November 2014 08:20, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 11/13/2014 12:04 PM, Guido van Rossum wrote:
> >
> > Do we need a PEP for this or can we just go ahead? I haven't heard any
> pushback on the basic premise that this is a
> > problem we'd like to fix.
>
> Given the confusion about what the problem is, and the possible fixes, I
> think a short PEP would be in order just so
> everyone is on the same page.
>

Agreed, I think it's worth having an explanatory PEP at least for
documentation purposes. It also makes it easier to reference from the 3.5
What's New, as there may be some code that's relying on the current
behaviour that may need adjusting to use a custom exception type rather
than StopIteration (or else refactoring to avoid crossing a generator frame
boundary).

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141114/6bfbfb12/attachment.html>

From ncoghlan at gmail.com  Fri Nov 14 08:45:55 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 14 Nov 2014 17:45:55 +1000
Subject: [Python-ideas] A way to tell the gc it's OK to delete an object
 with a __del__
In-Reply-To: <CALGmxEK=g_0WjuJL+eUDEmMC3VFAS+47BXwb09TMN3BPbpbEEw@mail.gmail.com>
References: <CALGmxEJAaxKYwaBvUxSJpw2ze7--+pYfG22NNEpd1u45E7J6CQ@mail.gmail.com>
 <5464FD65.9070507@mrabarnett.plus.com>
 <CALGmxEK=g_0WjuJL+eUDEmMC3VFAS+47BXwb09TMN3BPbpbEEw@mail.gmail.com>
Message-ID: <CADiSq7fmEYk_jPw0VGvQ-zZ47NJrKqf_3Mw5_94qhdpg1j=1zQ@mail.gmail.com>

On 14 November 2014 09:20, Chris Barker <chris.barker at noaa.gov> wrote:

> On Thu, Nov 13, 2014 at 10:50 AM, MRAB <python at mrabarnett.plus.com> wrote:
>
>> There's some code in this that you might find useful:
>>
>> http://permalink.gmane.org/gmane.comp.python.ideas/20334
>
>
> thanks -- that does look kind of like what I was thinking I'd need to do
> -- but in the long run what the gc to do it for me (which maybe it does in
> py3.4)
>
> But maybe I'll patch the netCDF lib with something like that -- folks will
> be using py < 3.4 for a long time yet :-(
>

PyPy's GC design is (very) different, and allows finalisation of reference
cycles involving __del__ even in Python 2. That path naturally comes with
its own C extension compatibility challenges, though.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141114/44358365/attachment-0001.html>

From guido at python.org  Fri Nov 14 18:30:54 2014
From: guido at python.org (Guido van Rossum)
Date: Fri, 14 Nov 2014 09:30:54 -0800
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CADiSq7f-wux41ify3ccYHLnJ2ggosysyd7an5QZA0HotnJ9MzA@mail.gmail.com>
References: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
 <20141106224537.4fc55146@fsol>
 <CADiSq7dHmHOP+z-G-3jjzSgBVpKPek9mp3mkpN+HZ3hfz9izig@mail.gmail.com>
 <CAP7+vJ+huTKoGHFpW04gVDdL5BZ64GWm2jNKA=duoN_FMHw2vA@mail.gmail.com>
 <54652EA0.20204@stoneleaf.us>
 <CADiSq7f-wux41ify3ccYHLnJ2ggosysyd7an5QZA0HotnJ9MzA@mail.gmail.com>
Message-ID: <CAP7+vJ+4+bF+5Mzao4L6Dy+kirsDKH_FC4_uuaya0TUB3CLqMw@mail.gmail.com>

Someone please volunteer to write this PEP. I can review, but I need to
save my Python time to work on the type hinting PEP.

On Thu, Nov 13, 2014 at 11:41 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 14 November 2014 08:20, Ethan Furman <ethan at stoneleaf.us> wrote:
>
>> On 11/13/2014 12:04 PM, Guido van Rossum wrote:
>> >
>> > Do we need a PEP for this or can we just go ahead? I haven't heard any
>> pushback on the basic premise that this is a
>> > problem we'd like to fix.
>>
>> Given the confusion about what the problem is, and the possible fixes, I
>> think a short PEP would be in order just so
>> everyone is on the same page.
>>
>
> Agreed, I think it's worth having an explanatory PEP at least for
> documentation purposes. It also makes it easier to reference from the 3.5
> What's New, as there may be some code that's relying on the current
> behaviour that may need adjusting to use a custom exception type rather
> than StopIteration (or else refactoring to avoid crossing a generator frame
> boundary).
>
> Regards,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141114/7e3e80a6/attachment.html>

From chris.barker at noaa.gov  Fri Nov 14 19:01:58 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 14 Nov 2014 10:01:58 -0800
Subject: [Python-ideas] A way to tell the gc it's OK to delete an object
 with a __del__
In-Reply-To: <CADiSq7fmEYk_jPw0VGvQ-zZ47NJrKqf_3Mw5_94qhdpg1j=1zQ@mail.gmail.com>
References: <CALGmxEJAaxKYwaBvUxSJpw2ze7--+pYfG22NNEpd1u45E7J6CQ@mail.gmail.com>
 <5464FD65.9070507@mrabarnett.plus.com>
 <CALGmxEK=g_0WjuJL+eUDEmMC3VFAS+47BXwb09TMN3BPbpbEEw@mail.gmail.com>
 <CADiSq7fmEYk_jPw0VGvQ-zZ47NJrKqf_3Mw5_94qhdpg1j=1zQ@mail.gmail.com>
Message-ID: <CALGmxELusNgV5fAV2w3mHfLvzxmAhjemyJtXuexdfrSsy+ofsw@mail.gmail.com>

On Thu, Nov 13, 2014 at 11:45 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> PyPy's GC design is (very) different, and allows finalisation of reference
> cycles involving __del__ even in Python 2. That path naturally comes with
> its own C extension compatibility challenges, though.
>

yup -- as does IronPython and Jython.

But particularly for the scientific stack, switching to PyPy is a LOT
harder than switching to 3.4 !

I certainly wouldn't do it for the garbage collection ;-)

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141114/904fcd55/attachment.html>

From danilopeixoto5 at hotmail.com  Fri Nov 14 23:54:38 2014
From: danilopeixoto5 at hotmail.com (Danilo Peixoto)
Date: Fri, 14 Nov 2014 22:54:38 +0000
Subject: [Python-ideas] Python IDE (New IDE for Python scripting)
Message-ID: <BAY180-W70B5623A0FBFB2ECC076118B8C0@phx.gbl>

An alternative IDE with a new dark and modern interface.
 
http://s27.postimg.org/r4cjxf00z/Python_IDE.png (Concept)
 
Concept shows the Python IDE functionality to organize projects and files in own software and quick information on the status bar.
Some additional features:

AutocompleteFind and replaceIncrease and decrease line indentand more... 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141114/3ec797da/attachment.html>

From rajshorya at gmail.com  Fri Nov 14 23:59:14 2014
From: rajshorya at gmail.com (Shorya Raj)
Date: Sat, 15 Nov 2014 11:59:14 +1300
Subject: [Python-ideas] Python IDE (New IDE for Python scripting)
In-Reply-To: <BAY180-W70B5623A0FBFB2ECC076118B8C0@phx.gbl>
References: <BAY180-W70B5623A0FBFB2ECC076118B8C0@phx.gbl>
Message-ID: <CANk+dkP8GKmM4FBNw+DeUYsXVPJdTTERoxqBokDBbWCvFoew+A@mail.gmail.com>

Seems like it would "borrow" things from the PyCharm IDE. Looks quite nice
though.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141115/571ae2f9/attachment.html>

From rymg19 at gmail.com  Sat Nov 15 00:43:41 2014
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Fri, 14 Nov 2014 17:43:41 -0600
Subject: [Python-ideas] Python IDE (New IDE for Python scripting)
In-Reply-To: <BAY180-W70B5623A0FBFB2ECC076118B8C0@phx.gbl>
References: <BAY180-W70B5623A0FBFB2ECC076118B8C0@phx.gbl>
Message-ID: <CAO41-mNUQrVtx5CGuiWT8HFvRRviNvc2YfxQ_L7Mq7PySHT=jQ@mail.gmail.com>

Wow...that looks awesome!

On Fri, Nov 14, 2014 at 4:54 PM, Danilo Peixoto <danilopeixoto5 at hotmail.com>
wrote:

> An alternative IDE with a new dark and modern interface.
>
> http://s27.postimg.org/r4cjxf00z/Python_IDE.png (Concept)
>
> Concept shows the Python IDE functionality to organize projects and files
> in own software and quick information on the status bar.
> Some additional features:
>
>
>    - Autocomplete
>    - Find and replace
>    - Increase and decrease line indent
>    - and more...
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Ryan
If anybody ever asks me why I prefer C++ to C, my answer will be simple:
"It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
nul-terminated."
Personal reality distortion fields are immune to contradictory evidence. -
srean
Check out my website: http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141114/adf5b014/attachment-0001.html>

From steve at pearwood.info  Sat Nov 15 02:17:12 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 15 Nov 2014 12:17:12 +1100
Subject: [Python-ideas] Python IDE (New IDE for Python scripting)
In-Reply-To: <BAY180-W70B5623A0FBFB2ECC076118B8C0@phx.gbl>
References: <BAY180-W70B5623A0FBFB2ECC076118B8C0@phx.gbl>
Message-ID: <20141115011712.GJ2748@ando.pearwood.info>

On Fri, Nov 14, 2014 at 10:54:38PM +0000, Danilo Peixoto wrote:

> An alternative IDE with a new dark and modern interface.

What about it? Do you have a proposal for it?

This list is for ideas for the future development of Python the 
programming language and the standard library. How does your idea relate 
to the language or stdlib?

Instead of posting a screen shot, do you have a repo where we can look 
at the code?


-- 
Steven

From rosuav at gmail.com  Sat Nov 15 02:11:39 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 15 Nov 2014 12:11:39 +1100
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CAP7+vJ+4+bF+5Mzao4L6Dy+kirsDKH_FC4_uuaya0TUB3CLqMw@mail.gmail.com>
References: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com>
 <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
 <20141106224537.4fc55146@fsol>
 <CADiSq7dHmHOP+z-G-3jjzSgBVpKPek9mp3mkpN+HZ3hfz9izig@mail.gmail.com>
 <CAP7+vJ+huTKoGHFpW04gVDdL5BZ64GWm2jNKA=duoN_FMHw2vA@mail.gmail.com>
 <54652EA0.20204@stoneleaf.us>
 <CADiSq7f-wux41ify3ccYHLnJ2ggosysyd7an5QZA0HotnJ9MzA@mail.gmail.com>
 <CAP7+vJ+4+bF+5Mzao4L6Dy+kirsDKH_FC4_uuaya0TUB3CLqMw@mail.gmail.com>
Message-ID: <CAPTjJmpkvHejmvjZVk27Px3f6eGtUmMd_PvASf4TYhb1iSLn_A@mail.gmail.com>

On Sat, Nov 15, 2014 at 4:30 AM, Guido van Rossum <guido at python.org> wrote:
> Someone please volunteer to write this PEP. I can review, but I need to save
> my Python time to work on the type hinting PEP.

After the stunning success :) of my last such endeavour
(exception-catching expressions), I guess I could put my hand up for
this one, if there's no better volunteer forthcoming.

ChrisA

From guido at python.org  Sat Nov 15 03:18:01 2014
From: guido at python.org (Guido van Rossum)
Date: Fri, 14 Nov 2014 18:18:01 -0800
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CAPTjJmpkvHejmvjZVk27Px3f6eGtUmMd_PvASf4TYhb1iSLn_A@mail.gmail.com>
References: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
 <20141106224537.4fc55146@fsol>
 <CADiSq7dHmHOP+z-G-3jjzSgBVpKPek9mp3mkpN+HZ3hfz9izig@mail.gmail.com>
 <CAP7+vJ+huTKoGHFpW04gVDdL5BZ64GWm2jNKA=duoN_FMHw2vA@mail.gmail.com>
 <54652EA0.20204@stoneleaf.us>
 <CADiSq7f-wux41ify3ccYHLnJ2ggosysyd7an5QZA0HotnJ9MzA@mail.gmail.com>
 <CAP7+vJ+4+bF+5Mzao4L6Dy+kirsDKH_FC4_uuaya0TUB3CLqMw@mail.gmail.com>
 <CAPTjJmpkvHejmvjZVk27Px3f6eGtUmMd_PvASf4TYhb1iSLn_A@mail.gmail.com>
Message-ID: <CAP7+vJKZK0opWiPkjUt-Ep86Vdm4=-PuKm9mXGTxuuUCFBxL1w@mail.gmail.com>

Sold!

Thank you.

On Fri, Nov 14, 2014 at 5:11 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Sat, Nov 15, 2014 at 4:30 AM, Guido van Rossum <guido at python.org>
> wrote:
> > Someone please volunteer to write this PEP. I can review, but I need to
> save
> > my Python time to work on the type hinting PEP.
>
> After the stunning success :) of my last such endeavour
> (exception-catching expressions), I guess I could put my hand up for
> this one, if there's no better volunteer forthcoming.
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141114/264f4870/attachment.html>

From Steve.Dower at microsoft.com  Sat Nov 15 06:29:40 2014
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Sat, 15 Nov 2014 05:29:40 +0000
Subject: [Python-ideas] Python IDE (New IDE for Python scripting)
In-Reply-To: <BAY180-W70B5623A0FBFB2ECC076118B8C0@phx.gbl>
References: <BAY180-W70B5623A0FBFB2ECC076118B8C0@phx.gbl>
Message-ID: <7689848e2b034a4c9bcd01f05b78831b@DM2PR0301MB0734.namprd03.prod.outlook.com>

There are already many alternatives to Idle, which I assume you're wanting to replace, and most are freely available and/or cross platform. This is a more crowded market than you may think, and there's no reason why the standard Python distribution is the right place for it (even Idle feels like unnecessary these days, but it wins because of history).

Cheers,
Steve

Top-posted from my Windows Phone
________________________________
From: Danilo Peixoto<mailto:danilopeixoto5 at hotmail.com>
Sent: ?11/?14/?2014 14:55
To: python-ideas at python.org<mailto:python-ideas at python.org>
Subject: [Python-ideas] Python IDE (New IDE for Python scripting)

An alternative IDE with a new dark and modern interface.

http://s27.postimg.org/r4cjxf00z/Python_IDE.png (Concept)

Concept shows the Python IDE functionality to organize projects and files in own software and quick information on the status bar.
Some additional features:


  *   Autocomplete
  *   Find and replace
  *   Increase and decrease line indent
  *   and more...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141115/866892da/attachment.html>

From gokoproject at gmail.com  Sat Nov 15 07:12:58 2014
From: gokoproject at gmail.com (John Wong)
Date: Sat, 15 Nov 2014 01:12:58 -0500
Subject: [Python-ideas] Python IDE (New IDE for Python scripting)
In-Reply-To: <20141115011712.GJ2748@ando.pearwood.info>
References: <BAY180-W70B5623A0FBFB2ECC076118B8C0@phx.gbl>
 <20141115011712.GJ2748@ando.pearwood.info>
Message-ID: <CACCLA57r4YJiL-whYa0BvJaOtfWtqOyN5Tbk9VoBfo6P_BX_vw@mail.gmail.com>

On Fri, Nov 14, 2014 at 8:17 PM, Steven D'Aprano <steve at pearwood.info>
wrote:

>
> This list is for ideas for the future development of Python the
> programming language and the standard library. How does your idea relate
> to the language or stdlib?
>

https://docs.python.org/2/library/idle.html
My guess is he wants to replace IDLE with a potentially shinier IDE out of
the box, but obviously you made a good point proposal/code would be really
nice.
I suggest Danilo have a look at http://legacy.python.org/dev/peps/pep-0434/
and http://lwn.net/Articles/546188/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141115/7f9e3597/attachment-0001.html>

From tjreedy at udel.edu  Sat Nov 15 07:12:42 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 15 Nov 2014 01:12:42 -0500
Subject: [Python-ideas] Python IDE (New IDE for Python scripting)
In-Reply-To: <BAY180-W70B5623A0FBFB2ECC076118B8C0@phx.gbl>
References: <BAY180-W70B5623A0FBFB2ECC076118B8C0@phx.gbl>
Message-ID: <m46qt5$egs$1@ger.gmane.org>

On 11/14/2014 5:54 PM, Danilo Peixoto wrote:
> An alternative IDE with a new dark and modern interface.

Q. When does a recycled decades-old design become new?'

A. When an industry becomes part of the fashion industry.

Except for the frame provided by the OS via tkinter, an Idle user can 
switch to white and colors on black, though it would be a bit tedious 
switching all 13 foreground/background pairs.  Should a built-in 
retro-black theme be added?  There is actually an issue (7949) about 
Idle not working well with dark GTK/KDE color schemes, and I believe 
such a color scheme would help.

> http://s27.postimg.org/r4cjxf00z/Python_IDE.png (Concept)
>
> Concept shows the Python IDE functionality to organize projects and
> files in own software and quick information on the status bar.

Displaying the manual entry for a selected function is an interesting 
idea, though that space is too small.  A popup would be better, I think.

A function signature, however, would usually fit nicely.  So Idle could 
put signatures on the currently mostly unused status line in addition to 
an ephemeral calltip popup box (which also includes part of docstrings). 
  It could also be less fussy about cursor positioning before allowing 
calltip to enable access to the information.

Thank you for provoking 4 concrete ideas for Idle improvement, even 
though perhaps not your intention.

> Some additional features:
>
>   * Autocomplete
>   * Find and replace
>   * Increase and decrease line indent
>   * and more...

Already in Idle, though improvements are needed.

-- 
Terry Jan Reedy


From rosuav at gmail.com  Sat Nov 15 07:23:21 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 15 Nov 2014 17:23:21 +1100
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CADiSq7f-wux41ify3ccYHLnJ2ggosysyd7an5QZA0HotnJ9MzA@mail.gmail.com>
References: <CADiSq7eTSRtdADEaVpxvfSbMUV5D6ekdmVsJmRT9V7n-UBGfMg@mail.gmail.com>
 <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com>
 <m3661u$dc8$1@ger.gmane.org>
 <CAP7+vJ+7TLqA_pAsTDH-Kw36BjxYpTzfrOY6r2HwruzPc+sf2g@mail.gmail.com>
 <CADiSq7dLR0kw+LikSOkqemd6QJaQgyC5TczYHnWvcJLF_f2M_g@mail.gmail.com>
 <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
 <20141106224537.4fc55146@fsol>
 <CADiSq7dHmHOP+z-G-3jjzSgBVpKPek9mp3mkpN+HZ3hfz9izig@mail.gmail.com>
 <CAP7+vJ+huTKoGHFpW04gVDdL5BZ64GWm2jNKA=duoN_FMHw2vA@mail.gmail.com>
 <54652EA0.20204@stoneleaf.us>
 <CADiSq7f-wux41ify3ccYHLnJ2ggosysyd7an5QZA0HotnJ9MzA@mail.gmail.com>
Message-ID: <CAPTjJmoSpJ4USHFN+=YZFMDOr+MegXhcDg6R9w=cmimzMGDdOw@mail.gmail.com>

On Fri, Nov 14, 2014 at 6:41 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Agreed, I think it's worth having an explanatory PEP at least for
> documentation purposes.

Draft has been sent to the PEP editors, but if anyone wants to preview
it, it's currently here:

https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-xxx.txt

ChrisA

From steve at pearwood.info  Sat Nov 15 07:24:53 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 15 Nov 2014 17:24:53 +1100
Subject: [Python-ideas] Python IDE (New IDE for Python scripting)
In-Reply-To: <CACCLA57r4YJiL-whYa0BvJaOtfWtqOyN5Tbk9VoBfo6P_BX_vw@mail.gmail.com>
References: <BAY180-W70B5623A0FBFB2ECC076118B8C0@phx.gbl>
 <20141115011712.GJ2748@ando.pearwood.info>
 <CACCLA57r4YJiL-whYa0BvJaOtfWtqOyN5Tbk9VoBfo6P_BX_vw@mail.gmail.com>
Message-ID: <20141115062453.GK2748@ando.pearwood.info>

On Sat, Nov 15, 2014 at 01:12:58AM -0500, John Wong wrote:
> On Fri, Nov 14, 2014 at 8:17 PM, Steven D'Aprano <steve at pearwood.info>
> wrote:
> 
> >
> > This list is for ideas for the future development of Python the
> > programming language and the standard library. How does your idea relate
> > to the language or stdlib?
> >
> 
> https://docs.python.org/2/library/idle.html
> My guess is he wants to replace IDLE with a potentially shinier IDE
.....^^^^^

Yes, and that was my second guess too. (I am aware of IDLE :-) My first 
guess was that the OP was spamming us trying to get hits for his pet 
project.

In either case, if there is only a mock-up of what this new and improved 
IDE will look like, but no actual code yet, then we're just wasting time 
discussing it.


-- 
Steven

From steve at pearwood.info  Sat Nov 15 07:48:48 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 15 Nov 2014 17:48:48 +1100
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CAP7+vJ+huTKoGHFpW04gVDdL5BZ64GWm2jNKA=duoN_FMHw2vA@mail.gmail.com>
References: <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
 <20141106224537.4fc55146@fsol>
 <CADiSq7dHmHOP+z-G-3jjzSgBVpKPek9mp3mkpN+HZ3hfz9izig@mail.gmail.com>
 <CAP7+vJ+huTKoGHFpW04gVDdL5BZ64GWm2jNKA=duoN_FMHw2vA@mail.gmail.com>
Message-ID: <20141115064848.GL2748@ando.pearwood.info>

On Thu, Nov 13, 2014 at 12:04:52PM -0800, Guido van Rossum wrote:

> Do we need a PEP for this or can we just go ahead? I haven't heard any
> pushback on the basic premise that this is a problem we'd like to fix.

*puts hand up*

I'm not convinced that this is *a problem to be fixed*. At worst it is a 
"gotcha" to be aware of.

The current behaviour is simple to understand: raising StopIteration 
halts the generator, end of story. I'm still not sure that I understand 
what the proposed fix is (a PEP will be good to explain that), but if I 
have understood it correctly, it turns a simple concept like 
"StopIteration halts the generator" into something more complicated: 
some StopIterations will halt the generator, others will be chained to a 
new, unrelated exception.

Judging by the complete lack of questions about this on the tutor and 
python-list mailing lists, the slight difference in behaviour between 
generator expressions and comprehensions is not an issue in practice. 
I've seen people ask about the leakage of variables from comprehensions; 
I've never seen people ask about the different treatment of 
StopIteration.

I have, however, seen people *rely* on that different treatment. E.g. to 
implement a short-circuiting generator expression that exits when a 
condition is reached:

    (expr for x in sequence if cond or stop())

where stop() raises StopIteration and halts the generator. If this 
change goes ahead, it will break code that does this.

Having generator expressions and comprehensions behave exactly the same 
leads to the question, why do we have comprehensions? I guess the answer 
is "historical reasons", but from a pragmatic point of view being able 
to choose between

    [expr for x in sequence]
    list(expr for x in sequence)

depending on how you want StopIteration to be treated may be useful.


-- 
Steven

From rosuav at gmail.com  Sat Nov 15 08:06:27 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 15 Nov 2014 18:06:27 +1100
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <20141115064848.GL2748@ando.pearwood.info>
References: <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
 <20141106224537.4fc55146@fsol>
 <CADiSq7dHmHOP+z-G-3jjzSgBVpKPek9mp3mkpN+HZ3hfz9izig@mail.gmail.com>
 <CAP7+vJ+huTKoGHFpW04gVDdL5BZ64GWm2jNKA=duoN_FMHw2vA@mail.gmail.com>
 <20141115064848.GL2748@ando.pearwood.info>
Message-ID: <CAPTjJmpD-0aqy886BWuG+hdWzEj0V2bTA_7+QCwW-u=HhaGTpw@mail.gmail.com>

On Sat, Nov 15, 2014 at 5:48 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> I'm still not sure that I understand
> what the proposed fix is (a PEP will be good to explain that)

Draft PEP exists, and now has your concerns incorporated.

https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-xxx.txt

ChrisA

From rosuav at gmail.com  Sat Nov 15 10:29:47 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 15 Nov 2014 20:29:47 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
Message-ID: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>

PEP: 479
Title: Change StopIteration handling inside generators
Version: $Revision$
Last-Modified: $Date$
Author: Chris Angelico <rosuav at gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 15-Nov-2014
Python-Version: 3.5
Post-History: 15-Nov-2014


Abstract
========

This PEP proposes a semantic change to ``StopIteration`` when raised
inside a generator, unifying the behaviour of list comprehensions and
generator expressions somewhat.


Rationale
=========

The interaction of generators and ``StopIteration`` is currently
somewhat surprising, and can conceal obscure bugs.  An unexpected
exception should not result in subtly altered behaviour, but should
cause a noisy and easily-debugged traceback.  Currently,
``StopIteration`` can be absorbed by the generator construct.


Proposal
========

If a ``StopIteration`` is about to bubble out of a generator frame, it
is replaced with some other exception (maybe ``RuntimeError``, maybe a
new custom ``Exception`` subclass, but *not* deriving from
``StopIteration``) which causes the ``next()`` call (which invoked the
generator) to fail, passing that exception out.  From then on it's
just like any old exception. [3]_


Consequences to existing code
=============================

This change will affect existing code that depends on
``StopIteration`` bubbling up.  The pure Python reference
implementation of ``groupby`` [1]_ currently has comments "Exit on
``StopIteration``" where it is expected that the exception will
propagate and then be handled.  This will be unusual, but not unknown,
and such constructs will fail.

(Nick Coghlan comments: """If you wanted to factor out a helper
function that terminated the generator you'd have to do "return
yield from helper()" rather than just "helper()".""")

As this can break code, it is proposed to utilize the ``__future__``
mechanism to introduce this, finally making it standard in Python 3.6
or 3.7.


Alternate proposals
===================

Supplying a specific exception to raise on return
-------------------------------------------------

Nick Coghlan suggested a means of providing a specific
``StopIteration`` instance to the generator; if any other instance of
``StopIteration`` is raised, it is an error, but if that particular
one is raised, the generator has properly completed.


Making return-triggered StopIterations obvious
----------------------------------------------

For certain situations, a simpler and fully backward-compatible
solution may be sufficient: when a generator returns, instead of
raising ``StopIteration``, it raises a specific subclass of
``StopIteration`` which can then be detected.  If it is not that
subclass, it is an escaping exception rather than a return statement.


Criticism
=========

Unofficial and apocryphal statistics suggest that this is seldom, if
ever, a problem. [4]_  Code does exist which relies on the current
behaviour, and there is the concern that this would be unnecessary
code churn to achieve little or no gain.


References
==========

.. [1] Initial mailing list comment
   (https://mail.python.org/pipermail/python-ideas/2014-November/029906.html)

.. [2] Pure Python implementation of groupby
   (https://docs.python.org/3/library/itertools.html#itertools.groupby)

.. [3] Proposal by GvR
   (https://mail.python.org/pipermail/python-ideas/2014-November/029953.html)

.. [4] Response by Steven D'Aprano
   (https://mail.python.org/pipermail/python-ideas/2014-November/029994.html)


Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:

From rosuav at gmail.com  Sat Nov 15 13:57:34 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 15 Nov 2014 23:57:34 +1100
Subject: [Python-ideas] Change how Generator Expressions handle
	StopIteration
In-Reply-To: <CANc-5Uyp8EY9h+q893NnmhUk7Hc_SEaJdx_RhReOOm7jgaGPRg@mail.gmail.com>
References: <CAP7+vJLSkYYFWDqdU58Lj7qhDM3CWXpZd05aosyo7RiVr+tBfw@mail.gmail.com>
 <5457F5E9.8060205@canterbury.ac.nz>
 <CAP7+vJ+2QnWhJZRuu+XNcra5mJckN+Bd5MWva9kbfe4mGsePRg@mail.gmail.com>
 <CADiSq7fOQd+MK34NdNL8Mmjip2uXBfOz06xHhciKLvKe8s5-ZA@mail.gmail.com>
 <CADiSq7fhOr+xS+gu3tzJ4KZy9_cWSgrSCBo576=1mOy2iTJ9uA@mail.gmail.com>
 <20141106101525.GC3597@ando.pearwood.info>
 <CAP7+vJ+YdLJU+LRMivkD+2r02Sh5MEaZTah4SsffuaZ-p1MUnQ@mail.gmail.com>
 <20141106224537.4fc55146@fsol>
 <CADiSq7dHmHOP+z-G-3jjzSgBVpKPek9mp3mkpN+HZ3hfz9izig@mail.gmail.com>
 <CAP7+vJ+huTKoGHFpW04gVDdL5BZ64GWm2jNKA=duoN_FMHw2vA@mail.gmail.com>
 <20141115064848.GL2748@ando.pearwood.info>
 <CAPTjJmpD-0aqy886BWuG+hdWzEj0V2bTA_7+QCwW-u=HhaGTpw@mail.gmail.com>
 <CANc-5Uyp8EY9h+q893NnmhUk7Hc_SEaJdx_RhReOOm7jgaGPRg@mail.gmail.com>
Message-ID: <CAPTjJmpxSY4bVfGP6mXqSrrW5x9Mwm2hnMvqdXWT4A45QFUN3w@mail.gmail.com>

On Sat, Nov 15, 2014 at 11:36 PM, Skip Montanaro
<skip.montanaro at gmail.com> wrote:
> On Sat, Nov 15, 2014 at 1:06 AM, Chris Angelico <rosuav at gmail.com> wrote:
>>
>> Draft PEP exists, and now has your concerns incorporated.
>>
>> https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-xxx.txt
>
>
> Chris,
>
> 404 for me...

(replying on-list as this will affect everyone now)

It's now been allocated an actual number:

https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-0479.txt

ChrisA

From ncoghlan at gmail.com  Sat Nov 15 15:13:38 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 16 Nov 2014 00:13:38 +1000
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
Message-ID: <CADiSq7cTjSe2dn3-K-JsTxN92fEY=mq9AJEs_bk45GQUXUs6Pg@mail.gmail.com>

On 15 November 2014 19:29, Chris Angelico <rosuav at gmail.com> wrote:

> PEP: 479
> Title: Change StopIteration handling inside generators
> Version: $Revision$
> Last-Modified: $Date$
> Author: Chris Angelico <rosuav at gmail.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 15-Nov-2014
> Python-Version: 3.5
> Post-History: 15-Nov-2014
>
>
> Abstract
> ========
>
> This PEP proposes a semantic change to ``StopIteration`` when raised
> inside a generator, unifying the behaviour of list comprehensions and
> generator expressions somewhat.
>
>
> Rationale
> =========
>
> The interaction of generators and ``StopIteration`` is currently
> somewhat surprising, and can conceal obscure bugs.  An unexpected
> exception should not result in subtly altered behaviour, but should
> cause a noisy and easily-debugged traceback.  Currently,
> ``StopIteration`` can be absorbed by the generator construct.
>

Thanks for the write-up!


Proposal
> ========
>
> If a ``StopIteration`` is about to bubble out of a generator frame, it
> is replaced with some other exception (maybe ``RuntimeError``, maybe a
> new custom ``Exception`` subclass, but *not* deriving from
> ``StopIteration``) which causes the ``next()`` call (which invoked the
> generator) to fail, passing that exception out.  From then on it's
> just like any old exception. [3]_
>

[snip]


> Alternate proposals
> ===================
>
> Supplying a specific exception to raise on return
> -------------------------------------------------
>
> Nick Coghlan suggested a means of providing a specific
> ``StopIteration`` instance to the generator; if any other instance of
> ``StopIteration`` is raised, it is an error, but if that particular
> one is raised, the generator has properly completed.
>

I think you can skip mentioning this particular idea in the PEP - I didn't
like it even when I posted it, and both of Guido's ideas are much better :)


> Making return-triggered StopIterations obvious
> ----------------------------------------------
>
> For certain situations, a simpler and fully backward-compatible
> solution may be sufficient: when a generator returns, instead of
> raising ``StopIteration``, it raises a specific subclass of
> ``StopIteration`` which can then be detected.  If it is not that
> subclass, it is an escaping exception rather than a return statement.
>

There's an additional subtlety with this idea: if we add a new
GeneratorReturn exception as a subclass of StopIteration, then generator
iterators would likely also have to change to replace GeneratorReturn with
a regular StopIteration (chaining appropriately via __cause__, and copying
the return value across).

>From the point of view of calling "next()" directly (rather than
implicitly) this particular change makes it straightforward to distinguish
between "the generator I called just finished" and "something inside the
generator threw StopIteration". Due to the subclassing, implict next()
invocations (e.g. in for loops, comprehensions, and container constructors)
won't notice any difference.

With such a change, we would actually likely modify the following code in
contextlib._GeneratorContextManager.__exit__:

            try:
                self.gen.throw(exc_type, value, traceback)
                raise RuntimeError("generator didn't stop after throw()")
            except StopIteration as exc:
                # Generator suppressed the exception
                # unless it's a StopIteration instance we threw in
                return exc is not value
            except:
                if sys.exc_info()[1] is not value:
                    raise

To be the slightly more self-explanatory:

            try:
                self.gen.throw(type, value, traceback)
                raise RuntimeError("generator didn't stop after throw()")
            except GeneratorReturn:
                # Generator suppressed the exception
                return True
            except:
                if sys.exc_info()[1] is not value:
                    raise

The current proposal in the PEP actually doesn't let us simplify this
contextlib code, but rather means we would have to make it more complicated
to impedance match generator semantics with the context management
protocol. To handle that change, we'd have to make the code something like
the following (for clarity, I've assumed a new RuntimeError subclass,
rather than RuntimeError itself):

            try:
                self.gen.throw(exc_type, value, traceback)
                raise RuntimeError("generator didn't stop after throw()")
            except StopIteration as exc:
                # Could becomes "return True" once the __future__ becomes
the default
                return exc is not value
            except UnexpectedStopIteration as exc:
                if exc.__cause__ is not value:
                    raise
            except:
                if sys.exc_info()[1] is not value:
                    raise

I definitely see value in adding a GeneratorReturn subclass to be able to
tell the "returned" vs "raised StopIteration" cases apart from outside the
generator (the current dance in contextlib only works because we have
existing knowledge of the exact exception that was thrown in). I'm
substantially less convinced of the benefit of changing generators to no
longer suppress StopIteration. Yes, it's currently a rather odd corner
case, but changing it *will* break code (at the very least, anyone using an
old version of contextlib2, or who are otherwise relying on their own copy
of contextlib rather than standard library one).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141116/a5c601eb/attachment.html>

From rosuav at gmail.com  Sat Nov 15 15:37:57 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 16 Nov 2014 01:37:57 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CADiSq7cTjSe2dn3-K-JsTxN92fEY=mq9AJEs_bk45GQUXUs6Pg@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CADiSq7cTjSe2dn3-K-JsTxN92fEY=mq9AJEs_bk45GQUXUs6Pg@mail.gmail.com>
Message-ID: <CAPTjJmrTMDpmuJydWRF7Wuq=u9bSOfHNEFNFPxsC_-Ow5gbZxw@mail.gmail.com>

On Sun, Nov 16, 2014 at 1:13 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 15 November 2014 19:29, Chris Angelico <rosuav at gmail.com> wrote:
>> Nick Coghlan suggested a means of providing a specific
>> ``StopIteration`` instance to the generator; if any other instance of
>> ``StopIteration`` is raised, it is an error, but if that particular
>> one is raised, the generator has properly completed.
>
>
> I think you can skip mentioning this particular idea in the PEP - I didn't
> like it even when I posted it, and both of Guido's ideas are much better :)

Doesn't hurt to have some rejected alternates there :)

>> For certain situations, a simpler and fully backward-compatible
>> solution may be sufficient: when a generator returns, instead of
>> raising ``StopIteration``, it raises a specific subclass of
>> ``StopIteration`` which can then be detected.  If it is not that
>> subclass, it is an escaping exception rather than a return statement.
>
> There's an additional subtlety with this idea: if we add a new
> GeneratorReturn exception as a subclass of StopIteration, then generator
> iterators would likely also have to change to replace GeneratorReturn with a
> regular StopIteration (chaining appropriately via __cause__, and copying the
> return value across).

Would have to do so automatically, meaning this is no simpler than the
current proposal? Or would have to be always explicitly written to
handle it?

> I definitely see value in adding a GeneratorReturn subclass to be able to
> tell the "returned" vs "raised StopIteration" cases apart from outside the
> generator (the current dance in contextlib only works because we have
> existing knowledge of the exact exception that was thrown in). I'm
> substantially less convinced of the benefit of changing generators to no
> longer suppress StopIteration. Yes, it's currently a rather odd corner case,
> but changing it *will* break code (at the very least, anyone using an old
> version of contextlib2, or who are otherwise relying on their own copy of
> contextlib rather than standard library one).

This is why it's proposed to use __future__ to protect it. If anyone's
still using an old version of contextlib2 once 3.7 comes along, it'll
break; but is there any reason to use Python 3.7 with a contextlib
from elsewhere than its standard library? (I'm not familiar with
contextlib2 or what it offers.)

ChrisA

From ncoghlan at gmail.com  Sat Nov 15 16:21:36 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 16 Nov 2014 01:21:36 +1000
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmrTMDpmuJydWRF7Wuq=u9bSOfHNEFNFPxsC_-Ow5gbZxw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CADiSq7cTjSe2dn3-K-JsTxN92fEY=mq9AJEs_bk45GQUXUs6Pg@mail.gmail.com>
 <CAPTjJmrTMDpmuJydWRF7Wuq=u9bSOfHNEFNFPxsC_-Ow5gbZxw@mail.gmail.com>
Message-ID: <CADiSq7fJxOVr5LtPBAaJhJ54PMnsWOpbD1+DyPYJwTcJ7=kmtw@mail.gmail.com>

On 16 November 2014 00:37, Chris Angelico <rosuav at gmail.com> wrote:

> On Sun, Nov 16, 2014 at 1:13 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >> For certain situations, a simpler and fully backward-compatible
> >> solution may be sufficient: when a generator returns, instead of
> >> raising ``StopIteration``, it raises a specific subclass of
> >> ``StopIteration`` which can then be detected.  If it is not that
> >> subclass, it is an escaping exception rather than a return statement.
> >
> > There's an additional subtlety with this idea: if we add a new
> > GeneratorReturn exception as a subclass of StopIteration, then generator
> > iterators would likely also have to change to replace GeneratorReturn
> with a
> > regular StopIteration (chaining appropriately via __cause__, and copying
> the
> > return value across).
>
> Would have to do so automatically, meaning this is no simpler than the
> current proposal? Or would have to be always explicitly written to
> handle it?
>

When GeneratorReturn escaped a generator frame, the interpreter would
automatically convert it into an ordinary StopIteration instance.

It's still simpler because it won't need the __future__ dance (as it
doesn't involve any backwards incompatible changes).


>  > I definitely see value in adding a GeneratorReturn subclass to be able
> to
> > tell the "returned" vs "raised StopIteration" cases apart from outside
> the
> > generator (the current dance in contextlib only works because we have
> > existing knowledge of the exact exception that was thrown in). I'm
> > substantially less convinced of the benefit of changing generators to no
> > longer suppress StopIteration. Yes, it's currently a rather odd corner
> case,
> > but changing it *will* break code (at the very least, anyone using an old
> > version of contextlib2, or who are otherwise relying on their own copy of
> > contextlib rather than standard library one).
>
> This is why it's proposed to use __future__ to protect it.


Using __future__ still imposes a large cost on the community - docs need
updating, code that relies on the existing behaviour has to be changed,
developers need to adjust their mental models of how the language works.

There needs to be a practical payoff for those costs - and at the moment,
it's looking like we can actually get a reasonably large fraction of the
gain without most of the pain by instead pursuing Guido's idea of a
separate StopIteration subclass to distinguish returning from the outermost
generator frame from raising StopIteration elsewhere in the generator.


> If anyone's
> still using an old version of contextlib2 once 3.7 comes along, it'll
> break; but is there any reason to use Python 3.7 with a contextlib
> from elsewhere than its standard library?


Same reason folks use it now: consistent behaviour and features across a
range of Python versions.

However, that's not the key point - the key point is that working through
the exact changes that would need to be made in contextlib persuaded me
that I was wrong when I concluded that contextlib wouldn't be negatively
affected.

It's not much more complicated, but if we can find a fully supported
example like that in the standard library, what other things might folks be
doing with generators that *don't* fall into the category of "overly clever
code that we don't mind breaking"?

>
> (I'm not familiar with
> contextlib2 or what it offers.)
>

contexlib2 ~= 3.3 era contextlib that runs as far back as 2.6 (I initially
created it as a proving ground for the idea that eventually become
contextlib.ExitStack).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141116/8c31ae21/attachment.html>

From rosuav at gmail.com  Sat Nov 15 16:56:43 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 16 Nov 2014 02:56:43 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CADiSq7fJxOVr5LtPBAaJhJ54PMnsWOpbD1+DyPYJwTcJ7=kmtw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CADiSq7cTjSe2dn3-K-JsTxN92fEY=mq9AJEs_bk45GQUXUs6Pg@mail.gmail.com>
 <CAPTjJmrTMDpmuJydWRF7Wuq=u9bSOfHNEFNFPxsC_-Ow5gbZxw@mail.gmail.com>
 <CADiSq7fJxOVr5LtPBAaJhJ54PMnsWOpbD1+DyPYJwTcJ7=kmtw@mail.gmail.com>
Message-ID: <CAPTjJmom4vv70na+aRYWYH7gxRznA0XNoHp8c005KyJWzOxBiA@mail.gmail.com>

On Sun, Nov 16, 2014 at 2:21 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 16 November 2014 00:37, Chris Angelico <rosuav at gmail.com> wrote:
>>
>> On Sun, Nov 16, 2014 at 1:13 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> >> For certain situations, a simpler and fully backward-compatible
>> >> solution may be sufficient: when a generator returns, instead of
>> >> raising ``StopIteration``, it raises a specific subclass of
>> >> ``StopIteration`` which can then be detected.  If it is not that
>> >> subclass, it is an escaping exception rather than a return statement.
>> >
>> > There's an additional subtlety with this idea: if we add a new
>> > GeneratorReturn exception as a subclass of StopIteration, then generator
>> > iterators would likely also have to change to replace GeneratorReturn
>> > with a
>> > regular StopIteration (chaining appropriately via __cause__, and copying
>> > the
>> > return value across).
>>
>> Would have to do so automatically, meaning this is no simpler than the
>> current proposal? Or would have to be always explicitly written to
>> handle it?
>
> When GeneratorReturn escaped a generator frame, the interpreter would
> automatically convert it into an ordinary StopIteration instance.

Okay, let me see if I have this straight. When a 'return' statement
(including an implicit one at end-of-function) is encountered in any
function which contains a 'yield' statement, it is implemented as
"raise GeneratorReturn(value)" rather than as "raise
StopIteration(value)" which is the current behaviour. However, if any
GeneratorReturn would be raised in any way other than the 'return'
statement, it would magically become a StopIteration instead. Is that
correct?

This does sound simpler. All the magic is in the boundary of the
generator itself, nothing more. If a __next__ method raises either
StopIteration or GeneratorReturn, or if any other function raises
them, there's no special handling.

Question: How does it "become" StopIteration? Is a new instance of
StopIteration formed which copies in the other's ``value``? Is the
type of this exception magically altered? Or is it a brand new
exception with the __cause__ or __context__ set to carry the original?

>> If anyone's
>> still using an old version of contextlib2 once 3.7 comes along, it'll
>> break; but is there any reason to use Python 3.7 with a contextlib
>> from elsewhere than its standard library?
>
>
> Same reason folks use it now: consistent behaviour and features across a
> range of Python versions.
>
> However, that's not the key point - the key point is that working through
> the exact changes that would need to be made in contextlib persuaded me that
> I was wrong when I concluded that contextlib wouldn't be negatively
> affected.
>
> It's not much more complicated, but if we can find a fully supported example
> like that in the standard library, what other things might folks be doing
> with generators that *don't* fall into the category of "overly clever code
> that we don't mind breaking"?

Fair enough. The breakage is a known problem, though; whatever's done
is likely to cause at least some issues. If the alternate you describe
above will break less (or almost none), then it'll be the best option.

>> (I'm not familiar with
>> contextlib2 or what it offers.)
>
> contexlib2 ~= 3.3 era contextlib that runs as far back as 2.6 (I initially
> created it as a proving ground for the idea that eventually become
> contextlib.ExitStack).

Thanks, I figured it'd be like that. Since contextlib exists in 2.7,
is contextlib2 meant to be legacy support only?

ChrisA

From ncoghlan at gmail.com  Sat Nov 15 17:51:33 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 16 Nov 2014 02:51:33 +1000
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmom4vv70na+aRYWYH7gxRznA0XNoHp8c005KyJWzOxBiA@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CADiSq7cTjSe2dn3-K-JsTxN92fEY=mq9AJEs_bk45GQUXUs6Pg@mail.gmail.com>
 <CAPTjJmrTMDpmuJydWRF7Wuq=u9bSOfHNEFNFPxsC_-Ow5gbZxw@mail.gmail.com>
 <CADiSq7fJxOVr5LtPBAaJhJ54PMnsWOpbD1+DyPYJwTcJ7=kmtw@mail.gmail.com>
 <CAPTjJmom4vv70na+aRYWYH7gxRznA0XNoHp8c005KyJWzOxBiA@mail.gmail.com>
Message-ID: <CADiSq7cLPp6MS1xaHzZ5+gzfdaOkCzKRi+g+nsxAkhP3VX9Psw@mail.gmail.com>

On 16 November 2014 01:56, Chris Angelico <rosuav at gmail.com> wrote:

> On Sun, Nov 16, 2014 at 2:21 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > On 16 November 2014 00:37, Chris Angelico <rosuav at gmail.com> wrote:
> >>
> >> On Sun, Nov 16, 2014 at 1:13 AM, Nick Coghlan <ncoghlan at gmail.com>
> wrote:
> >> >> For certain situations, a simpler and fully backward-compatible
> >> >> solution may be sufficient: when a generator returns, instead of
> >> >> raising ``StopIteration``, it raises a specific subclass of
> >> >> ``StopIteration`` which can then be detected.  If it is not that
> >> >> subclass, it is an escaping exception rather than a return statement.
> >> >
> >> > There's an additional subtlety with this idea: if we add a new
> >> > GeneratorReturn exception as a subclass of StopIteration, then
> generator
> >> > iterators would likely also have to change to replace GeneratorReturn
> >> > with a
> >> > regular StopIteration (chaining appropriately via __cause__, and
> copying
> >> > the
> >> > return value across).
> >>
> >> Would have to do so automatically, meaning this is no simpler than the
> >> current proposal? Or would have to be always explicitly written to
> >> handle it?
> >
> > When GeneratorReturn escaped a generator frame, the interpreter would
> > automatically convert it into an ordinary StopIteration instance.
>
> Okay, let me see if I have this straight. When a 'return' statement
> (including an implicit one at end-of-function) is encountered in any
> function which contains a 'yield' statement, it is implemented as
> "raise GeneratorReturn(value)" rather than as "raise
> StopIteration(value)" which is the current behaviour. However, if any
> GeneratorReturn would be raised in any way other than the 'return'
> statement, it would magically become a StopIteration instead. Is that
> correct?
>

That's not quite how generators work. While the "returning from a generator
is equivalent to raise StopIteration" model is close enough that it's
functionally equivalent to the actual behaviour in most cases (with the
main difference being in how try/except blocks and context managers inside
the generator react), this particular PEP is a situation where it's
important to have a clear picture of the underlying details.

When you have a generator iterator (the thing you get back when calling a
generator function), there are two key components:

* the generator iterator object itself
* the generator frame where the code is running

When you call next(gi), you're invoking the __next__ method on the
*generator iterator*. It's that method which restarts evaluation of the
generator frame at the point where it last left off, and interprets any
results.

Now, there are three things that can happen as a result of that frame
evaluation:

    1. It hits a yield point. In that case, gi.__next__ returns the yielded
value.

    2. It can return from the frame. In that case. gi.__next__ creates a
*new* StopIteration instance (with an appropriate return value set) and
raises it

    3. It can throw an exception. In that case, gi.__next__ just allows it
to propagate out (including if it's StopIteration)

The following example illustrates the difference between cases 2 and 3 (in
both cases, there's a StopIteration that terminates the hidden loop inside
the list() call, the difference is in where that StopIteration is raised):

>>> def genreturn():
...     yield
...     try:
...         return
...     except:
...         print("No exception")
...         raise
...
>>> list(genreturn())
[None]

>>> def genraise():
...     yield
...     try:
...         raise StopIteration
...     except:
...         print("Exception!")
...         raise
...
>>> list(genraise())
Exception!
[None]

(The possible outcomes of gi.send() and gi.throw() are the same as those of
next(gi). gi.throw() has the novel variant where the exception thrown in
may propagate back out)

The two change proposals being discussed are as follows:

Current PEP (backwards incompatible): Change outcome 3 to convert
StopIteration to RuntimeError (or a new exception type). Nothing else
changes.

Alternative (backwards compatible): Change outcome 2 to raise
GeneratorReturn instead of StopIteration and outcome 3 to convert
GeneratorReturn to StopIteration.

The alternative *doesn't* do anything about the odd discrepancy between
comprehensions and generator expressions that started the previous thread.
It just adds a new capability where code that knows it's specifically
dealing with a generator (like contextlib or asyncio) can more easily tell
the difference between outcomes 2 and 3.


> This does sound simpler. All the magic is in the boundary of the
> generator itself, nothing more. If a __next__ method raises either
> StopIteration or GeneratorReturn, or if any other function raises
> them, there's no special handling.
>

All the magic is actually at the generator boundary regardless. The key
differences between the two proposals are the decision to keep
StopIteration as a common parent exception, and allow it to continue
propagating out of generator frames unmodified.


> Question: How does it "become" StopIteration? Is a new instance of
> StopIteration formed which copies in the other's ``value``? Is the
> type of this exception magically altered? Or is it a brand new
> exception with the __cause__ or __context__ set to carry the original?
>

I'd suggest used the exception chaining machinery and creating a new
exception with __cause__ and the generator return value set appropriately.

 >> (I'm not familiar with
> >> contextlib2 or what it offers.)
> >
> > contexlib2 ~= 3.3 era contextlib that runs as far back as 2.6 (I
> initially
> > created it as a proving ground for the idea that eventually become
> > contextlib.ExitStack).
>
> Thanks, I figured it'd be like that. Since contextlib exists in 2.7,
> is contextlib2 meant to be legacy support only?
>

contextlib has actually been around since 2.5, but some features (most
notably ExitStack) weren't added until much later. Like unittest2,
contextlib2 allows access to newer stdlib features on older versions (I
haven't used it as a testing ground for new ideas since ExitStack).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141116/d8e182f2/attachment-0001.html>

From Steve.Dower at microsoft.com  Sat Nov 15 19:20:06 2014
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Sat, 15 Nov 2014 18:20:06 +0000
Subject: [Python-ideas] PEP 479: Change StopIteration handling
	inside	generators
In-Reply-To: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
Message-ID: <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com>

Since this changes the behavior of an object instance, how can __future__ help? If the generator definition is in a library but the code that raises StopIteration to terminate it is passed in from the users code, how is the user supposed to select the behavior they want? (This sounds to me like a similar problem to adding 'from __future__ import py3_string' to Py2, which we discussed a while ago. Happy to be shown that it isn't.)

Cheers,
Steve

Top-posted from my Windows Phone
________________________________
From: Chris Angelico<mailto:rosuav at gmail.com>
Sent: ?11/?15/?2014 1:30
To: python-ideas<mailto:python-ideas at python.org>
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators

PEP: 479
Title: Change StopIteration handling inside generators
Version: $Revision$
Last-Modified: $Date$
Author: Chris Angelico <rosuav at gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 15-Nov-2014
Python-Version: 3.5
Post-History: 15-Nov-2014


Abstract
========

This PEP proposes a semantic change to ``StopIteration`` when raised
inside a generator, unifying the behaviour of list comprehensions and
generator expressions somewhat.


Rationale
=========

The interaction of generators and ``StopIteration`` is currently
somewhat surprising, and can conceal obscure bugs.  An unexpected
exception should not result in subtly altered behaviour, but should
cause a noisy and easily-debugged traceback.  Currently,
``StopIteration`` can be absorbed by the generator construct.


Proposal
========

If a ``StopIteration`` is about to bubble out of a generator frame, it
is replaced with some other exception (maybe ``RuntimeError``, maybe a
new custom ``Exception`` subclass, but *not* deriving from
``StopIteration``) which causes the ``next()`` call (which invoked the
generator) to fail, passing that exception out.  From then on it's
just like any old exception. [3]_


Consequences to existing code
=============================

This change will affect existing code that depends on
``StopIteration`` bubbling up.  The pure Python reference
implementation of ``groupby`` [1]_ currently has comments "Exit on
``StopIteration``" where it is expected that the exception will
propagate and then be handled.  This will be unusual, but not unknown,
and such constructs will fail.

(Nick Coghlan comments: """If you wanted to factor out a helper
function that terminated the generator you'd have to do "return
yield from helper()" rather than just "helper()".""")

As this can break code, it is proposed to utilize the ``__future__``
mechanism to introduce this, finally making it standard in Python 3.6
or 3.7.


Alternate proposals
===================

Supplying a specific exception to raise on return
-------------------------------------------------

Nick Coghlan suggested a means of providing a specific
``StopIteration`` instance to the generator; if any other instance of
``StopIteration`` is raised, it is an error, but if that particular
one is raised, the generator has properly completed.


Making return-triggered StopIterations obvious
----------------------------------------------

For certain situations, a simpler and fully backward-compatible
solution may be sufficient: when a generator returns, instead of
raising ``StopIteration``, it raises a specific subclass of
``StopIteration`` which can then be detected.  If it is not that
subclass, it is an escaping exception rather than a return statement.


Criticism
=========

Unofficial and apocryphal statistics suggest that this is seldom, if
ever, a problem. [4]_  Code does exist which relies on the current
behaviour, and there is the concern that this would be unnecessary
code churn to achieve little or no gain.


References
==========

.. [1] Initial mailing list comment
   (https://mail.python.org/pipermail/python-ideas/2014-November/029906.html)

.. [2] Pure Python implementation of groupby
   (https://docs.python.org/3/library/itertools.html#itertools.groupby)

.. [3] Proposal by GvR
   (https://mail.python.org/pipermail/python-ideas/2014-November/029953.html)

.. [4] Response by Steven D'Aprano
   (https://mail.python.org/pipermail/python-ideas/2014-November/029994.html)


Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:
_______________________________________________
Python-ideas mailing list
Python-ideas at python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141115/db0d9115/attachment.html>

From apalala at gmail.com  Sat Nov 15 22:00:28 2014
From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=)
Date: Sat, 15 Nov 2014 21:00:28 +0000
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com>
Message-ID: <CAN1YFWt2ePj8YkGGSHDa8==e1mDiUy9oBgpM-hgUVmiUgS6oVA@mail.gmail.com>

Exactly!

The current behavior is not only likely undesirable, but it is also
undocumented.

Even if parts of stdlib rely on the current behavior, there's no need for a
deprecation (read __future__) period.

Undocumented features may change any time, because are mostly about
implementation quirks (Isn't that rule documented somewhere in the Python
docs?).

In short:

-1 deprecation (__future__); no need, because nothing documented gets broken
+1 fix it now (3.5); the fix may be a change in the docs to validate the
current behavior, and deprecate it (Yuk!)
+1 Nick's design, which kind of leaves it the same and kind of fixes it

p.s. What about 2.7? This fix is *not* a new feature.

Cheers,

-- Juanca

On Sat Nov 15 2014 at 1:50:06 PM Steve Dower <Steve.Dower at microsoft.com>
wrote:

>   Since this changes the behavior of an object instance, how can
> __future__ help? If the generator definition is in a library but the code
> that raises StopIteration to terminate it is passed in from the users code,
> how is the user supposed to select the behavior they want? (This sounds to
> me like a similar problem to adding 'from __future__ import py3_string' to
> Py2, which we discussed a while ago. Happy to be shown that it isn't.)
>
> Cheers,
> Steve
>
> Top-posted from my Windows Phone
>  ------------------------------
> From: Chris Angelico <rosuav at gmail.com>
> Sent: ?11/?15/?2014 1:30
> To: python-ideas <python-ideas at python.org>
> Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
> generators
>
>   PEP: 479
> Title: Change StopIteration handling inside generators
> Version: $Revision$
> Last-Modified: $Date$
> Author: Chris Angelico <rosuav at gmail.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 15-Nov-2014
> Python-Version: 3.5
> Post-History: 15-Nov-2014
>
>
> Abstract
> ========
>
> This PEP proposes a semantic change to ``StopIteration`` when raised
> inside a generator, unifying the behaviour of list comprehensions and
> generator expressions somewhat.
>
>
> Rationale
> =========
>
> The interaction of generators and ``StopIteration`` is currently
> somewhat surprising, and can conceal obscure bugs.  An unexpected
> exception should not result in subtly altered behaviour, but should
> cause a noisy and easily-debugged traceback.  Currently,
> ``StopIteration`` can be absorbed by the generator construct.
>
>
> Proposal
> ========
>
> If a ``StopIteration`` is about to bubble out of a generator frame, it
> is replaced with some other exception (maybe ``RuntimeError``, maybe a
> new custom ``Exception`` subclass, but *not* deriving from
> ``StopIteration``) which causes the ``next()`` call (which invoked the
> generator) to fail, passing that exception out.  From then on it's
> just like any old exception. [3]_
>
>
> Consequences to existing code
> =============================
>
> This change will affect existing code that depends on
> ``StopIteration`` bubbling up.  The pure Python reference
> implementation of ``groupby`` [1]_ currently has comments "Exit on
> ``StopIteration``" where it is expected that the exception will
> propagate and then be handled.  This will be unusual, but not unknown,
> and such constructs will fail.
>
> (Nick Coghlan comments: """If you wanted to factor out a helper
> function that terminated the generator you'd have to do "return
> yield from helper()" rather than just "helper()".""")
>
> As this can break code, it is proposed to utilize the ``__future__``
> mechanism to introduce this, finally making it standard in Python 3.6
> or 3.7.
>
>
> Alternate proposals
> ===================
>
> Supplying a specific exception to raise on return
> -------------------------------------------------
>
> Nick Coghlan suggested a means of providing a specific
> ``StopIteration`` instance to the generator; if any other instance of
> ``StopIteration`` is raised, it is an error, but if that particular
> one is raised, the generator has properly completed.
>
>
> Making return-triggered StopIterations obvious
> ----------------------------------------------
>
> For certain situations, a simpler and fully backward-compatible
> solution may be sufficient: when a generator returns, instead of
> raising ``StopIteration``, it raises a specific subclass of
> ``StopIteration`` which can then be detected.  If it is not that
> subclass, it is an escaping exception rather than a return statement.
>
>
> Criticism
> =========
>
> Unofficial and apocryphal statistics suggest that this is seldom, if
> ever, a problem. [4]_  Code does exist which relies on the current
> behaviour, and there is the concern that this would be unnecessary
> code churn to achieve little or no gain.
>
>
> References
> ==========
>
> .. [1] Initial mailing list comment
>    (
> https://mail.python.org/pipermail/python-ideas/2014-November/029906.html)
>
> .. [2] Pure Python implementation of groupby
>    (https://docs.python.org/3/library/itertools.html#itertools.groupby)
>
> .. [3] Proposal by GvR
>    (
> https://mail.python.org/pipermail/python-ideas/2014-November/029953.html)
>
> .. [4] Response by Steven D'Aprano
>    (
> https://mail.python.org/pipermail/python-ideas/2014-November/029994.html)
>
>
> Copyright
> =========
>
> This document has been placed in the public domain.
>
>
>
> ..
>    Local Variables:
>    mode: indented-text
>    indent-tabs-mode: nil
>    sentence-end-double-space: t
>    fill-column: 70
>    coding: utf-8
>    End:
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>   _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141115/dc045fb8/attachment-0001.html>

From rosuav at gmail.com  Sat Nov 15 23:40:02 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 16 Nov 2014 09:40:02 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CADiSq7cLPp6MS1xaHzZ5+gzfdaOkCzKRi+g+nsxAkhP3VX9Psw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CADiSq7cTjSe2dn3-K-JsTxN92fEY=mq9AJEs_bk45GQUXUs6Pg@mail.gmail.com>
 <CAPTjJmrTMDpmuJydWRF7Wuq=u9bSOfHNEFNFPxsC_-Ow5gbZxw@mail.gmail.com>
 <CADiSq7fJxOVr5LtPBAaJhJ54PMnsWOpbD1+DyPYJwTcJ7=kmtw@mail.gmail.com>
 <CAPTjJmom4vv70na+aRYWYH7gxRznA0XNoHp8c005KyJWzOxBiA@mail.gmail.com>
 <CADiSq7cLPp6MS1xaHzZ5+gzfdaOkCzKRi+g+nsxAkhP3VX9Psw@mail.gmail.com>
Message-ID: <CAPTjJmpgo_BagGVM6sp0WSOK-y8ZX+mvz_APuJXw4tcAWG0msA@mail.gmail.com>

On Sun, Nov 16, 2014 at 3:51 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 16 November 2014 01:56, Chris Angelico <rosuav at gmail.com> wrote:
>> Okay, let me see if I have this straight. When a 'return' statement
>> (including an implicit one at end-of-function) is encountered in any
>> function which contains a 'yield' statement, it is implemented as
>> "raise GeneratorReturn(value)" rather than as "raise
>> StopIteration(value)" which is the current behaviour. However, if any
>> GeneratorReturn would be raised in any way other than the 'return'
>> statement, it would magically become a StopIteration instead. Is that
>> correct?
>
> When you call next(gi), you're invoking the __next__ method on the
> *generator iterator*. It's that method which restarts evaluation of the
> generator frame at the point where it last left off, and interprets any
> results.
>
> Now, there are three things that can happen as a result of that frame
> evaluation:
>
>     1. It hits a yield point. In that case, gi.__next__ returns the yielded
> value.
>
>     2. It can return from the frame. In that case. gi.__next__ creates a
> *new* StopIteration instance (with an appropriate return value set) and
> raises it
>
>     3. It can throw an exception. In that case, gi.__next__ just allows it
> to propagate out (including if it's StopIteration)

Thank you for explaining.
-- Cameron

In case others were also oversimplifying in their heads, I've
summarized the above into the PEP.

> (The possible outcomes of gi.send() and gi.throw() are the same as those of
> next(gi). gi.throw() has the novel variant where the exception thrown in may
> propagate back out)

Should that variant affect this proposal? What should happen if you
throw StopIteration or GeneratorReturn into a generator?

> The two change proposals being discussed are as follows:
>
> Current PEP (backwards incompatible): Change outcome 3 to convert
> StopIteration to RuntimeError (or a new exception type). Nothing else
> changes.
>
> Alternative (backwards compatible): Change outcome 2 to raise
> GeneratorReturn instead of StopIteration and outcome 3 to convert
> GeneratorReturn to StopIteration.
>
> The alternative *doesn't* do anything about the odd discrepancy between
> comprehensions and generator expressions that started the previous thread.
> It just adds a new capability where code that knows it's specifically
> dealing with a generator (like contextlib or asyncio) can more easily tell
> the difference between outcomes 2 and 3.

Text along these lines added to PEP, thanks!

>> Question: How does it "become" StopIteration? Is a new instance of
>> StopIteration formed which copies in the other's ``value``? Is the
>> type of this exception magically altered? Or is it a brand new
>> exception with the __cause__ or __context__ set to carry the original?
>
> I'd suggest used the exception chaining machinery and creating a new
> exception with __cause__ and the generator return value set appropriately.

Makes sense. If the __cause__ is noticed at all (ie this doesn't just
quietly stop a loop), it wants to be very noisy.

>> Thanks, I figured it'd be like that. Since contextlib exists in 2.7,
>> is contextlib2 meant to be legacy support only?
>
> contextlib has actually been around since 2.5, but some features (most
> notably ExitStack) weren't added until much later. Like unittest2,
> contextlib2 allows access to newer stdlib features on older versions (I
> haven't used it as a testing ground for new ideas since ExitStack).

If there is breakage from this, it would simply mean "older versions
of contextlib2 are not compatible with Python 3.7, please upgrade your
contextlib2" - several of the variants make it perfectly possible to
write cross-version-compatible code. I would hope that this remains
the case.

ChrisA

From rosuav at gmail.com  Sun Nov 16 00:06:18 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 16 Nov 2014 10:06:18 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CADiSq7cLPp6MS1xaHzZ5+gzfdaOkCzKRi+g+nsxAkhP3VX9Psw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CADiSq7cTjSe2dn3-K-JsTxN92fEY=mq9AJEs_bk45GQUXUs6Pg@mail.gmail.com>
 <CAPTjJmrTMDpmuJydWRF7Wuq=u9bSOfHNEFNFPxsC_-Ow5gbZxw@mail.gmail.com>
 <CADiSq7fJxOVr5LtPBAaJhJ54PMnsWOpbD1+DyPYJwTcJ7=kmtw@mail.gmail.com>
 <CAPTjJmom4vv70na+aRYWYH7gxRznA0XNoHp8c005KyJWzOxBiA@mail.gmail.com>
 <CADiSq7cLPp6MS1xaHzZ5+gzfdaOkCzKRi+g+nsxAkhP3VX9Psw@mail.gmail.com>
Message-ID: <CAPTjJmoQ2bF5ijFMF5Ooq7QHHpnGDTVe2jD7ii+ab9oFW3K6mQ@mail.gmail.com>

On Sun, Nov 16, 2014 at 3:51 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 16 November 2014 01:56, Chris Angelico <rosuav at gmail.com> wrote:
>> Okay, let me see if I have this straight. When a 'return' statement
>> (including an implicit one at end-of-function) is encountered in any
>> function which contains a 'yield' statement, it is implemented as
>> "raise GeneratorReturn(value)" rather than as "raise
>> StopIteration(value)" which is the current behaviour. However, if any
>> GeneratorReturn would be raised in any way other than the 'return'
>> statement, it would magically become a StopIteration instead. Is that
>> correct?
>
> When you call next(gi), you're invoking the __next__ method on the
> *generator iterator*. It's that method which restarts evaluation of the
> generator frame at the point where it last left off, and interprets any
> results.
>
> Now, there are three things that can happen as a result of that frame
> evaluation:
>
>     1. It hits a yield point. In that case, gi.__next__ returns the yielded
> value.
>
>     2. It can return from the frame. In that case. gi.__next__ creates a
> *new* StopIteration instance (with an appropriate return value set) and
> raises it
>
>     3. It can throw an exception. In that case, gi.__next__ just allows it
> to propagate out (including if it's StopIteration)

Thank you for explaining.
-- Cameron

In case others were also oversimplifying in their heads, I've
summarized the above into the PEP.

> (The possible outcomes of gi.send() and gi.throw() are the same as those of
> next(gi). gi.throw() has the novel variant where the exception thrown in may
> propagate back out)

Should that variant affect this proposal? What should happen if you
throw StopIteration or GeneratorReturn into a generator?

> The two change proposals being discussed are as follows:
>
> Current PEP (backwards incompatible): Change outcome 3 to convert
> StopIteration to RuntimeError (or a new exception type). Nothing else
> changes.
>
> Alternative (backwards compatible): Change outcome 2 to raise
> GeneratorReturn instead of StopIteration and outcome 3 to convert
> GeneratorReturn to StopIteration.
>
> The alternative *doesn't* do anything about the odd discrepancy between
> comprehensions and generator expressions that started the previous thread.
> It just adds a new capability where code that knows it's specifically
> dealing with a generator (like contextlib or asyncio) can more easily tell
> the difference between outcomes 2 and 3.

Text along these lines added to PEP, thanks!

>> Question: How does it "become" StopIteration? Is a new instance of
>> StopIteration formed which copies in the other's ``value``? Is the
>> type of this exception magically altered? Or is it a brand new
>> exception with the __cause__ or __context__ set to carry the original?
>
> I'd suggest used the exception chaining machinery and creating a new
> exception with __cause__ and the generator return value set appropriately.

Makes sense. If the __cause__ is noticed at all (ie this doesn't just
quietly stop a loop), it wants to be very noisy.

>> Thanks, I figured it'd be like that. Since contextlib exists in 2.7,
>> is contextlib2 meant to be legacy support only?
>
> contextlib has actually been around since 2.5, but some features (most
> notably ExitStack) weren't added until much later. Like unittest2,
> contextlib2 allows access to newer stdlib features on older versions (I
> haven't used it as a testing ground for new ideas since ExitStack).

If there is breakage from this, it would simply mean "older versions
of contextlib2 are not compatible with Python 3.7, please upgrade your
contextlib2" - several of the variants make it perfectly possible to
write cross-version-compatible code. I would hope that this remains
the case.

Latest version of PEP text incorporating the above changes:

https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-0479.txt

(My apologies if this email has gone through more than once. I'm
having major issues with my internet connection at the moment, and
delivery is failing and being retried. Hopefully it really *is*
failing, and not just saying so.)

ChrisA

From rosuav at gmail.com  Sun Nov 16 00:06:29 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 16 Nov 2014 10:06:29 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com>
Message-ID: <CAPTjJmqacEJqexWnUU1UBoC_VcsPKSoVw1_9uqYOyAXAbsLxYw@mail.gmail.com>

On Sun, Nov 16, 2014 at 5:20 AM, Steve Dower <Steve.Dower at microsoft.com> wrote:
> Since this changes the behavior of an object instance, how can __future__
> help? If the generator definition is in a library but the code that raises
> StopIteration to terminate it is passed in from the users code, how is the
> user supposed to select the behavior they want? (This sounds to me like a
> similar problem to adding 'from __future__ import py3_string' to Py2, which
> we discussed a while ago. Happy to be shown that it isn't.)

The behaviour selection would have to be based on the generator's
definition. This proposal, in all its variants, is about what happens
as the generator terminates; if you call on someone else's generator,
and that someone hasn't applied the __future__ directive, you'll be in
the current situation of not being able to distinguish 'return' from
'raise StopIteration'. But for your own generators, you can guarantee
that they're distinct.

ChrisA

From rosuav at gmail.com  Sun Nov 16 00:18:22 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 16 Nov 2014 10:18:22 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAN1YFWt2ePj8YkGGSHDa8==e1mDiUy9oBgpM-hgUVmiUgS6oVA@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com>
 <CAN1YFWt2ePj8YkGGSHDa8==e1mDiUy9oBgpM-hgUVmiUgS6oVA@mail.gmail.com>
Message-ID: <CAPTjJmqHfWNj3Y8t3MDP35n+jrV3YfXKCB=o93FCQhAerTF+xw@mail.gmail.com>

On Sun, Nov 16, 2014 at 8:00 AM, Juancarlo A?ez <apalala at gmail.com> wrote:
> Exactly!
>
> The current behavior is not only likely undesirable, but it is also
> undocumented.

I'm not sure about that. As Steven said, the current behaviour is simple:

1) When 'yield' is reached, a value is yielded.
2) When 'return' is reached, StopIteration is raised.
3) When an exception is raised, it is permitted to bubble up.

Whether that is *correct* or not is the point of this PEP, but it is
at least simple, and while it may not be documented per se, changing
it is likely to break code.

> Even if parts of stdlib rely on the current behavior, there's no need for a
> deprecation (read __future__) period.
>
> Undocumented features may change any time, because are mostly about
> implementation quirks (Isn't that rule documented somewhere in the Python
> docs?).

Maybe not, but let's get the proposal settled before figuring out how
much deprecation period is needed.

> In short:
>
> +1 fix it now (3.5); the fix may be a change in the docs to validate the
> current behavior, and deprecate it (Yuk!)

That would be pretty much what happens if the PEP is rejected: the
current behaviour will be effectively validated (at least to the
extent of "it's not worth the breakage").

> p.s. What about 2.7? This fix is *not* a new feature.

That ultimately depends on the release manager, but I would not aim
this at 2.7. Nick's proposal introduces a new exception type, which I
think cuts this out of 2.7 consideration right there; both active
proposals involve distinct changes to behaviour. I believe both of
them require *at a minimum* a feature release, and quite probably a
deprecation period (although that part may be arguable, as mentioned
above).

ChrisA

From rob.cliffe at btinternet.com  Sun Nov 16 01:49:01 2014
From: rob.cliffe at btinternet.com (Rob Cliffe)
Date: Sun, 16 Nov 2014 00:49:01 +0000
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
 generators
In-Reply-To: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
Message-ID: <5467F47D.2080307@btinternet.com>


On 15/11/2014 09:29, Chris Angelico wrote:
> PEP: 479
> Title: Change StopIteration handling inside generators
Thanks Chris for volunteering to do the donkey work again.
I think the final draft of the PEP would be much more comprehensible and 
useful if it contained one or more examples illustrating how the old and 
new behaviour differ.  I realise that this may not be appropriate until 
a consensus has been reached on what exactly the new behaviour *is*, so 
apologies if my grandmother already intends to suck this particular egg.
Rob Cliffe

From rosuav at gmail.com  Sun Nov 16 02:21:48 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 16 Nov 2014 12:21:48 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <5467F47D.2080307@btinternet.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <5467F47D.2080307@btinternet.com>
Message-ID: <CAPTjJmq-5HuyWrS1_=xYkReLheQc3qg2XoTYAg6uALGmBH2Q3w@mail.gmail.com>

On Sun, Nov 16, 2014 at 11:49 AM, Rob Cliffe <rob.cliffe at btinternet.com> wrote:
> On 15/11/2014 09:29, Chris Angelico wrote:
>>
>> PEP: 479
>> Title: Change StopIteration handling inside generators
>
> Thanks Chris for volunteering to do the donkey work again.
> I think the final draft of the PEP would be much more comprehensible and
> useful if it contained one or more examples illustrating how the old and new
> behaviour differ.  I realise that this may not be appropriate until a
> consensus has been reached on what exactly the new behaviour *is*, so
> apologies if my grandmother already intends to suck this particular egg.

Agreed. And agreed on the analysis; I can't add examples till I know
for sure what I'm adding examples _of_. The latest edit expanded on
the details of the proposals, so it now may be possible to consider
examples, but possibly we're still bikeshedding the nature of the
proposals themselves.

Correction. We're DEFINITELY still bikeshedding etc etc, but possibly
we're still doing so to the extent that it's not worth adding examples
yet. :)

ChrisA

From greg.ewing at canterbury.ac.nz  Sun Nov 16 23:05:01 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 17 Nov 2014 11:05:01 +1300
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
 generators
In-Reply-To: <CAPTjJmqacEJqexWnUU1UBoC_VcsPKSoVw1_9uqYOyAXAbsLxYw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com>
 <CAPTjJmqacEJqexWnUU1UBoC_VcsPKSoVw1_9uqYOyAXAbsLxYw@mail.gmail.com>
Message-ID: <54691F8D.1080703@canterbury.ac.nz>

Chris Angelico wrote:
> if you call on someone else's generator,
> and that someone hasn't applied the __future__ directive, you'll be in
> the current situation of not being able to distinguish 'return' from
> 'raise StopIteration'. But for your own generators, you can guarantee
> that they're distinct.

This suggests that the use of a __future__ directive is not
really appropriate, since it can affect code outside of the
module with the __future__ directive in it.

-- 
Greg

From steve at pearwood.info  Mon Nov 17 01:03:02 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 17 Nov 2014 11:03:02 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <54691F8D.1080703@canterbury.ac.nz>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com>
 <CAPTjJmqacEJqexWnUU1UBoC_VcsPKSoVw1_9uqYOyAXAbsLxYw@mail.gmail.com>
 <54691F8D.1080703@canterbury.ac.nz>
Message-ID: <20141117000302.GS2748@ando.pearwood.info>

On Mon, Nov 17, 2014 at 11:05:01AM +1300, Greg Ewing wrote:
> Chris Angelico wrote:
> >if you call on someone else's generator,
> >and that someone hasn't applied the __future__ directive, you'll be in
> >the current situation of not being able to distinguish 'return' from
> >'raise StopIteration'. But for your own generators, you can guarantee
> >that they're distinct.
> 
> This suggests that the use of a __future__ directive is not
> really appropriate, since it can affect code outside of the
> module with the __future__ directive in it.

I don't see how that is different from any other __future__ directive. 
They are all per-module, and if you gain access to an object from 
another module, it will behave as specified in the module that created 
it, not the module that imported it. How is this different?


-- 
Steven

From rosuav at gmail.com  Mon Nov 17 02:29:05 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 17 Nov 2014 12:29:05 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <20141117000302.GS2748@ando.pearwood.info>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com>
 <CAPTjJmqacEJqexWnUU1UBoC_VcsPKSoVw1_9uqYOyAXAbsLxYw@mail.gmail.com>
 <54691F8D.1080703@canterbury.ac.nz>
 <20141117000302.GS2748@ando.pearwood.info>
Message-ID: <CAPTjJmoHgSSL0L3Bh5_qXDB=mtd-6_X70WhD-_WqoXj3qzZuSw@mail.gmail.com>

On Mon, Nov 17, 2014 at 11:03 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Mon, Nov 17, 2014 at 11:05:01AM +1300, Greg Ewing wrote:
>> Chris Angelico wrote:
>> >if you call on someone else's generator,
>> >and that someone hasn't applied the __future__ directive, you'll be in
>> >the current situation of not being able to distinguish 'return' from
>> >'raise StopIteration'. But for your own generators, you can guarantee
>> >that they're distinct.
>>
>> This suggests that the use of a __future__ directive is not
>> really appropriate, since it can affect code outside of the
>> module with the __future__ directive in it.
>
> I don't see how that is different from any other __future__ directive.
> They are all per-module, and if you gain access to an object from
> another module, it will behave as specified in the module that created
> it, not the module that imported it. How is this different?

Well, let's see. For feature in sorted(__future__.all_feature_names):

absolute_import: Affects implementation of a keyword
barry_as_FLUFL: Not entirely sure what this one actually accomplishes. :)
division: Changes the meaning of one operator.
generators: Introduces a keyword
nested_scopes: Alters the compilation of source to byte-code(?)
print_function: Removes a keyword
unicode_literals: Alters the type used for literals
with_statement: Introduces a keyword

Apart from the joke, it seems that every __future__ directive is there
to affect the compilation, not execution, of its module: that is, once
a module has been compiled to .pyc, it shouldn't matter whether it
used __future__ or not. Regardless of unicode_literals, you can create
bytes literals with b'asdf' and unicode literals with u'asdf'. I'm not
entirely sure about division (can you call on true-division without
the future directive?), but in any case, it's all done at compilation
time, as can be seen interactively:

Python 2.7.3 (default, Mar 13 2014, 11:03:55)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> def division1(x,y):
...     return x/y, x//y, x%y
...
>>> from __future__ import division
>>> def division2(x,y):
...     return x/y, x//y, x%y
...
>>> import dis
>>> dis.dis(division1)
  2           0 LOAD_FAST                0 (x)
              3 LOAD_FAST                1 (y)
              6 BINARY_DIVIDE
              7 LOAD_FAST                0 (x)
             10 LOAD_FAST                1 (y)
             13 BINARY_FLOOR_DIVIDE
             14 LOAD_FAST                0 (x)
             17 LOAD_FAST                1 (y)
             20 BINARY_MODULO
             21 BUILD_TUPLE              3
             24 RETURN_VALUE
>>> dis.dis(division2)
  2           0 LOAD_FAST                0 (x)
              3 LOAD_FAST                1 (y)
              6 BINARY_TRUE_DIVIDE
              7 LOAD_FAST                0 (x)
             10 LOAD_FAST                1 (y)
             13 BINARY_FLOOR_DIVIDE
             14 LOAD_FAST                0 (x)
             17 LOAD_FAST                1 (y)
             20 BINARY_MODULO
             21 BUILD_TUPLE              3
             24 RETURN_VALUE

So to make this consistent with all other __future__ directives, there
would need to be some kind of safe way to define this: perhaps an
attribute on the generator object. Something like this:

>>> def gen(x):
...     yield x
...     raise StopIteration(42)
...
>>> g=gen(123)
>>> list(g)
[123]
>>> gen.__distinguish_returns__ = True
>>> g=gen(123)
>>> list(g)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in gen
StopIteration: 42

The attribute on the function would be what affects behaviour; the
__future__ directive applies that attribute to all generator functions
in its module (including genexprs). Once the __future__ directive
becomes automatic, the attribute can and will be dropped - any code
which interrogates it MUST be prepared to stop interrogating it once
the feature applies to all modules.

Does that sound reasonable? Should it be added to the PEP?

ChrisA

From guido at python.org  Mon Nov 17 02:58:34 2014
From: guido at python.org (Guido van Rossum)
Date: Sun, 16 Nov 2014 17:58:34 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmoHgSSL0L3Bh5_qXDB=mtd-6_X70WhD-_WqoXj3qzZuSw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com>
 <CAPTjJmqacEJqexWnUU1UBoC_VcsPKSoVw1_9uqYOyAXAbsLxYw@mail.gmail.com>
 <54691F8D.1080703@canterbury.ac.nz> <20141117000302.GS2748@ando.pearwood.info>
 <CAPTjJmoHgSSL0L3Bh5_qXDB=mtd-6_X70WhD-_WqoXj3qzZuSw@mail.gmail.com>
Message-ID: <CAP7+vJJ0m_PipGY0S172Nt3cZvJwCQFmOhxNCBioggFAYzU4kw@mail.gmail.com>

(I'm catching up on this thread from the end.)

On Sun, Nov 16, 2014 at 5:29 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Mon, Nov 17, 2014 at 11:03 AM, Steven D'Aprano <steve at pearwood.info>
> wrote:
> > On Mon, Nov 17, 2014 at 11:05:01AM +1300, Greg Ewing wrote:
> >> Chris Angelico wrote:
> >> >if you call on someone else's generator,
> >> >and that someone hasn't applied the __future__ directive, you'll be in
> >> >the current situation of not being able to distinguish 'return' from
> >> >'raise StopIteration'. But for your own generators, you can guarantee
> >> >that they're distinct.
> >>
> >> This suggests that the use of a __future__ directive is not
> >> really appropriate, since it can affect code outside of the
> >> module with the __future__ directive in it.
> >
> > I don't see how that is different from any other __future__ directive.
> > They are all per-module, and if you gain access to an object from
> > another module, it will behave as specified in the module that created
> > it, not the module that imported it. How is this different?
>
> Well, let's see. For feature in sorted(__future__.all_feature_names):
>
> absolute_import: Affects implementation of a keyword
> barry_as_FLUFL: Not entirely sure what this one actually accomplishes. :)
> division: Changes the meaning of one operator.
> generators: Introduces a keyword
> nested_scopes: Alters the compilation of source to byte-code(?)
> print_function: Removes a keyword
> unicode_literals: Alters the type used for literals
> with_statement: Introduces a keyword
>
> Apart from the joke, it seems that every __future__ directive is there
> to affect the compilation, not execution, of its module: that is, once
> a module has been compiled to .pyc, it shouldn't matter whether it
> used __future__ or not. Regardless of unicode_literals, you can create
> bytes literals with b'asdf' and unicode literals with u'asdf'. I'm not
> entirely sure about division (can you call on true-division without
> the future directive?), but in any case, it's all done at compilation
> time, as can be seen interactively:
>
> Python 2.7.3 (default, Mar 13 2014, 11:03:55)
> [GCC 4.7.2] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> def division1(x,y):
> ...     return x/y, x//y, x%y
> ...
> >>> from __future__ import division
> >>> def division2(x,y):
> ...     return x/y, x//y, x%y
> ...
> >>> import dis
> >>> dis.dis(division1)
>   2           0 LOAD_FAST                0 (x)
>               3 LOAD_FAST                1 (y)
>               6 BINARY_DIVIDE
>               7 LOAD_FAST                0 (x)
>              10 LOAD_FAST                1 (y)
>              13 BINARY_FLOOR_DIVIDE
>              14 LOAD_FAST                0 (x)
>              17 LOAD_FAST                1 (y)
>              20 BINARY_MODULO
>              21 BUILD_TUPLE              3
>              24 RETURN_VALUE
> >>> dis.dis(division2)
>   2           0 LOAD_FAST                0 (x)
>               3 LOAD_FAST                1 (y)
>               6 BINARY_TRUE_DIVIDE
>               7 LOAD_FAST                0 (x)
>              10 LOAD_FAST                1 (y)
>              13 BINARY_FLOOR_DIVIDE
>              14 LOAD_FAST                0 (x)
>              17 LOAD_FAST                1 (y)
>              20 BINARY_MODULO
>              21 BUILD_TUPLE              3
>              24 RETURN_VALUE
>
> So to make this consistent with all other __future__ directives, there
> would need to be some kind of safe way to define this: perhaps an
> attribute on the generator object. Something like this:
>
> >>> def gen(x):
> ...     yield x
> ...     raise StopIteration(42)
> ...
> >>> g=gen(123)
> >>> list(g)
> [123]
> >>> gen.__distinguish_returns__ = True
> >>> g=gen(123)
> >>> list(g)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "<stdin>", line 3, in gen
> StopIteration: 42
>
> The attribute on the function would be what affects behaviour; the
> __future__ directive applies that attribute to all generator functions
> in its module (including genexprs). Once the __future__ directive
> becomes automatic, the attribute can and will be dropped - any code
> which interrogates it MUST be prepared to stop interrogating it once
> the feature applies to all modules.
>
> Does that sound reasonable? Should it be added to the PEP?
>

I agree with you and Steven that this is a fine use of __future__. What a
generator does with a StopIteration that is about to bubble out of its
frame is up to that generator. I don't think it needs to be a flag on the
*function* though -- IMO it should be a flag on the code object. (And the
flag should somehow be transferred to the stack frame when the function is
executed, so the right action can be taken when an exception is about to
bubble out of that frame.)

One other small point: let's change the PEP to just propose RuntimeError,
and move the "some other exception" to the "rejected ideas" section.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141116/51c6d37d/attachment-0001.html>

From rosuav at gmail.com  Mon Nov 17 03:11:15 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 17 Nov 2014 13:11:15 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAP7+vJJ0m_PipGY0S172Nt3cZvJwCQFmOhxNCBioggFAYzU4kw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com>
 <CAPTjJmqacEJqexWnUU1UBoC_VcsPKSoVw1_9uqYOyAXAbsLxYw@mail.gmail.com>
 <54691F8D.1080703@canterbury.ac.nz>
 <20141117000302.GS2748@ando.pearwood.info>
 <CAPTjJmoHgSSL0L3Bh5_qXDB=mtd-6_X70WhD-_WqoXj3qzZuSw@mail.gmail.com>
 <CAP7+vJJ0m_PipGY0S172Nt3cZvJwCQFmOhxNCBioggFAYzU4kw@mail.gmail.com>
Message-ID: <CAPTjJmogxXu1QkkXhSrkRac2HD5Ki2NKLTaLARdB+=xeOnL-dw@mail.gmail.com>

On Mon, Nov 17, 2014 at 12:58 PM, Guido van Rossum <guido at python.org> wrote:
> I agree with you and Steven that this is a fine use of __future__. What a
> generator does with a StopIteration that is about to bubble out of its frame
> is up to that generator. I don't think it needs to be a flag on the
> *function* though -- IMO it should be a flag on the code object. (And the
> flag should somehow be transferred to the stack frame when the function is
> executed, so the right action can be taken when an exception is about to
> bubble out of that frame.)
>
> One other small point: let's change the PEP to just propose RuntimeError,
> and move the "some other exception" to the "rejected ideas" section.

Changes incorporated, thanks! I'm not familiar with the details of
stack frame handling, so I've taken the cop-out approach and just
quoted you directly into the PEP.

PEP draft:
https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-0479.txt

GitHub hosted repo, if you want to follow changes etc:
https://github.com/Rosuav/GenStopIter

ChrisA

From greg.ewing at canterbury.ac.nz  Mon Nov 17 04:22:42 2014
From: greg.ewing at canterbury.ac.nz (Greg)
Date: Mon, 17 Nov 2014 16:22:42 +1300
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
 generators
In-Reply-To: <CAPTjJmoHgSSL0L3Bh5_qXDB=mtd-6_X70WhD-_WqoXj3qzZuSw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com>
 <CAPTjJmqacEJqexWnUU1UBoC_VcsPKSoVw1_9uqYOyAXAbsLxYw@mail.gmail.com>
 <54691F8D.1080703@canterbury.ac.nz> <20141117000302.GS2748@ando.pearwood.info>
 <CAPTjJmoHgSSL0L3Bh5_qXDB=mtd-6_X70WhD-_WqoXj3qzZuSw@mail.gmail.com>
Message-ID: <54696A02.6050602@canterbury.ac.nz>

On 17/11/2014 2:29 p.m., Chris Angelico wrote:

> barry_as_FLUFL: Not entirely sure what this one actually accomplishes. :)

It determines whether "not equal" is spelled "!=" or "<>", so it
fits the pattern of being compile-time-only.

-- 
Greg


From rosuav at gmail.com  Mon Nov 17 04:34:52 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 17 Nov 2014 14:34:52 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <54696A02.6050602@canterbury.ac.nz>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com>
 <CAPTjJmqacEJqexWnUU1UBoC_VcsPKSoVw1_9uqYOyAXAbsLxYw@mail.gmail.com>
 <54691F8D.1080703@canterbury.ac.nz>
 <20141117000302.GS2748@ando.pearwood.info>
 <CAPTjJmoHgSSL0L3Bh5_qXDB=mtd-6_X70WhD-_WqoXj3qzZuSw@mail.gmail.com>
 <54696A02.6050602@canterbury.ac.nz>
Message-ID: <CAPTjJmrvt5wLkE5DLgYNwgHyLuTkh4nyszM3ur-jV8ZkO+90LA@mail.gmail.com>

On Mon, Nov 17, 2014 at 2:22 PM, Greg <greg.ewing at canterbury.ac.nz> wrote:
> On 17/11/2014 2:29 p.m., Chris Angelico wrote:
>
>> barry_as_FLUFL: Not entirely sure what this one actually accomplishes. :)
>
>
> It determines whether "not equal" is spelled "!=" or "<>", so it
> fits the pattern of being compile-time-only.

Right. So, are there any __future__ directives that have any effect on
byte-code? I'm not seeing any, though that may just mean I didn't
recognize one.

ChrisA

From g.brandl at gmx.net  Mon Nov 17 10:26:13 2014
From: g.brandl at gmx.net (Georg Brandl)
Date: Mon, 17 Nov 2014 10:26:13 +0100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmogxXu1QkkXhSrkRac2HD5Ki2NKLTaLARdB+=xeOnL-dw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com>
 <CAPTjJmqacEJqexWnUU1UBoC_VcsPKSoVw1_9uqYOyAXAbsLxYw@mail.gmail.com>
 <54691F8D.1080703@canterbury.ac.nz>
 <20141117000302.GS2748@ando.pearwood.info>
 <CAPTjJmoHgSSL0L3Bh5_qXDB=mtd-6_X70WhD-_WqoXj3qzZuSw@mail.gmail.com>
 <CAP7+vJJ0m_PipGY0S172Nt3cZvJwCQFmOhxNCBioggFAYzU4kw@mail.gmail.com>
 <CAPTjJmogxXu1QkkXhSrkRac2HD5Ki2NKLTaLARdB+=xeOnL-dw@mail.gmail.com>
Message-ID: <m4cevm$735$1@ger.gmane.org>

On 11/17/2014 03:11 AM, Chris Angelico wrote:
> On Mon, Nov 17, 2014 at 12:58 PM, Guido van Rossum <guido at python.org> wrote:
>> I agree with you and Steven that this is a fine use of __future__. What a
>> generator does with a StopIteration that is about to bubble out of its frame
>> is up to that generator. I don't think it needs to be a flag on the
>> *function* though -- IMO it should be a flag on the code object. (And the
>> flag should somehow be transferred to the stack frame when the function is
>> executed, so the right action can be taken when an exception is about to
>> bubble out of that frame.)
>>
>> One other small point: let's change the PEP to just propose RuntimeError,
>> and move the "some other exception" to the "rejected ideas" section.
> 
> Changes incorporated, thanks! I'm not familiar with the details of
> stack frame handling, so I've taken the cop-out approach and just
> quoted you directly into the PEP.
> 
> PEP draft:
> https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-0479.txt

I fixed a typo and syntax and made a content clarification ("generator
constructed" -> "generator function constructed"; the generator object
itself is constructed when the function is called) and committed the changes.

Georg


From rosuav at gmail.com  Mon Nov 17 11:27:22 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 17 Nov 2014 21:27:22 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <m4cevm$735$1@ger.gmane.org>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com>
 <CAPTjJmqacEJqexWnUU1UBoC_VcsPKSoVw1_9uqYOyAXAbsLxYw@mail.gmail.com>
 <54691F8D.1080703@canterbury.ac.nz>
 <20141117000302.GS2748@ando.pearwood.info>
 <CAPTjJmoHgSSL0L3Bh5_qXDB=mtd-6_X70WhD-_WqoXj3qzZuSw@mail.gmail.com>
 <CAP7+vJJ0m_PipGY0S172Nt3cZvJwCQFmOhxNCBioggFAYzU4kw@mail.gmail.com>
 <CAPTjJmogxXu1QkkXhSrkRac2HD5Ki2NKLTaLARdB+=xeOnL-dw@mail.gmail.com>
 <m4cevm$735$1@ger.gmane.org>
Message-ID: <CAPTjJmp2XZy56dWRUQPgMYZhG-8pwqSPFjagxYmJ_3rVmb9_GA@mail.gmail.com>

On Mon, Nov 17, 2014 at 8:26 PM, Georg Brandl <g.brandl at gmx.net> wrote:
> I fixed a typo and syntax and made a content clarification ("generator
> constructed" -> "generator function constructed"; the generator object
> itself is constructed when the function is called) and committed the changes.

Thanks Georg! This means today's version is now visible here:

http://legacy.python.org/dev/peps/pep-0479/

ChrisA

From ncoghlan at gmail.com  Mon Nov 17 13:56:32 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 17 Nov 2014 22:56:32 +1000
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmrvt5wLkE5DLgYNwgHyLuTkh4nyszM3ur-jV8ZkO+90LA@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com>
 <CAPTjJmqacEJqexWnUU1UBoC_VcsPKSoVw1_9uqYOyAXAbsLxYw@mail.gmail.com>
 <54691F8D.1080703@canterbury.ac.nz>
 <20141117000302.GS2748@ando.pearwood.info>
 <CAPTjJmoHgSSL0L3Bh5_qXDB=mtd-6_X70WhD-_WqoXj3qzZuSw@mail.gmail.com>
 <54696A02.6050602@canterbury.ac.nz>
 <CAPTjJmrvt5wLkE5DLgYNwgHyLuTkh4nyszM3ur-jV8ZkO+90LA@mail.gmail.com>
Message-ID: <CADiSq7dtEphyJFWOFj6edhte0BA6aAjoXo72r7RLNMu+XbdNSA@mail.gmail.com>

On 17 November 2014 13:34, Chris Angelico <rosuav at gmail.com> wrote:

> On Mon, Nov 17, 2014 at 2:22 PM, Greg <greg.ewing at canterbury.ac.nz> wrote:
> > On 17/11/2014 2:29 p.m., Chris Angelico wrote:
> >
> >> barry_as_FLUFL: Not entirely sure what this one actually accomplishes.
> :)
> >
> >
> > It determines whether "not equal" is spelled "!=" or "<>", so it
> > fits the pattern of being compile-time-only.
>
> Right. So, are there any __future__ directives that have any effect on
> byte-code? I'm not seeing any, though that may just mean I didn't
> recognize one.
>

True division (in Python 2) is a nice simple one to look at, since it just
swaps one bytecode for another (BINARY_DIVIDE -> BINARY_TRUE_DIVIDE)

>>> def floor_div(x, y):
...     return x / y
...
>>> from __future__ import division
>>> def true_div(x, y):
...     return x / y
...
>>> import dis
>>> dis.dis(floor_div)
  2           0 LOAD_FAST                0 (x)
              3 LOAD_FAST                1 (y)
              6 BINARY_DIVIDE
              7 RETURN_VALUE
>>> dis.dis(true_div)
  2           0 LOAD_FAST                0 (x)
              3 LOAD_FAST                1 (y)
              6 BINARY_TRUE_DIVIDE
              7 RETURN_VALUE

The compiler actually stores a whole pile of useful info on code objects
that doesn't show up in the disassembly output (switching to Python 3 for
more up to date dis module goodness):

>>> import dis
>>> def true_div(x, y):
...     return x / y
...
>>> dis.show_code(true_div)
Name:              true_div
Filename:          <stdin>
Argument count:    2
Kw-only arguments: 0
Number of locals:  2
Stack size:        2
Flags:             OPTIMIZED, NEWLOCALS, NOFREE
Constants:
   0: None
Variable names:
   0: x
   1: y

So conveying to the generator iterator whether or not "from __future__
import generator_return" was in effect would just be a matter of the
compiler setting a new flag on the generator code object. For *affected
generators* (i.e. those defined in a module where the new future statement
was in effect), StopIteration escaping would be considered a RuntimeError.

For almost all code, such RuntimeErrors would look like any other
RuntimError raised by a broken generator implementation.

The only code which would *have* to change immediately as a "Porting to
Python 3.5" requirement is code like that in contextlib, which throws
StopIteration into generators, and currently expects to get it back out
unmodified. Such code will need to be updated to also handle RuntimError
instances where the direct cause is the StopIteration exception that was
thrown in.

Other affected code (such as the "next() bubbling up" groupby example)
would keep working unless the __future__ statement was in effect.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141117/697a10e9/attachment.html>

From jeanpierreda at gmail.com  Mon Nov 17 14:50:36 2014
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Mon, 17 Nov 2014 05:50:36 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
Message-ID: <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>

On Sat, Nov 15, 2014 at 1:29 AM, Chris Angelico <rosuav at gmail.com> wrote:
> The interaction of generators and ``StopIteration`` is currently
> somewhat surprising, and can conceal obscure bugs.  An unexpected
> exception should not result in subtly altered behaviour, but should
> cause a noisy and easily-debugged traceback.  Currently,
> ``StopIteration`` can be absorbed by the generator construct.

Specifically it's absorbed by the caller of the generator, because the
caller doesn't know the difference between next(x) raising
StopIteration because the iterator specifically wants to stop, vs
because of accident.

As another alternative, how about a new iterator protocol that is
defined without this ambiguity? Code at the bottom of my post to help
explain:  define a new method __nextx__ which doesn't use
StopIteration for any signalling, instead, it returns None if there
are no values to return, and returns a special value Some(v) if it
wants to return a value v.  Both next(it) and nextx(it) are made to
work for any iterator that is defined using either protocol, but for
loops and Python builtins all use nextx internally. Generators define
__next__ unless you from __future__ import iterators, in which case
they define __nextx__ instead.

In this way, old code can't tell the difference between accidental
StopIteration and deliberate StopIteration, but new code (using nextx
instead of next, and using __future__ import'd generators) can. No
backwards incompatibility is introduced, and you can still insert
StopIteration into a generator and get it back out -- using both
next() where it is ambiguous and nextx() where it is not.

Yes, it's ugly to have two different iterator protocols, but not that
ugly. In fact, this would be Python's third (I have omitted that third
protocol in the below example, for the sake of clarity). I find the
proposed solution more scary, in that it's sort of a "hack" to get
around an old mistake, rather than a correction to that mistake, and
it introduces complexity that can't be removed in principle. (Also,
it's very unusual.)

class Some:
    def __init__(self, value):
        self.value = value

def next(it):
    v = nextx(it)
    if v is None:
        raise StopIteration
    return v.value

def nextx(it):
    if hasattr(it, '__nextx__'):
        v = it.__nextx__()
        if v is None or isinstance(v, Some):
            return v
        raise TypeError("__nextx__ must return Some(...) or None, not
%r" % (v,))
    if hasattr(it, '__next__'):
        try:
            return Some(it.__next__())
        except StopIteration:
            return None
    raise TypeError

-- Devin

From rosuav at gmail.com  Mon Nov 17 15:04:01 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 18 Nov 2014 01:04:01 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
Message-ID: <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>

On Tue, Nov 18, 2014 at 12:50 AM, Devin Jeanpierre
<jeanpierreda at gmail.com> wrote:
> As another alternative, how about a new iterator protocol that is
> defined without this ambiguity? Code at the bottom of my post to help
> explain:  define a new method __nextx__ which doesn't use
> StopIteration for any signalling, instead, it returns None if there
> are no values to return, and returns a special value Some(v) if it
> wants to return a value v.  Both next(it) and nextx(it) are made to
> work for any iterator that is defined using either protocol, but for
> loops and Python builtins all use nextx internally. Generators define
> __next__ unless you from __future__ import iterators, in which case
> they define __nextx__ instead.

I had actually contemplated adding a "what if __next__ returned a
sentinel instead of raising an exception" possibility to the PEP, if
only for completeness. Since someone else has suggested it too now, it
may be worth doing.

Rather than a wrapper around every returned value, what I'd be
inclined toward is a special sentinel that otherwise cannot be
returned. This could be a dedicated, arbitrary object, or something
globally unique, or something locally unique. One option that comes to
mind is to have the generator return _itself_ to signal that it's
returned.

I don't think this option will be better than the current front
runners, but would you like me to add it for completeness? The biggest
downside is that it might give a false positive; you can't, for
instance, have an iterator "all_objects()" which returns, like the
name says, every object currently known to Python. (I don't know that
CPython is capable of implementing that, but there's no reason another
Python couldn't, and it might be useful.) I expect that's why the
exception system was used instead; can anyone confirm that?

ChrisA

From ram at rachum.com  Mon Nov 17 16:50:27 2014
From: ram at rachum.com (Ram Rachum)
Date: Mon, 17 Nov 2014 17:50:27 +0200
Subject: [Python-ideas] Implement `itertools.permutations.__getitem__`
	and `itertools.permutations.index`
In-Reply-To: <CANXboVb5K4cfTb1JsRbQdOzHgroFRAi2H006dfFc6Y-Qv=vnQA@mail.gmail.com>
References: <bf592625-2396-471f-af87-47ad0375819e@googlegroups.com>
 <20140505171538.GR4273@ando>
 <CAEfz+TwtUovHQrsqMUcdihY-MxFxBVyDFXOg32jD30XQ0Gke9Q@mail.gmail.com>
 <20140506023902.GV4273@ando>
 <CALWZvp4_RphieoFU+MCCKeT10=6iu7xDXFq2W8zMg+xj0QDeZQ@mail.gmail.com>
 <CACac1F_kneXtP8wbDZqGp5T4_shwEqbLNSdUAuDEr3Hopdy43w@mail.gmail.com>
 <CALWZvp6OcLpZY-Hczk6Hw59sLpaYS0SeTjf1Vm4sS+i8qyvjSA@mail.gmail.com>
 <CANXboVZwE47SPD8ywcQ19reNb0hyDknaLEX1aoX-NkBzrSK24w@mail.gmail.com>
 <CALWZvp7Eh29rR8OcSUy4ZsK9w5CSbSj6MxWYru1bWt5jUX_66A@mail.gmail.com>
 <CANXboVZxF5J21O78udpGr1SndLxUpY2xYdq9oEP=f6hPq3kBGg@mail.gmail.com>
 <18aa148c-f6cb-414c-9f6c-de5011568204@googlegroups.com>
 <CANXboVb5K4cfTb1JsRbQdOzHgroFRAi2H006dfFc6Y-Qv=vnQA@mail.gmail.com>
Message-ID: <CANXboVZqDY5pNC0yioa7ZXzPTdSR3yWp08MRVV8e=D2Uuha5OA@mail.gmail.com>

Hi guys,

I just wanted to give an update on this: I just released my own code that
does this to PyPI: https://pypi.python.org/pypi/combi


Thanks,
Ram.

On Tue, Jun 10, 2014 at 9:24 AM, Ram Rachum <ram at rachum.com> wrote:

> I'll email you if/when it's released :)
>
>
> On Tue, Jun 10, 2014 at 9:04 AM, Neil Girdhar <mistersheik at gmail.com>
> wrote:
>
>> I really like this and hope that it eventually makes it into the stdlib.
>> It's also a good argument for your other suggestion whereby some of the
>> itertools to return Iterables rather than Iterators like range does.
>>
>> Best,
>>
>> Neil
>>
>> On Wednesday, May 7, 2014 1:43:20 PM UTC-4, Ram Rachum wrote:
>>
>>> I'm probably going to implement it in my python_toolbox package. I
>>> already implemented 30% and it's really cool. It's at the point where I
>>> doubt that I want it in the stdlib because I've gotten so much awesome
>>> functionality into it and I'd hate to (a) have 80% of it stripped and (b)
>>> have the class names changed to be non-Pythonic :)
>>>
>>>
>>> On Wed, May 7, 2014 at 8:40 PM, Tal Einat <tale... at gmail.com> wrote:
>>>
>>> On Wed, May 7, 2014 at 8:21 PM, Ram Rachum <ram.r... at gmail.com> wrote:
>>>> > Hi Tal,
>>>> >
>>>> > I'm using it for a project of my own (optimizing keyboard layout) but
>>>> I
>>>> > can't make the case that it's useful for the stdlib. I'd understand
>>>> if it
>>>> > would be omitted for not being enough of a common need.
>>>>
>>>> At the least, this (a function for getting a specific permutation by
>>>> lexicographical-order index) could make a nice cookbook recipe.
>>>>
>>>> - Tal
>>>>
>>>
>>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>> --
>>
>> ---
>> You received this message because you are subscribed to a topic in the
>> Google Groups "python-ideas" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/python-ideas/in2gUQMFUzA/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> python-ideas+unsubscribe at googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141117/c0dba90b/attachment-0001.html>

From guido at python.org  Mon Nov 17 18:28:41 2014
From: guido at python.org (Guido van Rossum)
Date: Mon, 17 Nov 2014 09:28:41 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmp2XZy56dWRUQPgMYZhG-8pwqSPFjagxYmJ_3rVmb9_GA@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com>
 <CAPTjJmqacEJqexWnUU1UBoC_VcsPKSoVw1_9uqYOyAXAbsLxYw@mail.gmail.com>
 <54691F8D.1080703@canterbury.ac.nz> <20141117000302.GS2748@ando.pearwood.info>
 <CAPTjJmoHgSSL0L3Bh5_qXDB=mtd-6_X70WhD-_WqoXj3qzZuSw@mail.gmail.com>
 <CAP7+vJJ0m_PipGY0S172Nt3cZvJwCQFmOhxNCBioggFAYzU4kw@mail.gmail.com>
 <CAPTjJmogxXu1QkkXhSrkRac2HD5Ki2NKLTaLARdB+=xeOnL-dw@mail.gmail.com>
 <m4cevm$735$1@ger.gmane.org>
 <CAPTjJmp2XZy56dWRUQPgMYZhG-8pwqSPFjagxYmJ_3rVmb9_GA@mail.gmail.com>
Message-ID: <CAP7+vJJY=Phea--AaOehp-SQky0_BYYKYsAjW-+Mwc-TBoU41Q@mail.gmail.com>

On Mon, Nov 17, 2014 at 2:27 AM, Chris Angelico <rosuav at gmail.com> wrote:

> On Mon, Nov 17, 2014 at 8:26 PM, Georg Brandl <g.brandl at gmx.net> wrote:
> > I fixed a typo and syntax and made a content clarification ("generator
> > constructed" -> "generator function constructed"; the generator object
> > itself is constructed when the function is called) and committed the
> changes.
>
> Thanks Georg! This means today's version is now visible here:
>
> http://legacy.python.org/dev/peps/pep-0479/
>

Off-topic: the new python.org site now supports PEPs, so please switch to
URLs like this: https://www.python.org/dev/peps/pep-0479/ (if you don't
like the formatting send a pull request to
https://github.com/python/pythondotorg).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141117/1d295dd0/attachment.html>

From guido at python.org  Mon Nov 17 18:40:20 2014
From: guido at python.org (Guido van Rossum)
Date: Mon, 17 Nov 2014 09:40:20 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
Message-ID: <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>

I don't want to contemplate a new __next__ protocol. The existing protocol
was carefully designed and tuned to have minimal memory overhead (even to
the point where the exception instance returned may be reused). Wapping
each result would just result in an extra allocation + deallocation per
iteration, unless you can play games with reference counts or do something
else to complicate the semantics). Introducing __nextx__ would require
thousands of libraries implementing this to incur churn as they feel the
pressure to switch to the new protocol, and the compatibility issue would
be felt everywhere.

The problem we're trying to fix is unique to generators (thereby also
implicating generator expressions).

On Mon, Nov 17, 2014 at 6:04 AM, Chris Angelico <rosuav at gmail.com> wrote:

> On Tue, Nov 18, 2014 at 12:50 AM, Devin Jeanpierre
> <jeanpierreda at gmail.com> wrote:
> > As another alternative, how about a new iterator protocol that is
> > defined without this ambiguity? Code at the bottom of my post to help
> > explain:  define a new method __nextx__ which doesn't use
> > StopIteration for any signalling, instead, it returns None if there
> > are no values to return, and returns a special value Some(v) if it
> > wants to return a value v.  Both next(it) and nextx(it) are made to
> > work for any iterator that is defined using either protocol, but for
> > loops and Python builtins all use nextx internally. Generators define
> > __next__ unless you from __future__ import iterators, in which case
> > they define __nextx__ instead.
>
> I had actually contemplated adding a "what if __next__ returned a
> sentinel instead of raising an exception" possibility to the PEP, if
> only for completeness. Since someone else has suggested it too now, it
> may be worth doing.
>
> Rather than a wrapper around every returned value, what I'd be
> inclined toward is a special sentinel that otherwise cannot be
> returned. This could be a dedicated, arbitrary object, or something
> globally unique, or something locally unique. One option that comes to
> mind is to have the generator return _itself_ to signal that it's
> returned.
>
> I don't think this option will be better than the current front
> runners, but would you like me to add it for completeness? The biggest
> downside is that it might give a false positive; you can't, for
> instance, have an iterator "all_objects()" which returns, like the
> name says, every object currently known to Python. (I don't know that
> CPython is capable of implementing that, but there's no reason another
> Python couldn't, and it might be useful.) I expect that's why the
> exception system was used instead; can anyone confirm that?
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141117/3222ca89/attachment.html>

From mistersheik at gmail.com  Mon Nov 17 18:33:56 2014
From: mistersheik at gmail.com (Neil Girdhar)
Date: Mon, 17 Nov 2014 12:33:56 -0500
Subject: [Python-ideas] Implement `itertools.permutations.__getitem__`
	and `itertools.permutations.index`
In-Reply-To: <CANXboVZqDY5pNC0yioa7ZXzPTdSR3yWp08MRVV8e=D2Uuha5OA@mail.gmail.com>
References: <bf592625-2396-471f-af87-47ad0375819e@googlegroups.com>
 <20140505171538.GR4273@ando>
 <CAEfz+TwtUovHQrsqMUcdihY-MxFxBVyDFXOg32jD30XQ0Gke9Q@mail.gmail.com>
 <20140506023902.GV4273@ando>
 <CALWZvp4_RphieoFU+MCCKeT10=6iu7xDXFq2W8zMg+xj0QDeZQ@mail.gmail.com>
 <CACac1F_kneXtP8wbDZqGp5T4_shwEqbLNSdUAuDEr3Hopdy43w@mail.gmail.com>
 <CALWZvp6OcLpZY-Hczk6Hw59sLpaYS0SeTjf1Vm4sS+i8qyvjSA@mail.gmail.com>
 <CANXboVZwE47SPD8ywcQ19reNb0hyDknaLEX1aoX-NkBzrSK24w@mail.gmail.com>
 <CALWZvp7Eh29rR8OcSUy4ZsK9w5CSbSj6MxWYru1bWt5jUX_66A@mail.gmail.com>
 <CANXboVZxF5J21O78udpGr1SndLxUpY2xYdq9oEP=f6hPq3kBGg@mail.gmail.com>
 <18aa148c-f6cb-414c-9f6c-de5011568204@googlegroups.com>
 <CANXboVb5K4cfTb1JsRbQdOzHgroFRAi2H006dfFc6Y-Qv=vnQA@mail.gmail.com>
 <CANXboVZqDY5pNC0yioa7ZXzPTdSR3yWp08MRVV8e=D2Uuha5OA@mail.gmail.com>
Message-ID: <CAA68w_mT0TvOh7gTye3n9wNR-xy8QV5EPP0PUj6vwxvQFDFTRg@mail.gmail.com>

Looks great!

Why did you go with "get_rapplied", "unrapplied", etc. instead of having a
copy() method and using settable properties?

On Mon, Nov 17, 2014 at 10:50 AM, Ram Rachum <ram at rachum.com> wrote:

> Hi guys,
>
> I just wanted to give an update on this: I just released my own code that
> does this to PyPI: https://pypi.python.org/pypi/combi
>
>
> Thanks,
> Ram.
>
> On Tue, Jun 10, 2014 at 9:24 AM, Ram Rachum <ram at rachum.com> wrote:
>
>> I'll email you if/when it's released :)
>>
>>
>> On Tue, Jun 10, 2014 at 9:04 AM, Neil Girdhar <mistersheik at gmail.com>
>> wrote:
>>
>>> I really like this and hope that it eventually makes it into the
>>> stdlib.  It's also a good argument for your other suggestion whereby some
>>> of the itertools to return Iterables rather than Iterators like range does.
>>>
>>> Best,
>>>
>>> Neil
>>>
>>> On Wednesday, May 7, 2014 1:43:20 PM UTC-4, Ram Rachum wrote:
>>>
>>>> I'm probably going to implement it in my python_toolbox package. I
>>>> already implemented 30% and it's really cool. It's at the point where I
>>>> doubt that I want it in the stdlib because I've gotten so much awesome
>>>> functionality into it and I'd hate to (a) have 80% of it stripped and (b)
>>>> have the class names changed to be non-Pythonic :)
>>>>
>>>>
>>>> On Wed, May 7, 2014 at 8:40 PM, Tal Einat <tale... at gmail.com> wrote:
>>>>
>>>> On Wed, May 7, 2014 at 8:21 PM, Ram Rachum <ram.r... at gmail.com> wrote:
>>>>> > Hi Tal,
>>>>> >
>>>>> > I'm using it for a project of my own (optimizing keyboard layout)
>>>>> but I
>>>>> > can't make the case that it's useful for the stdlib. I'd understand
>>>>> if it
>>>>> > would be omitted for not being enough of a common need.
>>>>>
>>>>> At the least, this (a function for getting a specific permutation by
>>>>> lexicographical-order index) could make a nice cookbook recipe.
>>>>>
>>>>> - Tal
>>>>>
>>>>
>>>>
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "python-ideas" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/python-ideas/in2gUQMFUzA/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> python-ideas+unsubscribe at googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141117/f0aaa2c2/attachment-0001.html>

From ram at rachum.com  Mon Nov 17 18:44:46 2014
From: ram at rachum.com (Ram Rachum)
Date: Mon, 17 Nov 2014 19:44:46 +0200
Subject: [Python-ideas] Implement `itertools.permutations.__getitem__`
	and `itertools.permutations.index`
In-Reply-To: <CAA68w_mT0TvOh7gTye3n9wNR-xy8QV5EPP0PUj6vwxvQFDFTRg@mail.gmail.com>
References: <bf592625-2396-471f-af87-47ad0375819e@googlegroups.com>
 <20140505171538.GR4273@ando>
 <CAEfz+TwtUovHQrsqMUcdihY-MxFxBVyDFXOg32jD30XQ0Gke9Q@mail.gmail.com>
 <20140506023902.GV4273@ando>
 <CALWZvp4_RphieoFU+MCCKeT10=6iu7xDXFq2W8zMg+xj0QDeZQ@mail.gmail.com>
 <CACac1F_kneXtP8wbDZqGp5T4_shwEqbLNSdUAuDEr3Hopdy43w@mail.gmail.com>
 <CALWZvp6OcLpZY-Hczk6Hw59sLpaYS0SeTjf1Vm4sS+i8qyvjSA@mail.gmail.com>
 <CANXboVZwE47SPD8ywcQ19reNb0hyDknaLEX1aoX-NkBzrSK24w@mail.gmail.com>
 <CALWZvp7Eh29rR8OcSUy4ZsK9w5CSbSj6MxWYru1bWt5jUX_66A@mail.gmail.com>
 <CANXboVZxF5J21O78udpGr1SndLxUpY2xYdq9oEP=f6hPq3kBGg@mail.gmail.com>
 <18aa148c-f6cb-414c-9f6c-de5011568204@googlegroups.com>
 <CANXboVb5K4cfTb1JsRbQdOzHgroFRAi2H006dfFc6Y-Qv=vnQA@mail.gmail.com>
 <CANXboVZqDY5pNC0yioa7ZXzPTdSR3yWp08MRVV8e=D2Uuha5OA@mail.gmail.com>
 <CAA68w_mT0TvOh7gTye3n9wNR-xy8QV5EPP0PUj6vwxvQFDFTRg@mail.gmail.com>
Message-ID: <CANXboVYQV+U06NWCLezDn8pmtsLv3=KCx86pF=wCx2WUrG9oKA@mail.gmail.com>

Thanks :)

Settable properties are a no-go because I wanted permutation spaces to be
immutable. Since the cost of creating a new space is nil (permutations are
created on-demand, not on space creation) there isn't a reason to mutate an
existing space.

I don't think using a copy() method to would be very nice. But I guess it's
a matter of taste.

On Mon, Nov 17, 2014 at 7:33 PM, Neil Girdhar <mistersheik at gmail.com> wrote:

> Looks great!
>
> Why did you go with "get_rapplied", "unrapplied", etc. instead of having a
> copy() method and using settable properties?
>
> On Mon, Nov 17, 2014 at 10:50 AM, Ram Rachum <ram at rachum.com> wrote:
>
>> Hi guys,
>>
>> I just wanted to give an update on this: I just released my own code that
>> does this to PyPI: https://pypi.python.org/pypi/combi
>>
>>
>> Thanks,
>> Ram.
>>
>> On Tue, Jun 10, 2014 at 9:24 AM, Ram Rachum <ram at rachum.com> wrote:
>>
>>> I'll email you if/when it's released :)
>>>
>>>
>>> On Tue, Jun 10, 2014 at 9:04 AM, Neil Girdhar <mistersheik at gmail.com>
>>> wrote:
>>>
>>>> I really like this and hope that it eventually makes it into the
>>>> stdlib.  It's also a good argument for your other suggestion whereby some
>>>> of the itertools to return Iterables rather than Iterators like range does.
>>>>
>>>> Best,
>>>>
>>>> Neil
>>>>
>>>> On Wednesday, May 7, 2014 1:43:20 PM UTC-4, Ram Rachum wrote:
>>>>
>>>>> I'm probably going to implement it in my python_toolbox package. I
>>>>> already implemented 30% and it's really cool. It's at the point where I
>>>>> doubt that I want it in the stdlib because I've gotten so much awesome
>>>>> functionality into it and I'd hate to (a) have 80% of it stripped and (b)
>>>>> have the class names changed to be non-Pythonic :)
>>>>>
>>>>>
>>>>> On Wed, May 7, 2014 at 8:40 PM, Tal Einat <tale... at gmail.com> wrote:
>>>>>
>>>>> On Wed, May 7, 2014 at 8:21 PM, Ram Rachum <ram.r... at gmail.com> wrote:
>>>>>> > Hi Tal,
>>>>>> >
>>>>>> > I'm using it for a project of my own (optimizing keyboard layout)
>>>>>> but I
>>>>>> > can't make the case that it's useful for the stdlib. I'd understand
>>>>>> if it
>>>>>> > would be omitted for not being enough of a common need.
>>>>>>
>>>>>> At the least, this (a function for getting a specific permutation by
>>>>>> lexicographical-order index) could make a nice cookbook recipe.
>>>>>>
>>>>>> - Tal
>>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> Python-ideas mailing list
>>>> Python-ideas at python.org
>>>> https://mail.python.org/mailman/listinfo/python-ideas
>>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>> --
>>>>
>>>> ---
>>>> You received this message because you are subscribed to a topic in the
>>>> Google Groups "python-ideas" group.
>>>> To unsubscribe from this topic, visit
>>>> https://groups.google.com/d/topic/python-ideas/in2gUQMFUzA/unsubscribe.
>>>> To unsubscribe from this group and all its topics, send an email to
>>>> python-ideas+unsubscribe at googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141117/2037e21e/attachment.html>

From jeanpierreda at gmail.com  Mon Nov 17 18:53:17 2014
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Mon, 17 Nov 2014 09:53:17 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
Message-ID: <CABicbJKeSWpMqA_NidZcdinMgqTKEa1-NTw_u02t+UzTv9d=kw@mail.gmail.com>

On Mon, Nov 17, 2014 at 9:40 AM, Guido van Rossum <guido at python.org> wrote:
> I don't want to contemplate a new __next__ protocol. The existing protocol
> was carefully designed and tuned to have minimal memory overhead (even to
> the point where the exception instance returned may be reused). Wapping each
> result would just result in an extra allocation + deallocation per
> iteration, unless you can play games with reference counts or do something
> else to complicate the semantics). Introducing __nextx__ would require
> thousands of libraries implementing this to incur churn as they feel the
> pressure to switch to the new protocol, and the compatibility issue would be
> felt everywhere.

This sounds totally reasonable to me.

> The problem we're trying to fix is unique to generators (thereby also
> implicating generator expressions).

I suppose since you're only fixing generators, then that is literally
the only problem you are trying to fix, but it is more general than
that. I have encountered this sort of problem writing __next__ by hand
in Python -- that is, that bugs inside code I call result in silent
control flow changes rather than a visible exception.

-- Devin

From mistersheik at gmail.com  Mon Nov 17 19:01:32 2014
From: mistersheik at gmail.com (Neil Girdhar)
Date: Mon, 17 Nov 2014 13:01:32 -0500
Subject: [Python-ideas] Implement `itertools.permutations.__getitem__`
	and `itertools.permutations.index`
In-Reply-To: <CANXboVYQV+U06NWCLezDn8pmtsLv3=KCx86pF=wCx2WUrG9oKA@mail.gmail.com>
References: <bf592625-2396-471f-af87-47ad0375819e@googlegroups.com>
 <20140505171538.GR4273@ando>
 <CAEfz+TwtUovHQrsqMUcdihY-MxFxBVyDFXOg32jD30XQ0Gke9Q@mail.gmail.com>
 <20140506023902.GV4273@ando>
 <CALWZvp4_RphieoFU+MCCKeT10=6iu7xDXFq2W8zMg+xj0QDeZQ@mail.gmail.com>
 <CACac1F_kneXtP8wbDZqGp5T4_shwEqbLNSdUAuDEr3Hopdy43w@mail.gmail.com>
 <CALWZvp6OcLpZY-Hczk6Hw59sLpaYS0SeTjf1Vm4sS+i8qyvjSA@mail.gmail.com>
 <CANXboVZwE47SPD8ywcQ19reNb0hyDknaLEX1aoX-NkBzrSK24w@mail.gmail.com>
 <CALWZvp7Eh29rR8OcSUy4ZsK9w5CSbSj6MxWYru1bWt5jUX_66A@mail.gmail.com>
 <CANXboVZxF5J21O78udpGr1SndLxUpY2xYdq9oEP=f6hPq3kBGg@mail.gmail.com>
 <18aa148c-f6cb-414c-9f6c-de5011568204@googlegroups.com>
 <CANXboVb5K4cfTb1JsRbQdOzHgroFRAi2H006dfFc6Y-Qv=vnQA@mail.gmail.com>
 <CANXboVZqDY5pNC0yioa7ZXzPTdSR3yWp08MRVV8e=D2Uuha5OA@mail.gmail.com>
 <CAA68w_mT0TvOh7gTye3n9wNR-xy8QV5EPP0PUj6vwxvQFDFTRg@mail.gmail.com>
 <CANXboVYQV+U06NWCLezDn8pmtsLv3=KCx86pF=wCx2WUrG9oKA@mail.gmail.com>
Message-ID: <CAA68w_nx3RTeeMEoZSWjchdSgemc2dNVnq0uTQcDP_+ELTFFMw@mail.gmail.com>

Sure, but even list has copy().  I meant settable property on the generator
not on the generated permutation.

On Mon, Nov 17, 2014 at 12:44 PM, Ram Rachum <ram at rachum.com> wrote:

> Thanks :)
>
> Settable properties are a no-go because I wanted permutation spaces to be
> immutable. Since the cost of creating a new space is nil (permutations are
> created on-demand, not on space creation) there isn't a reason to mutate an
> existing space.
>
> I don't think using a copy() method to would be very nice. But I guess
> it's a matter of taste.
>
> On Mon, Nov 17, 2014 at 7:33 PM, Neil Girdhar <mistersheik at gmail.com>
> wrote:
>
>> Looks great!
>>
>> Why did you go with "get_rapplied", "unrapplied", etc. instead of having
>> a copy() method and using settable properties?
>>
>> On Mon, Nov 17, 2014 at 10:50 AM, Ram Rachum <ram at rachum.com> wrote:
>>
>>> Hi guys,
>>>
>>> I just wanted to give an update on this: I just released my own code
>>> that does this to PyPI: https://pypi.python.org/pypi/combi
>>>
>>>
>>> Thanks,
>>> Ram.
>>>
>>> On Tue, Jun 10, 2014 at 9:24 AM, Ram Rachum <ram at rachum.com> wrote:
>>>
>>>> I'll email you if/when it's released :)
>>>>
>>>>
>>>> On Tue, Jun 10, 2014 at 9:04 AM, Neil Girdhar <mistersheik at gmail.com>
>>>> wrote:
>>>>
>>>>> I really like this and hope that it eventually makes it into the
>>>>> stdlib.  It's also a good argument for your other suggestion whereby some
>>>>> of the itertools to return Iterables rather than Iterators like range does.
>>>>>
>>>>> Best,
>>>>>
>>>>> Neil
>>>>>
>>>>> On Wednesday, May 7, 2014 1:43:20 PM UTC-4, Ram Rachum wrote:
>>>>>
>>>>>> I'm probably going to implement it in my python_toolbox package. I
>>>>>> already implemented 30% and it's really cool. It's at the point where I
>>>>>> doubt that I want it in the stdlib because I've gotten so much awesome
>>>>>> functionality into it and I'd hate to (a) have 80% of it stripped and (b)
>>>>>> have the class names changed to be non-Pythonic :)
>>>>>>
>>>>>>
>>>>>> On Wed, May 7, 2014 at 8:40 PM, Tal Einat <tale... at gmail.com> wrote:
>>>>>>
>>>>>> On Wed, May 7, 2014 at 8:21 PM, Ram Rachum <ram.r... at gmail.com>
>>>>>>> wrote:
>>>>>>> > Hi Tal,
>>>>>>> >
>>>>>>> > I'm using it for a project of my own (optimizing keyboard layout)
>>>>>>> but I
>>>>>>> > can't make the case that it's useful for the stdlib. I'd
>>>>>>> understand if it
>>>>>>> > would be omitted for not being enough of a common need.
>>>>>>>
>>>>>>> At the least, this (a function for getting a specific permutation by
>>>>>>> lexicographical-order index) could make a nice cookbook recipe.
>>>>>>>
>>>>>>> - Tal
>>>>>>>
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> Python-ideas mailing list
>>>>> Python-ideas at python.org
>>>>> https://mail.python.org/mailman/listinfo/python-ideas
>>>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>>> --
>>>>>
>>>>> ---
>>>>> You received this message because you are subscribed to a topic in the
>>>>> Google Groups "python-ideas" group.
>>>>> To unsubscribe from this topic, visit
>>>>> https://groups.google.com/d/topic/python-ideas/in2gUQMFUzA/unsubscribe
>>>>> .
>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>> python-ideas+unsubscribe at googlegroups.com.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141117/0500580b/attachment-0001.html>

From guido at python.org  Mon Nov 17 19:01:46 2014
From: guido at python.org (Guido van Rossum)
Date: Mon, 17 Nov 2014 10:01:46 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CADiSq7dtEphyJFWOFj6edhte0BA6aAjoXo72r7RLNMu+XbdNSA@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com>
 <CAPTjJmqacEJqexWnUU1UBoC_VcsPKSoVw1_9uqYOyAXAbsLxYw@mail.gmail.com>
 <54691F8D.1080703@canterbury.ac.nz> <20141117000302.GS2748@ando.pearwood.info>
 <CAPTjJmoHgSSL0L3Bh5_qXDB=mtd-6_X70WhD-_WqoXj3qzZuSw@mail.gmail.com>
 <54696A02.6050602@canterbury.ac.nz>
 <CAPTjJmrvt5wLkE5DLgYNwgHyLuTkh4nyszM3ur-jV8ZkO+90LA@mail.gmail.com>
 <CADiSq7dtEphyJFWOFj6edhte0BA6aAjoXo72r7RLNMu+XbdNSA@mail.gmail.com>
Message-ID: <CAP7+vJ+DwxYkwog9fakyqQsd4LsBnLce=aWJgW8dN-7dcWxvCA@mail.gmail.com>

On Mon, Nov 17, 2014 at 4:56 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 17 November 2014 13:34, Chris Angelico <rosuav at gmail.com> wrote:
>
>> Right. So, are there any __future__ directives that have any effect on
>> byte-code? I'm not seeing any, though that may just mean I didn't
>> recognize one.
>>
>
> True division (in Python 2) is a nice simple one to look at, since it just
> swaps one bytecode for another (BINARY_DIVIDE -> BINARY_TRUE_DIVIDE)
>
> >>> def floor_div(x, y):
> ...     return x / y
> ...
> >>> from __future__ import division
> >>> def true_div(x, y):
> ...     return x / y
> ...
> >>> import dis
> >>> dis.dis(floor_div)
>   2           0 LOAD_FAST                0 (x)
>               3 LOAD_FAST                1 (y)
>               6 BINARY_DIVIDE
>               7 RETURN_VALUE
> >>> dis.dis(true_div)
>   2           0 LOAD_FAST                0 (x)
>               3 LOAD_FAST                1 (y)
>               6 BINARY_TRUE_DIVIDE
>               7 RETURN_VALUE
>
> The compiler actually stores a whole pile of useful info on code objects
> that doesn't show up in the disassembly output (switching to Python 3 for
> more up to date dis module goodness):
>
> >>> import dis
> >>> def true_div(x, y):
> ...     return x / y
> ...
> >>> dis.show_code(true_div)
> Name:              true_div
> Filename:          <stdin>
> Argument count:    2
> Kw-only arguments: 0
> Number of locals:  2
> Stack size:        2
> Flags:             OPTIMIZED, NEWLOCALS, NOFREE
> Constants:
>    0: None
> Variable names:
>    0: x
>    1: y
>
> So conveying to the generator iterator whether or not "from __future__
> import generator_return" was in effect would just be a matter of the
> compiler setting a new flag on the generator code object. For *affected
> generators* (i.e. those defined in a module where the new future statement
> was in effect), StopIteration escaping would be considered a RuntimeError.
>
> For almost all code, such RuntimeErrors would look like any other
> RuntimError raised by a broken generator implementation.
>
> The only code which would *have* to change immediately as a "Porting to
> Python 3.5" requirement is code like that in contextlib, which throws
> StopIteration into generators, and currently expects to get it back out
> unmodified. Such code will need to be updated to also handle RuntimError
> instances where the direct cause is the StopIteration exception that was
> thrown in.
>
> Other affected code (such as the "next() bubbling up" groupby example)
> would keep working unless the __future__ statement was in effect.
>

Thanks, this is exactly what I was thinking of. The new flag could be named
REPLACE_STOPITERATION. Then the __future__ import could be named
replace_stopiteration_in_generators (it needs more description than the
flag name because the flag is already defined in the context of a
generator, while the __future__ import must still establish that context).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141117/7b43a54f/attachment.html>

From rosuav at gmail.com  Mon Nov 17 19:02:36 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 18 Nov 2014 05:02:36 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAP7+vJJY=Phea--AaOehp-SQky0_BYYKYsAjW-+Mwc-TBoU41Q@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com>
 <CAPTjJmqacEJqexWnUU1UBoC_VcsPKSoVw1_9uqYOyAXAbsLxYw@mail.gmail.com>
 <54691F8D.1080703@canterbury.ac.nz>
 <20141117000302.GS2748@ando.pearwood.info>
 <CAPTjJmoHgSSL0L3Bh5_qXDB=mtd-6_X70WhD-_WqoXj3qzZuSw@mail.gmail.com>
 <CAP7+vJJ0m_PipGY0S172Nt3cZvJwCQFmOhxNCBioggFAYzU4kw@mail.gmail.com>
 <CAPTjJmogxXu1QkkXhSrkRac2HD5Ki2NKLTaLARdB+=xeOnL-dw@mail.gmail.com>
 <m4cevm$735$1@ger.gmane.org>
 <CAPTjJmp2XZy56dWRUQPgMYZhG-8pwqSPFjagxYmJ_3rVmb9_GA@mail.gmail.com>
 <CAP7+vJJY=Phea--AaOehp-SQky0_BYYKYsAjW-+Mwc-TBoU41Q@mail.gmail.com>
Message-ID: <CAPTjJmqVAvPFRZq_XCOfDuaK3qahQjqrXCypMW+hg+zh11SQAA@mail.gmail.com>

On Tue, Nov 18, 2014 at 4:28 AM, Guido van Rossum <guido at python.org> wrote:
> Off-topic: the new python.org site now supports PEPs, so please switch to
> URLs like this: https://www.python.org/dev/peps/pep-0479/ (if you don't like
> the formatting send a pull request to
> https://github.com/python/pythondotorg).

Oh, nice. Google searches for PEPs still find legacy rather than that,
so it may be worth setting legacy to redirect to www. Formatting looks
fine, except that the subheadings look bolder than the main; I'll
check for issues and post one, though probably not a PR.

ChrisA

From rosuav at gmail.com  Mon Nov 17 19:09:49 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 18 Nov 2014 05:09:49 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
Message-ID: <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>

On Tue, Nov 18, 2014 at 4:40 AM, Guido van Rossum <guido at python.org> wrote:
> Wapping each result would just result in an extra allocation + deallocation
> per iteration...

Which is why I would be more inclined to use a sentinel of some
sort... but that has its own problems. There's no perfect solution, so
status quo wins unless a really compelling case can be made. I could
toss something into the Alternate Proposals section, but I wouldn't be
personally supporting it.

ChrisA

From guido at python.org  Mon Nov 17 19:09:36 2014
From: guido at python.org (Guido van Rossum)
Date: Mon, 17 Nov 2014 10:09:36 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CABicbJKeSWpMqA_NidZcdinMgqTKEa1-NTw_u02t+UzTv9d=kw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
 <CABicbJKeSWpMqA_NidZcdinMgqTKEa1-NTw_u02t+UzTv9d=kw@mail.gmail.com>
Message-ID: <CAP7+vJKP4Ms9tT5i6+jUnvToi55MJKFL0ZzNaX6dp7TMj_1KEA@mail.gmail.com>

On Mon, Nov 17, 2014 at 9:53 AM, Devin Jeanpierre <jeanpierreda at gmail.com>
wrote:
[Guido]

> > The problem we're trying to fix is unique to generators (thereby also
> > implicating generator expressions).
>
> I suppose since you're only fixing generators, then that is literally
> the only problem you are trying to fix, but it is more general than
> that. I have encountered this sort of problem writing __next__ by hand
> in Python -- that is, that bugs inside code I call result in silent
> control flow changes rather than a visible exception.
>

I assume this is something where the __next__() method on your iterator
class calls next() on some other iterator and accidentally doesn't catch
the StopIteration coming out of it (or, more likely, this happens several
calls deep, making it more interesting to debug).

That particular problem is not unique to __next__ and StopIteration -- the
same thing can (and does!) happen with __getitem__ and KeyError or
IndexError, and with __getattr[ibute]__ and AttributeError.

In all these cases I think there isn't much we can do apart from adding
lint rules. If you are writing __next__ as a method on an iterator class,
one way or another you are going to have to raise StopIteration when there
isn't another element, and similarly __getitem__ has to raise KeyError or
IndexError, etc.

In the generator case, we have a better way to signal the end -- a return
statement (or falling off the end). And that's why we can even contemplate
doing something different when StopIteration is raised in the generator.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141117/167343c8/attachment-0001.html>

From rosuav at gmail.com  Mon Nov 17 19:12:50 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 18 Nov 2014 05:12:50 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CABicbJKeSWpMqA_NidZcdinMgqTKEa1-NTw_u02t+UzTv9d=kw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
 <CABicbJKeSWpMqA_NidZcdinMgqTKEa1-NTw_u02t+UzTv9d=kw@mail.gmail.com>
Message-ID: <CAPTjJmqvRFfaifk0ZtBcEgkiOJBL4LtvqjbR_HYdR_pY2KVUAA@mail.gmail.com>

On Tue, Nov 18, 2014 at 4:53 AM, Devin Jeanpierre
<jeanpierreda at gmail.com> wrote:
> I suppose since you're only fixing generators, then that is literally
> the only problem you are trying to fix, but it is more general than
> that. I have encountered this sort of problem writing __next__ by hand
> in Python -- that is, that bugs inside code I call result in silent
> control flow changes rather than a visible exception.

If you're writing __next__ by hand, there's nothing anyone else can do
about a bubbled-up StopIteration. What you can do is wrap your code in
try/except:

def __next__(self):
    try:
        # whatever
    except StopIteration:
        raise RuntimeError
    raise StopIteration

If your "whatever" section returns a value, that's what the result
will be. If it fails to return a value, StopIteration will be raised.
And if StopIteration is raised, it'll become RuntimeError.

But this has to be inside __next__. This can't be done externally.

(Note, untested code. May have bugs.)

ChrisA

From guido at python.org  Mon Nov 17 19:17:21 2014
From: guido at python.org (Guido van Rossum)
Date: Mon, 17 Nov 2014 10:17:21 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
 <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
Message-ID: <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>

On Mon, Nov 17, 2014 at 10:09 AM, Chris Angelico <rosuav at gmail.com> wrote:

> On Tue, Nov 18, 2014 at 4:40 AM, Guido van Rossum <guido at python.org>
> wrote:
> > Wapping each result would just result in an extra allocation +
> deallocation
> > per iteration...
>
> Which is why I would be more inclined to use a sentinel of some
> sort... but that has its own problems. There's no perfect solution, so
> status quo wins unless a really compelling case can be made. I could
> toss something into the Alternate Proposals section, but I wouldn't be
> personally supporting it.
>

Trust me, we went down this rabbit hole when we designed the iterator
protocol. The problem is that that sentinel object must have a name
(otherwise how would you know when you had seen the sentinel), which means
that there is at least one dict (the namespace defining that name, probably
the builtins module) that has the sentinel as one of its values, which
means that iterating over that particular dict's values would see the
sentinel as a legitimate value (and terminate prematurely). You could fix
this by allocating a unique sentinel for every iteration, but there would
be some additional overhead for that too (the caller and the iterator both
need to hold on to the sentinel object).

In any case, as I tried to say before, redesigning the iterator protocol is
most definitely out of scope here (you could write a separate PEP and I'd
reject it instantly).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141117/7d74b374/attachment.html>

From rosuav at gmail.com  Mon Nov 17 19:23:25 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 18 Nov 2014 05:23:25 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
 <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
Message-ID: <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>

On Tue, Nov 18, 2014 at 5:17 AM, Guido van Rossum <guido at python.org> wrote:
> Trust me, we went down this rabbit hole when we designed the iterator
> protocol. The problem is that that sentinel object must have a name
> (otherwise how would you know when you had seen the sentinel), which means
> that there is at least one dict (the namespace defining that name, probably
> the builtins module) that has the sentinel as one of its values, which means
> that iterating over that particular dict's values would see the sentinel as
> a legitimate value (and terminate prematurely). You could fix this by
> allocating a unique sentinel for every iteration, but there would be some
> additional overhead for that too (the caller and the iterator both need to
> hold on to the sentinel object).

That's a much more solid argument against it than I had. (Also amusing
to try to contemplate.) The iterator protocol absolutely demands a
completely out-of-band means of signalling "I have nothing to return
now".

> In any case, as I tried to say before, redesigning the iterator protocol is
> most definitely out of scope here (you could write a separate PEP and I'd
> reject it instantly).

That's a PEP I'll leave for someone else to write. :)

ChrisA

From ethan at stoneleaf.us  Mon Nov 17 19:26:54 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 17 Nov 2014 10:26:54 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
 generators
In-Reply-To: <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
 <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
Message-ID: <546A3DEE.5040206@stoneleaf.us>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/17/2014 10:23 AM, Chris Angelico wrote:
> On Tue, Nov 18, 2014 at 5:17 AM, Guido van Rossum wrote:
>> 
>> In any case, as I tried to say before, redesigning the iterator protocol is most definitely out of scope here
>> (you could write a separate PEP and I'd reject it instantly).
> 
> That's a PEP I'll leave for someone else to write. :)

On the other hand, if you did write it (now), and Guido rejected it (of course), then that would mean PEP 479 is sure
to be accepted!  Third time's the charm!  ;)

- --
~Ethan~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQIcBAEBAgAGBQJUaj3uAAoJENZ7D1rrH75NzRYP/2NcwwNOQvD71RlhJG0k0sUY
xLgjzLSCVQbR1jliF1Ej9BEZhFhd1I4lg/msjDPNBSHABPtsbrRTzZiac4x3OAEz
5L5qCB7hIaHg+gDKEbCiOl5NAZUbZL+Qs1tFdlSjeCti/YUAVE1DiD4WQ+kXHxZ4
OF9Nj5urHrGQYh//HR3RryQddihPBkM9Y40oucRwQE32jzcJacgmEvOM4aVvJHb8
2rBfl7VjpHwC2dV51R3cYpwIu2Qqk9i4kEnkB9xDtj9HQl3YT+qEyAuZWOs7v3cH
cJOjBuIxJGYF/CbjRE5/mCFegvGv4Lf2C8QSeKlqR549IPhso+qcnLvTQNdBWer0
g9L8Gsk1whUR9GQ3HGaupAkAo8rbZKM9fjvEtnPvcWAPcGZDfeFG5MEl82SCOsLS
aUl+v5u2sVozxIsY9fr8s30X2HZReaVjuJFDLCosZmi3RcqYL7O0x5+Mo4GPgsbi
YxyCYU0c0LTTk7yljweYL/oWXyUS/hAxz2VNMyAjDNw1Uy1kWwS6qAW5FcMxfdlz
I92CBx0PXPLz1hhHYsJ/YnHIJpL9+HRHHRRdJOnHOGCyb7AX6jN3ruMGYaTClrYa
G2ez3JMeu0NhTuzpFF2mjO0PXTu/HuHq3bdv44jhZS0Zn67OQmNk845bZi5U3EaW
oJTRKMM/PTZNpspQE0/n
=ZYwx
-----END PGP SIGNATURE-----

From ram at rachum.com  Mon Nov 17 20:37:05 2014
From: ram at rachum.com (Ram Rachum)
Date: Mon, 17 Nov 2014 21:37:05 +0200
Subject: [Python-ideas] Implement `itertools.permutations.__getitem__`
	and `itertools.permutations.index`
In-Reply-To: <CAA68w_nx3RTeeMEoZSWjchdSgemc2dNVnq0uTQcDP_+ELTFFMw@mail.gmail.com>
References: <bf592625-2396-471f-af87-47ad0375819e@googlegroups.com>
 <20140505171538.GR4273@ando>
 <CAEfz+TwtUovHQrsqMUcdihY-MxFxBVyDFXOg32jD30XQ0Gke9Q@mail.gmail.com>
 <20140506023902.GV4273@ando>
 <CALWZvp4_RphieoFU+MCCKeT10=6iu7xDXFq2W8zMg+xj0QDeZQ@mail.gmail.com>
 <CACac1F_kneXtP8wbDZqGp5T4_shwEqbLNSdUAuDEr3Hopdy43w@mail.gmail.com>
 <CALWZvp6OcLpZY-Hczk6Hw59sLpaYS0SeTjf1Vm4sS+i8qyvjSA@mail.gmail.com>
 <CANXboVZwE47SPD8ywcQ19reNb0hyDknaLEX1aoX-NkBzrSK24w@mail.gmail.com>
 <CALWZvp7Eh29rR8OcSUy4ZsK9w5CSbSj6MxWYru1bWt5jUX_66A@mail.gmail.com>
 <CANXboVZxF5J21O78udpGr1SndLxUpY2xYdq9oEP=f6hPq3kBGg@mail.gmail.com>
 <18aa148c-f6cb-414c-9f6c-de5011568204@googlegroups.com>
 <CANXboVb5K4cfTb1JsRbQdOzHgroFRAi2H006dfFc6Y-Qv=vnQA@mail.gmail.com>
 <CANXboVZqDY5pNC0yioa7ZXzPTdSR3yWp08MRVV8e=D2Uuha5OA@mail.gmail.com>
 <CAA68w_mT0TvOh7gTye3n9wNR-xy8QV5EPP0PUj6vwxvQFDFTRg@mail.gmail.com>
 <CANXboVYQV+U06NWCLezDn8pmtsLv3=KCx86pF=wCx2WUrG9oKA@mail.gmail.com>
 <CAA68w_nx3RTeeMEoZSWjchdSgemc2dNVnq0uTQcDP_+ELTFFMw@mail.gmail.com>
Message-ID: <CANXboVb5c1=N8Q+sngHcL-czwOST8m4De9KpFfR7YSKhFBS0eg@mail.gmail.com>

I'm not sure what you mean. Can you please give an example?

On Mon, Nov 17, 2014 at 8:01 PM, Neil Girdhar <mistersheik at gmail.com> wrote:

> Sure, but even list has copy().  I meant settable property on the
> generator not on the generated permutation.
>
> On Mon, Nov 17, 2014 at 12:44 PM, Ram Rachum <ram at rachum.com> wrote:
>
>> Thanks :)
>>
>> Settable properties are a no-go because I wanted permutation spaces to be
>> immutable. Since the cost of creating a new space is nil (permutations are
>> created on-demand, not on space creation) there isn't a reason to mutate an
>> existing space.
>>
>> I don't think using a copy() method to would be very nice. But I guess
>> it's a matter of taste.
>>
>> On Mon, Nov 17, 2014 at 7:33 PM, Neil Girdhar <mistersheik at gmail.com>
>> wrote:
>>
>>> Looks great!
>>>
>>> Why did you go with "get_rapplied", "unrapplied", etc. instead of having
>>> a copy() method and using settable properties?
>>>
>>> On Mon, Nov 17, 2014 at 10:50 AM, Ram Rachum <ram at rachum.com> wrote:
>>>
>>>> Hi guys,
>>>>
>>>> I just wanted to give an update on this: I just released my own code
>>>> that does this to PyPI: https://pypi.python.org/pypi/combi
>>>>
>>>>
>>>> Thanks,
>>>> Ram.
>>>>
>>>> On Tue, Jun 10, 2014 at 9:24 AM, Ram Rachum <ram at rachum.com> wrote:
>>>>
>>>>> I'll email you if/when it's released :)
>>>>>
>>>>>
>>>>> On Tue, Jun 10, 2014 at 9:04 AM, Neil Girdhar <mistersheik at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I really like this and hope that it eventually makes it into the
>>>>>> stdlib.  It's also a good argument for your other suggestion whereby some
>>>>>> of the itertools to return Iterables rather than Iterators like range does.
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Neil
>>>>>>
>>>>>> On Wednesday, May 7, 2014 1:43:20 PM UTC-4, Ram Rachum wrote:
>>>>>>
>>>>>>> I'm probably going to implement it in my python_toolbox package. I
>>>>>>> already implemented 30% and it's really cool. It's at the point where I
>>>>>>> doubt that I want it in the stdlib because I've gotten so much awesome
>>>>>>> functionality into it and I'd hate to (a) have 80% of it stripped and (b)
>>>>>>> have the class names changed to be non-Pythonic :)
>>>>>>>
>>>>>>>
>>>>>>> On Wed, May 7, 2014 at 8:40 PM, Tal Einat <tale... at gmail.com> wrote:
>>>>>>>
>>>>>>> On Wed, May 7, 2014 at 8:21 PM, Ram Rachum <ram.r... at gmail.com>
>>>>>>>> wrote:
>>>>>>>> > Hi Tal,
>>>>>>>> >
>>>>>>>> > I'm using it for a project of my own (optimizing keyboard layout)
>>>>>>>> but I
>>>>>>>> > can't make the case that it's useful for the stdlib. I'd
>>>>>>>> understand if it
>>>>>>>> > would be omitted for not being enough of a common need.
>>>>>>>>
>>>>>>>> At the least, this (a function for getting a specific permutation by
>>>>>>>> lexicographical-order index) could make a nice cookbook recipe.
>>>>>>>>
>>>>>>>> - Tal
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Python-ideas mailing list
>>>>>> Python-ideas at python.org
>>>>>> https://mail.python.org/mailman/listinfo/python-ideas
>>>>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>>>> --
>>>>>>
>>>>>> ---
>>>>>> You received this message because you are subscribed to a topic in
>>>>>> the Google Groups "python-ideas" group.
>>>>>> To unsubscribe from this topic, visit
>>>>>> https://groups.google.com/d/topic/python-ideas/in2gUQMFUzA/unsubscribe
>>>>>> .
>>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>>> python-ideas+unsubscribe at googlegroups.com.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141117/28b0c5e5/attachment-0001.html>

From guido at python.org  Mon Nov 17 20:38:27 2014
From: guido at python.org (Guido van Rossum)
Date: Mon, 17 Nov 2014 11:38:27 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <546A3DEE.5040206@stoneleaf.us>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
 <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
Message-ID: <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>

FWIW, I spent some time this morning close-reading the PEP, and made a
somewhat significant set of updates -- adding more specifics about the
proposed new __futue__ statement and the new code object flag, tracking
down a few more examples of code that would be affected, and some other
minor edits. Here's the diff:

  https://hg.python.org/peps/rev/8de949863677

Hopefully the new version will soon be here:

  https://www.python.org/dev/peps/pep-0479

Note that I am definitely not yet deciding on this PEP. I would love it if
people sent in examples of code using generator expressions that would be
affected by this change (either by highlighting a bug in the code or by
breaking what currently works).

If this PEP gets rejected, we could resurrect the GeneratorExit proposal
currently listed as an alternative -- although the more I think about that
the less I think it's worth it, except for the very specific case of
asyncio (thinking of which, I should add something to the PEP about that
too).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141117/73d57f08/attachment.html>

From mistersheik at gmail.com  Mon Nov 17 20:40:39 2014
From: mistersheik at gmail.com (Neil Girdhar)
Date: Mon, 17 Nov 2014 14:40:39 -0500
Subject: [Python-ideas] Implement `itertools.permutations.__getitem__`
	and `itertools.permutations.index`
In-Reply-To: <CANXboVb5c1=N8Q+sngHcL-czwOST8m4De9KpFfR7YSKhFBS0eg@mail.gmail.com>
References: <bf592625-2396-471f-af87-47ad0375819e@googlegroups.com>
 <20140505171538.GR4273@ando>
 <CAEfz+TwtUovHQrsqMUcdihY-MxFxBVyDFXOg32jD30XQ0Gke9Q@mail.gmail.com>
 <20140506023902.GV4273@ando>
 <CALWZvp4_RphieoFU+MCCKeT10=6iu7xDXFq2W8zMg+xj0QDeZQ@mail.gmail.com>
 <CACac1F_kneXtP8wbDZqGp5T4_shwEqbLNSdUAuDEr3Hopdy43w@mail.gmail.com>
 <CALWZvp6OcLpZY-Hczk6Hw59sLpaYS0SeTjf1Vm4sS+i8qyvjSA@mail.gmail.com>
 <CANXboVZwE47SPD8ywcQ19reNb0hyDknaLEX1aoX-NkBzrSK24w@mail.gmail.com>
 <CALWZvp7Eh29rR8OcSUy4ZsK9w5CSbSj6MxWYru1bWt5jUX_66A@mail.gmail.com>
 <CANXboVZxF5J21O78udpGr1SndLxUpY2xYdq9oEP=f6hPq3kBGg@mail.gmail.com>
 <18aa148c-f6cb-414c-9f6c-de5011568204@googlegroups.com>
 <CANXboVb5K4cfTb1JsRbQdOzHgroFRAi2H006dfFc6Y-Qv=vnQA@mail.gmail.com>
 <CANXboVZqDY5pNC0yioa7ZXzPTdSR3yWp08MRVV8e=D2Uuha5OA@mail.gmail.com>
 <CAA68w_mT0TvOh7gTye3n9wNR-xy8QV5EPP0PUj6vwxvQFDFTRg@mail.gmail.com>
 <CANXboVYQV+U06NWCLezDn8pmtsLv3=KCx86pF=wCx2WUrG9oKA@mail.gmail.com>
 <CAA68w_nx3RTeeMEoZSWjchdSgemc2dNVnq0uTQcDP_+ELTFFMw@mail.gmail.com>
 <CANXboVb5c1=N8Q+sngHcL-czwOST8m4De9KpFfR7YSKhFBS0eg@mail.gmail.com>
Message-ID: <CAA68w_mXTa8+iT0Yyffo26MGeQETEfYD_=7iVCknUVQNmmg9wQ@mail.gmail.com>

perm_space = PermSpace(3)

perm_space.degrees = 3

list(perm_space)

perm_space.degrees = None

list(perm_space)

etc.

On Mon, Nov 17, 2014 at 2:37 PM, Ram Rachum <ram at rachum.com> wrote:

> I'm not sure what you mean. Can you please give an example?
>
> On Mon, Nov 17, 2014 at 8:01 PM, Neil Girdhar <mistersheik at gmail.com>
> wrote:
>
>> Sure, but even list has copy().  I meant settable property on the
>> generator not on the generated permutation.
>>
>> On Mon, Nov 17, 2014 at 12:44 PM, Ram Rachum <ram at rachum.com> wrote:
>>
>>> Thanks :)
>>>
>>> Settable properties are a no-go because I wanted permutation spaces to
>>> be immutable. Since the cost of creating a new space is nil (permutations
>>> are created on-demand, not on space creation) there isn't a reason to
>>> mutate an existing space.
>>>
>>> I don't think using a copy() method to would be very nice. But I guess
>>> it's a matter of taste.
>>>
>>> On Mon, Nov 17, 2014 at 7:33 PM, Neil Girdhar <mistersheik at gmail.com>
>>> wrote:
>>>
>>>> Looks great!
>>>>
>>>> Why did you go with "get_rapplied", "unrapplied", etc. instead of
>>>> having a copy() method and using settable properties?
>>>>
>>>> On Mon, Nov 17, 2014 at 10:50 AM, Ram Rachum <ram at rachum.com> wrote:
>>>>
>>>>> Hi guys,
>>>>>
>>>>> I just wanted to give an update on this: I just released my own code
>>>>> that does this to PyPI: https://pypi.python.org/pypi/combi
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Ram.
>>>>>
>>>>> On Tue, Jun 10, 2014 at 9:24 AM, Ram Rachum <ram at rachum.com> wrote:
>>>>>
>>>>>> I'll email you if/when it's released :)
>>>>>>
>>>>>>
>>>>>> On Tue, Jun 10, 2014 at 9:04 AM, Neil Girdhar <mistersheik at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I really like this and hope that it eventually makes it into the
>>>>>>> stdlib.  It's also a good argument for your other suggestion whereby some
>>>>>>> of the itertools to return Iterables rather than Iterators like range does.
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Neil
>>>>>>>
>>>>>>> On Wednesday, May 7, 2014 1:43:20 PM UTC-4, Ram Rachum wrote:
>>>>>>>
>>>>>>>> I'm probably going to implement it in my python_toolbox package. I
>>>>>>>> already implemented 30% and it's really cool. It's at the point where I
>>>>>>>> doubt that I want it in the stdlib because I've gotten so much awesome
>>>>>>>> functionality into it and I'd hate to (a) have 80% of it stripped and (b)
>>>>>>>> have the class names changed to be non-Pythonic :)
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, May 7, 2014 at 8:40 PM, Tal Einat <tale... at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> On Wed, May 7, 2014 at 8:21 PM, Ram Rachum <ram.r... at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>> > Hi Tal,
>>>>>>>>> >
>>>>>>>>> > I'm using it for a project of my own (optimizing keyboard
>>>>>>>>> layout) but I
>>>>>>>>> > can't make the case that it's useful for the stdlib. I'd
>>>>>>>>> understand if it
>>>>>>>>> > would be omitted for not being enough of a common need.
>>>>>>>>>
>>>>>>>>> At the least, this (a function for getting a specific permutation
>>>>>>>>> by
>>>>>>>>> lexicographical-order index) could make a nice cookbook recipe.
>>>>>>>>>
>>>>>>>>> - Tal
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Python-ideas mailing list
>>>>>>> Python-ideas at python.org
>>>>>>> https://mail.python.org/mailman/listinfo/python-ideas
>>>>>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>>>>> --
>>>>>>>
>>>>>>> ---
>>>>>>> You received this message because you are subscribed to a topic in
>>>>>>> the Google Groups "python-ideas" group.
>>>>>>> To unsubscribe from this topic, visit
>>>>>>> https://groups.google.com/d/topic/python-ideas/in2gUQMFUzA/unsubscribe
>>>>>>> .
>>>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>>>> python-ideas+unsubscribe at googlegroups.com.
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141117/3a179996/attachment.html>

From ram at rachum.com  Mon Nov 17 20:57:55 2014
From: ram at rachum.com (Ram Rachum)
Date: Mon, 17 Nov 2014 21:57:55 +0200
Subject: [Python-ideas] Implement `itertools.permutations.__getitem__`
	and `itertools.permutations.index`
In-Reply-To: <CAA68w_mXTa8+iT0Yyffo26MGeQETEfYD_=7iVCknUVQNmmg9wQ@mail.gmail.com>
References: <bf592625-2396-471f-af87-47ad0375819e@googlegroups.com>
 <20140505171538.GR4273@ando>
 <CAEfz+TwtUovHQrsqMUcdihY-MxFxBVyDFXOg32jD30XQ0Gke9Q@mail.gmail.com>
 <20140506023902.GV4273@ando>
 <CALWZvp4_RphieoFU+MCCKeT10=6iu7xDXFq2W8zMg+xj0QDeZQ@mail.gmail.com>
 <CACac1F_kneXtP8wbDZqGp5T4_shwEqbLNSdUAuDEr3Hopdy43w@mail.gmail.com>
 <CALWZvp6OcLpZY-Hczk6Hw59sLpaYS0SeTjf1Vm4sS+i8qyvjSA@mail.gmail.com>
 <CANXboVZwE47SPD8ywcQ19reNb0hyDknaLEX1aoX-NkBzrSK24w@mail.gmail.com>
 <CALWZvp7Eh29rR8OcSUy4ZsK9w5CSbSj6MxWYru1bWt5jUX_66A@mail.gmail.com>
 <CANXboVZxF5J21O78udpGr1SndLxUpY2xYdq9oEP=f6hPq3kBGg@mail.gmail.com>
 <18aa148c-f6cb-414c-9f6c-de5011568204@googlegroups.com>
 <CANXboVb5K4cfTb1JsRbQdOzHgroFRAi2H006dfFc6Y-Qv=vnQA@mail.gmail.com>
 <CANXboVZqDY5pNC0yioa7ZXzPTdSR3yWp08MRVV8e=D2Uuha5OA@mail.gmail.com>
 <CAA68w_mT0TvOh7gTye3n9wNR-xy8QV5EPP0PUj6vwxvQFDFTRg@mail.gmail.com>
 <CANXboVYQV+U06NWCLezDn8pmtsLv3=KCx86pF=wCx2WUrG9oKA@mail.gmail.com>
 <CAA68w_nx3RTeeMEoZSWjchdSgemc2dNVnq0uTQcDP_+ELTFFMw@mail.gmail.com>
 <CANXboVb5c1=N8Q+sngHcL-czwOST8m4De9KpFfR7YSKhFBS0eg@mail.gmail.com>
 <CAA68w_mXTa8+iT0Yyffo26MGeQETEfYD_=7iVCknUVQNmmg9wQ@mail.gmail.com>
Message-ID: <CANXboVZ8daLpimuYyZPvSeoLUS169dfZBz70UsTKS59_vm8W+w@mail.gmail.com>

Ah, I understand. I don't like it, though I'm not sure I can really express
why. One reason is that this would mean that PermSpaces would be mutable,
and thus not hashable and not usable as keys in dicts and sets. It would
also mean I couldn't use the `CachedProperty` pattern all over the class as
I do today. So these are a couple of reasons, there might be more that
didn't come to me now.

On Mon, Nov 17, 2014 at 9:40 PM, Neil Girdhar <mistersheik at gmail.com> wrote:

> perm_space = PermSpace(3)
>
> perm_space.degrees = 3
>
> list(perm_space)
>
> perm_space.degrees = None
>
> list(perm_space)
>
> etc.
>
> On Mon, Nov 17, 2014 at 2:37 PM, Ram Rachum <ram at rachum.com> wrote:
>
>> I'm not sure what you mean. Can you please give an example?
>>
>> On Mon, Nov 17, 2014 at 8:01 PM, Neil Girdhar <mistersheik at gmail.com>
>> wrote:
>>
>>> Sure, but even list has copy().  I meant settable property on the
>>> generator not on the generated permutation.
>>>
>>> On Mon, Nov 17, 2014 at 12:44 PM, Ram Rachum <ram at rachum.com> wrote:
>>>
>>>> Thanks :)
>>>>
>>>> Settable properties are a no-go because I wanted permutation spaces to
>>>> be immutable. Since the cost of creating a new space is nil (permutations
>>>> are created on-demand, not on space creation) there isn't a reason to
>>>> mutate an existing space.
>>>>
>>>> I don't think using a copy() method to would be very nice. But I guess
>>>> it's a matter of taste.
>>>>
>>>> On Mon, Nov 17, 2014 at 7:33 PM, Neil Girdhar <mistersheik at gmail.com>
>>>> wrote:
>>>>
>>>>> Looks great!
>>>>>
>>>>> Why did you go with "get_rapplied", "unrapplied", etc. instead of
>>>>> having a copy() method and using settable properties?
>>>>>
>>>>> On Mon, Nov 17, 2014 at 10:50 AM, Ram Rachum <ram at rachum.com> wrote:
>>>>>
>>>>>> Hi guys,
>>>>>>
>>>>>> I just wanted to give an update on this: I just released my own code
>>>>>> that does this to PyPI: https://pypi.python.org/pypi/combi
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Ram.
>>>>>>
>>>>>> On Tue, Jun 10, 2014 at 9:24 AM, Ram Rachum <ram at rachum.com> wrote:
>>>>>>
>>>>>>> I'll email you if/when it's released :)
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jun 10, 2014 at 9:04 AM, Neil Girdhar <mistersheik at gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> I really like this and hope that it eventually makes it into the
>>>>>>>> stdlib.  It's also a good argument for your other suggestion whereby some
>>>>>>>> of the itertools to return Iterables rather than Iterators like range does.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>>
>>>>>>>> Neil
>>>>>>>>
>>>>>>>> On Wednesday, May 7, 2014 1:43:20 PM UTC-4, Ram Rachum wrote:
>>>>>>>>
>>>>>>>>> I'm probably going to implement it in my python_toolbox package. I
>>>>>>>>> already implemented 30% and it's really cool. It's at the point where I
>>>>>>>>> doubt that I want it in the stdlib because I've gotten so much awesome
>>>>>>>>> functionality into it and I'd hate to (a) have 80% of it stripped and (b)
>>>>>>>>> have the class names changed to be non-Pythonic :)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, May 7, 2014 at 8:40 PM, Tal Einat <tale... at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> On Wed, May 7, 2014 at 8:21 PM, Ram Rachum <ram.r... at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>> > Hi Tal,
>>>>>>>>>> >
>>>>>>>>>> > I'm using it for a project of my own (optimizing keyboard
>>>>>>>>>> layout) but I
>>>>>>>>>> > can't make the case that it's useful for the stdlib. I'd
>>>>>>>>>> understand if it
>>>>>>>>>> > would be omitted for not being enough of a common need.
>>>>>>>>>>
>>>>>>>>>> At the least, this (a function for getting a specific permutation
>>>>>>>>>> by
>>>>>>>>>> lexicographical-order index) could make a nice cookbook recipe.
>>>>>>>>>>
>>>>>>>>>> - Tal
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Python-ideas mailing list
>>>>>>>> Python-ideas at python.org
>>>>>>>> https://mail.python.org/mailman/listinfo/python-ideas
>>>>>>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>>>>>> --
>>>>>>>>
>>>>>>>> ---
>>>>>>>> You received this message because you are subscribed to a topic in
>>>>>>>> the Google Groups "python-ideas" group.
>>>>>>>> To unsubscribe from this topic, visit
>>>>>>>> https://groups.google.com/d/topic/python-ideas/in2gUQMFUzA/unsubscribe
>>>>>>>> .
>>>>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>>>>> python-ideas+unsubscribe at googlegroups.com.
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141117/0f2b6120/attachment-0001.html>

From mistersheik at gmail.com  Mon Nov 17 20:59:50 2014
From: mistersheik at gmail.com (Neil Girdhar)
Date: Mon, 17 Nov 2014 14:59:50 -0500
Subject: [Python-ideas] Implement `itertools.permutations.__getitem__`
	and `itertools.permutations.index`
In-Reply-To: <CANXboVZ8daLpimuYyZPvSeoLUS169dfZBz70UsTKS59_vm8W+w@mail.gmail.com>
References: <bf592625-2396-471f-af87-47ad0375819e@googlegroups.com>
 <20140505171538.GR4273@ando>
 <CAEfz+TwtUovHQrsqMUcdihY-MxFxBVyDFXOg32jD30XQ0Gke9Q@mail.gmail.com>
 <20140506023902.GV4273@ando>
 <CALWZvp4_RphieoFU+MCCKeT10=6iu7xDXFq2W8zMg+xj0QDeZQ@mail.gmail.com>
 <CACac1F_kneXtP8wbDZqGp5T4_shwEqbLNSdUAuDEr3Hopdy43w@mail.gmail.com>
 <CALWZvp6OcLpZY-Hczk6Hw59sLpaYS0SeTjf1Vm4sS+i8qyvjSA@mail.gmail.com>
 <CANXboVZwE47SPD8ywcQ19reNb0hyDknaLEX1aoX-NkBzrSK24w@mail.gmail.com>
 <CALWZvp7Eh29rR8OcSUy4ZsK9w5CSbSj6MxWYru1bWt5jUX_66A@mail.gmail.com>
 <CANXboVZxF5J21O78udpGr1SndLxUpY2xYdq9oEP=f6hPq3kBGg@mail.gmail.com>
 <18aa148c-f6cb-414c-9f6c-de5011568204@googlegroups.com>
 <CANXboVb5K4cfTb1JsRbQdOzHgroFRAi2H006dfFc6Y-Qv=vnQA@mail.gmail.com>
 <CANXboVZqDY5pNC0yioa7ZXzPTdSR3yWp08MRVV8e=D2Uuha5OA@mail.gmail.com>
 <CAA68w_mT0TvOh7gTye3n9wNR-xy8QV5EPP0PUj6vwxvQFDFTRg@mail.gmail.com>
 <CANXboVYQV+U06NWCLezDn8pmtsLv3=KCx86pF=wCx2WUrG9oKA@mail.gmail.com>
 <CAA68w_nx3RTeeMEoZSWjchdSgemc2dNVnq0uTQcDP_+ELTFFMw@mail.gmail.com>
 <CANXboVb5c1=N8Q+sngHcL-czwOST8m4De9KpFfR7YSKhFBS0eg@mail.gmail.com>
 <CAA68w_mXTa8+iT0Yyffo26MGeQETEfYD_=7iVCknUVQNmmg9wQ@mail.gmail.com>
 <CANXboVZ8daLpimuYyZPvSeoLUS169dfZBz70UsTKS59_vm8W+w@mail.gmail.com>
Message-ID: <CAA68w_kqmeMoKz20iMsHuvJyn3N4y=BrNu_Z=-5yDFMVxLvcoQ@mail.gmail.com>

Those are good reasons, thanks. :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141117/4f5f4586/attachment.html>

From steve at pearwood.info  Tue Nov 18 00:05:17 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 18 Nov 2014 10:05:17 +1100
Subject: [Python-ideas] Implement `itertools.permutations.__getitem__`
	and `itertools.permutations.index`
In-Reply-To: <CANXboVZ8daLpimuYyZPvSeoLUS169dfZBz70UsTKS59_vm8W+w@mail.gmail.com>
References: <CANXboVZxF5J21O78udpGr1SndLxUpY2xYdq9oEP=f6hPq3kBGg@mail.gmail.com>
 <18aa148c-f6cb-414c-9f6c-de5011568204@googlegroups.com>
 <CANXboVb5K4cfTb1JsRbQdOzHgroFRAi2H006dfFc6Y-Qv=vnQA@mail.gmail.com>
 <CANXboVZqDY5pNC0yioa7ZXzPTdSR3yWp08MRVV8e=D2Uuha5OA@mail.gmail.com>
 <CAA68w_mT0TvOh7gTye3n9wNR-xy8QV5EPP0PUj6vwxvQFDFTRg@mail.gmail.com>
 <CANXboVYQV+U06NWCLezDn8pmtsLv3=KCx86pF=wCx2WUrG9oKA@mail.gmail.com>
 <CAA68w_nx3RTeeMEoZSWjchdSgemc2dNVnq0uTQcDP_+ELTFFMw@mail.gmail.com>
 <CANXboVb5c1=N8Q+sngHcL-czwOST8m4De9KpFfR7YSKhFBS0eg@mail.gmail.com>
 <CAA68w_mXTa8+iT0Yyffo26MGeQETEfYD_=7iVCknUVQNmmg9wQ@mail.gmail.com>
 <CANXboVZ8daLpimuYyZPvSeoLUS169dfZBz70UsTKS59_vm8W+w@mail.gmail.com>
Message-ID: <20141117230517.GU2748@ando.pearwood.info>

On Mon, Nov 17, 2014 at 09:57:55PM +0200, Ram Rachum wrote:
> Ah, I understand. I don't like it, though I'm not sure I can really express
> why. One reason is that this would mean that PermSpaces would be mutable,
> and thus not hashable and not usable as keys in dicts and sets.

I'm having trouble thinking of any reason to use a PermSpace as a dict 
key or set element. Do you have a concrete example of doing so?



-- 
Steven

From ram at rachum.com  Tue Nov 18 00:18:14 2014
From: ram at rachum.com (Ram Rachum)
Date: Tue, 18 Nov 2014 01:18:14 +0200
Subject: [Python-ideas] Implement `itertools.permutations.__getitem__`
	and `itertools.permutations.index`
In-Reply-To: <20141117230517.GU2748@ando.pearwood.info>
References: <CANXboVZxF5J21O78udpGr1SndLxUpY2xYdq9oEP=f6hPq3kBGg@mail.gmail.com>
 <18aa148c-f6cb-414c-9f6c-de5011568204@googlegroups.com>
 <CANXboVb5K4cfTb1JsRbQdOzHgroFRAi2H006dfFc6Y-Qv=vnQA@mail.gmail.com>
 <CANXboVZqDY5pNC0yioa7ZXzPTdSR3yWp08MRVV8e=D2Uuha5OA@mail.gmail.com>
 <CAA68w_mT0TvOh7gTye3n9wNR-xy8QV5EPP0PUj6vwxvQFDFTRg@mail.gmail.com>
 <CANXboVYQV+U06NWCLezDn8pmtsLv3=KCx86pF=wCx2WUrG9oKA@mail.gmail.com>
 <CAA68w_nx3RTeeMEoZSWjchdSgemc2dNVnq0uTQcDP_+ELTFFMw@mail.gmail.com>
 <CANXboVb5c1=N8Q+sngHcL-czwOST8m4De9KpFfR7YSKhFBS0eg@mail.gmail.com>
 <CAA68w_mXTa8+iT0Yyffo26MGeQETEfYD_=7iVCknUVQNmmg9wQ@mail.gmail.com>
 <CANXboVZ8daLpimuYyZPvSeoLUS169dfZBz70UsTKS59_vm8W+w@mail.gmail.com>
 <20141117230517.GU2748@ando.pearwood.info>
Message-ID: <CANXboVZ9VD9yx=v_zuNF6e2gRYr7=vpYBreHxQWNNWKPe+aA4A@mail.gmail.com>

I actually did that in the project I made Combi for. I had a bunch of perm
spaces that signified certain configurations (can't get more specific,
sorry) and I wanted to find the best configuration in each space. So I had
a dict from each space to the best configuration in that space.

On Tue, Nov 18, 2014 at 1:05 AM, Steven D'Aprano <steve at pearwood.info>
wrote:

> On Mon, Nov 17, 2014 at 09:57:55PM +0200, Ram Rachum wrote:
> > Ah, I understand. I don't like it, though I'm not sure I can really
> express
> > why. One reason is that this would mean that PermSpaces would be mutable,
> > and thus not hashable and not usable as keys in dicts and sets.
>
> I'm having trouble thinking of any reason to use a PermSpace as a dict
> key or set element. Do you have a concrete example of doing so?
>
>
>
> --
> Steven
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141118/effa7ab4/attachment.html>

From steve at pearwood.info  Tue Nov 18 00:56:42 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 18 Nov 2014 10:56:42 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
 <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
Message-ID: <20141117235641.GW2748@ando.pearwood.info>

On Mon, Nov 17, 2014 at 11:38:27AM -0800, Guido van Rossum wrote:

> Hopefully the new version will soon be here:
> 
>   https://www.python.org/dev/peps/pep-0479
> 
> Note that I am definitely not yet deciding on this PEP. I would love it if
> people sent in examples of code using generator expressions that would be
> affected by this change (either by highlighting a bug in the code or by
> breaking what currently works).

Over a week ago I raised this issue on python-list mailing list. I 
expected a storm of bike-shedding, because that's the sort of place p-l 
is :-) but got just two people commenting.

The thread, for anyone interested:

https://mail.python.org/pipermail/python-list/2014-November/680757.html

One response suggested that it is not generators which do the wrong 
thing, but comprehensions, and that comprehensions should be changed to 
behave like generators:

https://mail.python.org/pipermail/python-list/2014-November/680758.html

That should probably be put in the PEP, even if it is not an option 
being considered, it at least evidence that "some people" find the 
behaviour of generators more natural than that of comprehensions.


-- 
Steve

From guido at python.org  Tue Nov 18 01:13:23 2014
From: guido at python.org (Guido van Rossum)
Date: Mon, 17 Nov 2014 16:13:23 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <20141117235641.GW2748@ando.pearwood.info>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
 <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
Message-ID: <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>

Thanks, I added the survey and a summary to the PEP. If you hear about any
examples please do forward them here!

On Mon, Nov 17, 2014 at 3:56 PM, Steven D'Aprano <steve at pearwood.info>
wrote:

> On Mon, Nov 17, 2014 at 11:38:27AM -0800, Guido van Rossum wrote:
>
> > Hopefully the new version will soon be here:
> >
> >   https://www.python.org/dev/peps/pep-0479
> >
> > Note that I am definitely not yet deciding on this PEP. I would love it
> if
> > people sent in examples of code using generator expressions that would be
> > affected by this change (either by highlighting a bug in the code or by
> > breaking what currently works).
>
> Over a week ago I raised this issue on python-list mailing list. I
> expected a storm of bike-shedding, because that's the sort of place p-l
> is :-) but got just two people commenting.
>
> The thread, for anyone interested:
>
> https://mail.python.org/pipermail/python-list/2014-November/680757.html
>
> One response suggested that it is not generators which do the wrong
> thing, but comprehensions, and that comprehensions should be changed to
> behave like generators:
>
> https://mail.python.org/pipermail/python-list/2014-November/680758.html
>
> That should probably be put in the PEP, even if it is not an option
> being considered, it at least evidence that "some people" find the
> behaviour of generators more natural than that of comprehensions.
>
>
> --
> Steve
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141117/897e6224/attachment-0001.html>

From guido at python.org  Tue Nov 18 05:50:14 2014
From: guido at python.org (Guido van Rossum)
Date: Mon, 17 Nov 2014 20:50:14 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
 <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
Message-ID: <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>

Nick,

I think we've gone through enough clarifications of the PEP now to be clear
on the proposal. I saw in one of your earliest replies (right after Chris
posted his first draft) that you're hesitant to support the PEP because of
what would have to change to contextlib. What I couldn't quite read is
whether you think that the proposal by itself is not an improvement, or
whether you're just worried about compatibility.

Apparently you know of a large group of users who use an older 3rd party
version of contextlib, and for whom that older, 3rd party contextlib should
keep working with future versions of Python 3 without updating their
version of contextlib -- did I get that right? What exactly is the
constraint there that makes their version of contextlib immutable even
though the version of Python they are using may move forward?

Separate from this special case, I am also worried about backward
compatibility, and I have yet to get a good idea for how widespread code is
that depends on StopIteration bubbling out from generators. I also don't
have a good idea how often this issue bites users, but I have a feeling it
does bite. E.g. this quote from c.l.py (
https://mail.python.org/pipermail/python-list/2014-November/680775.html):

"""
I did find it annoying occasionally that raising StopIteration inside a
generator expression conveys a different behavior than elsewhere. It did
take me quite a while to understand why that is so, but after that it
did not cause me much of a headache anymore.
"""

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141117/63ccfade/attachment.html>

From wolfgang.maier at biologie.uni-freiburg.de  Tue Nov 18 18:40:30 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Tue, 18 Nov 2014 18:40:30 +0100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
 <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
Message-ID: <546B848E.6040406@biologie.uni-freiburg.de>

On 11/18/2014 05:50 AM, Guido van Rossum wrote:
>
> Separate from this special case, I am also worried about backward
> compatibility, and I have yet to get a good idea for how widespread code
> is that depends on StopIteration bubbling out from generators. I also
> don't have a good idea how often this issue bites users, but I have a
> feeling it does bite. E.g. this quote from c.l.py <http://c.l.py>
> (https://mail.python.org/pipermail/python-list/2014-November/680775.html):
>
> """
> I did find it annoying occasionally that raising StopIteration inside a
> generator expression conveys a different behavior than elsewhere. It did
> take me quite a while to understand why that is so, but after that it
> did not cause me much of a headache anymore.
> """
>

I just remembered one use of the current behavior. Two years ago or so, 
I was suggesting on this list a possibility for early termination of 
comprehensions when a particular value is encountered. In other words, 
an equivalent to:

l = []
for x in seq:
     if x == y:
         break
     l.append(x)

At the time, somebody suggested (roughly):

def stop():
     raise StopIteration

l = list(x for x in seq if x!=y or stop())

which, for the very reasons discussed in this thread, works only as a 
generator expression and not in comprehension form.

I used this solution in some not particularly important piece of code so 
I wouldn't despair if it wasn't compatible with the next release of the 
language.
Also, I have a feeling that some of you may consider this sort of a hack 
in the first place.
Just thought I'd mention it here for completeness.

Wolfgang



From ethan at stoneleaf.us  Tue Nov 18 20:09:43 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 18 Nov 2014 11:09:43 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
 generators
In-Reply-To: <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
 <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
Message-ID: <546B9977.9020903@stoneleaf.us>

On 11/17/2014 08:50 PM, Guido van Rossum wrote:
> 
> Separate from this special case, I am also worried about backward
> compatibility, and I have yet to get a good idea for how widespread code is
> that depends on StopIteration bubbling out from generators. I also don't
> have a good idea how often this issue bites users, but I have a feeling it
> does bite. 

One argument for making the change*:  When we're writing __next__, or __getattr__, etc., it is obvious that we are
playing with internals and have to be extra careful of what other exceptions might be raised in that code.
Contrariwise, the only indication of something special about a generator is the presence of the yield keyword -- for
ordinary use (such as in for loops) it doesn't matter whether the called function returns a list, tuple, iterator,
generator, or whatever, as long as it can be iterated over, and so when writing a generator, or converting an
iterable-returning function into a generator, there's nothing obvious saying, "Hey!  Watch out for a StopIteration
somewhere else in this block of code!"

* I make no statement as to how strong this argument is, but there you have it.  :)

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141118/0a674754/attachment.sig>

From guido at python.org  Tue Nov 18 21:36:35 2014
From: guido at python.org (Guido van Rossum)
Date: Tue, 18 Nov 2014 12:36:35 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <546B9977.9020903@stoneleaf.us>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
 <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <546B9977.9020903@stoneleaf.us>
Message-ID: <CAP7+vJLbivfd7D==-B3AjGawdVopOFpmyKa5uxCYSgNM-riYLw@mail.gmail.com>

That's pretty much the reason I gave this my attention at all. :-)

On Tue, Nov 18, 2014 at 11:09 AM, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 11/17/2014 08:50 PM, Guido van Rossum wrote:
> >
> > Separate from this special case, I am also worried about backward
> > compatibility, and I have yet to get a good idea for how widespread code
> is
> > that depends on StopIteration bubbling out from generators. I also don't
> > have a good idea how often this issue bites users, but I have a feeling
> it
> > does bite.
>
> One argument for making the change*:  When we're writing __next__, or
> __getattr__, etc., it is obvious that we are
> playing with internals and have to be extra careful of what other
> exceptions might be raised in that code.
> Contrariwise, the only indication of something special about a generator
> is the presence of the yield keyword -- for
> ordinary use (such as in for loops) it doesn't matter whether the called
> function returns a list, tuple, iterator,
> generator, or whatever, as long as it can be iterated over, and so when
> writing a generator, or converting an
> iterable-returning function into a generator, there's nothing obvious
> saying, "Hey!  Watch out for a StopIteration
> somewhere else in this block of code!"
>
> * I make no statement as to how strong this argument is, but there you
> have it.  :)
>
> --
> ~Ethan~
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141118/0a096e47/attachment.html>

From tjreedy at udel.edu  Tue Nov 18 23:54:57 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 18 Nov 2014 17:54:57 -0500
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <546B848E.6040406@biologie.uni-freiburg.de>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
 <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <546B848E.6040406@biologie.uni-freiburg.de>
Message-ID: <m4giog$gko$1@ger.gmane.org>

On 11/18/2014 12:40 PM, Wolfgang Maier wrote:

> I just remembered one use of the current behavior. Two years ago or so,
> I was suggesting on this list a possibility for early termination of
> comprehensions when a particular value is encountered. In other words,
> an equivalent to:
>
> l = []
> for x in seq:
>      if x == y:
>          break
>      l.append(x)

I believe I thought then that one should write the explicit loop rather 
than overload the 'comprehension' concept.

> At the time, somebody suggested (roughly):
>
> def stop():
>      raise StopIteration
>
> l = list(x for x in seq if x!=y or stop())

If stop is defined in another file, such as 'utility', this is a bit 
nasty.  A new maintainer comes along and changes that to a list 
comprehension, or perhaps decides a set rather than a list is needed, 
and changes it to a set comprehension instead of set() call and bingo!, 
a bug.  Or someone imitates the pattern, but with [] instead of list.

> which, for the very reasons discussed in this thread, works only as a
> generator expression and not in comprehension form.

With this example, where the StopIteration source could be much more 
obscure than next(), I now understand Guido's concern about 
hard-to-understand bugs.  From a maintainability view, it should not 
matter if one calls a function on a naked comprehension (making it a 
genexp) or uses the

> I used this solution in some not particularly important piece of code so
> I wouldn't despair if it wasn't compatible with the next release of the
> language.
> Also, I have a feeling that some of you may consider this sort of a hack
> in the first place.


-- 
Terry Jan Reedy


From rosuav at gmail.com  Wed Nov 19 00:08:29 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 19 Nov 2014 10:08:29 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <m4giog$gko$1@ger.gmane.org>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
 <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <546B848E.6040406@biologie.uni-freiburg.de>
 <m4giog$gko$1@ger.gmane.org>
Message-ID: <CAPTjJmq3-iMsKqRfVCsW0PtAHqLcRD3ECe6Es2G1jr+zW1syUQ@mail.gmail.com>

On Wed, Nov 19, 2014 at 9:54 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 11/18/2014 12:40 PM, Wolfgang Maier wrote:
>
>> I just remembered one use of the current behavior. Two years ago or so,
>> I was suggesting on this list a possibility for early termination of
>> comprehensions when a particular value is encountered. In other words,
>> an equivalent to:
>>
>> l = []
>> for x in seq:
>>      if x == y:
>>          break
>>      l.append(x)
>
>
> I believe I thought then that one should write the explicit loop rather than
> overload the 'comprehension' concept.

I'm not sure about that. Comprehensions can already be filtered; is it
such a jump from there to a "filter" that aborts on a certain
condition? It may not be language-supported, but I don't see that it's
illogical; and any use of a loop that appends to a list is rightly
considered code smell.

ChrisA

From steve at pearwood.info  Wed Nov 19 02:15:48 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 19 Nov 2014 12:15:48 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmq3-iMsKqRfVCsW0PtAHqLcRD3ECe6Es2G1jr+zW1syUQ@mail.gmail.com>
References: <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <546B848E.6040406@biologie.uni-freiburg.de> <m4giog$gko$1@ger.gmane.org>
 <CAPTjJmq3-iMsKqRfVCsW0PtAHqLcRD3ECe6Es2G1jr+zW1syUQ@mail.gmail.com>
Message-ID: <20141119011548.GZ2748@ando.pearwood.info>

On Wed, Nov 19, 2014 at 10:08:29AM +1100, Chris Angelico wrote:
> On Wed, Nov 19, 2014 at 9:54 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> > On 11/18/2014 12:40 PM, Wolfgang Maier wrote:
> >
> >> I just remembered one use of the current behavior. Two years ago or so,
> >> I was suggesting on this list a possibility for early termination of
> >> comprehensions when a particular value is encountered. In other words,
> >> an equivalent to:
> >>
> >> l = []
> >> for x in seq:
> >>      if x == y:
> >>          break
> >>      l.append(x)
> >
> >
> > I believe I thought then that one should write the explicit loop rather than
> > overload the 'comprehension' concept.
> 
> I'm not sure about that. Comprehensions can already be filtered; is it
> such a jump from there to a "filter" that aborts on a certain
> condition? It may not be language-supported, but I don't see that it's
> illogical;

It certainly isn't. It's an obvious extension to the concept: terminate 
the loop rather than filter it. At least two languages support early 
termination:

http://clojuredocs.org/clojure_core/clojure.core/for
http://docs.racket-lang.org/guide/for.html


and it keeps getting asked for:

http://www.reddit.com/r/Python/comments/ciec3/is_there_anything_like_a_list_comprehension/
http://stackoverflow.com/questions/5505891/using-while-in-list-comprehension-or-generator-expressions
http://stackoverflow.com/questions/16931214/short-circuiting-list-comprehensions
https://www.daniweb.com/software-development/python/threads/293381/break-a-list-comprehension
https://mail.python.org/pipermail/python-ideas/2014-February/026036.html
https://mail.python.org/pipermail/python-ideas/2013-January/018969.html

There's a rejected PEP:

https://www.python.org/dev/peps/pep-3142/

and alternative solutions (write an explicit generator function, use 
itertools.takewhile). So there's obviously a need for this sort of 
thing, and (expr for x in iterable if cond() or stop()) seems to be a 
common solution.

I'm not sure if that's a neat trick or a deplorable hack :-) but either 
way this PEP will break code using it.

> and any use of a loop that appends to a list is rightly
> considered code smell.

I'm afraid I don't understand that comment. Why is appending to a list 
inside a loop a code smell? That's exactly what list comps do.



-- 
Steven

From rosuav at gmail.com  Wed Nov 19 03:01:17 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 19 Nov 2014 13:01:17 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <20141119011548.GZ2748@ando.pearwood.info>
References: <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <546B848E.6040406@biologie.uni-freiburg.de>
 <m4giog$gko$1@ger.gmane.org>
 <CAPTjJmq3-iMsKqRfVCsW0PtAHqLcRD3ECe6Es2G1jr+zW1syUQ@mail.gmail.com>
 <20141119011548.GZ2748@ando.pearwood.info>
Message-ID: <CAPTjJmok5ahEx=mi07JDM9bLfC3rxzhWhoxf4ah3yhEbiJWrHQ@mail.gmail.com>

On Wed, Nov 19, 2014 at 12:15 PM, Steven D'Aprano <steve at pearwood.info> wrote:
>> and any use of a loop that appends to a list is rightly
>> considered code smell.
>
> I'm afraid I don't understand that comment. Why is appending to a list
> inside a loop a code smell? That's exactly what list comps do.

That's precisely why. If I write code like this:

l = []
for i in something:
    l.append(func(i))

then I should rework it into a comprehension. Having a filter doesn't
change that:

l = []
for i in something:
    if i:
        l.append(func(i))

That's still possible with a list comp, and should be rewritten as
one. But having a break in there *does* change it, because there's no
way in the language to do that. The question is: Is it better to abuse
StopIteration or to turn the list comp back into an explicit loop? And
if anyone chose the former, their code will break.

ChrisA

From guido at python.org  Wed Nov 19 03:56:42 2014
From: guido at python.org (Guido van Rossum)
Date: Tue, 18 Nov 2014 18:56:42 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmok5ahEx=mi07JDM9bLfC3rxzhWhoxf4ah3yhEbiJWrHQ@mail.gmail.com>
References: <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <546B848E.6040406@biologie.uni-freiburg.de> <m4giog$gko$1@ger.gmane.org>
 <CAPTjJmq3-iMsKqRfVCsW0PtAHqLcRD3ECe6Es2G1jr+zW1syUQ@mail.gmail.com>
 <20141119011548.GZ2748@ando.pearwood.info>
 <CAPTjJmok5ahEx=mi07JDM9bLfC3rxzhWhoxf4ah3yhEbiJWrHQ@mail.gmail.com>
Message-ID: <CAP7+vJ+_3Dj6iSuFdErUDha21Mb3VWgqK31neSKU+23sVOx1iA@mail.gmail.com>

On Tue, Nov 18, 2014 at 6:01 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Wed, Nov 19, 2014 at 12:15 PM, Steven D'Aprano <steve at pearwood.info>
> wrote:
> >> and any use of a loop that appends to a list is rightly
> >> considered code smell.
> >
> > I'm afraid I don't understand that comment. Why is appending to a list
> > inside a loop a code smell? That's exactly what list comps do.
>
> That's precisely why. If I write code like this:
>
> l = []
> for i in something:
>     l.append(func(i))
>
> then I should rework it into a comprehension. Having a filter doesn't
> change that:
>
> l = []
> for i in something:
>     if i:
>         l.append(func(i))
>
> That's still possible with a list comp, and should be rewritten as
> one. But having a break in there *does* change it, because there's no
> way in the language to do that. The question is: Is it better to abuse
> StopIteration or to turn the list comp back into an explicit loop? And
> if anyone chose the former, their code will break.
>

Not everything you do with an explicit loop can be done with a
comprehension, and that's by design. Comprehensions should be easier to
reason about than code using for-loops. And generator expressions should
work the same way, except for producing results in a lazy fashion.

The StopIteration hack breaks this equivalence and hampers the ability to
reason, since you can't tell whether a predicate might raise StopIteration.

It was never my intention that generator expressions behaved this way -- it
was an accidental feature that surprised me when it was first shown to me,
and I've never gotten used to it. (And I don't care whether you say it is
"obvious", call it "stop()", and only use it in an "idiomatic" fashion --
it's still a surprise for anyone who has to debug code involving it.)

The only thing standing in the way of fixing this is the recognition that
there may be a fair amount of code out there that depends on this hack, and
which will have to be rewritten.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141118/1bc0128e/attachment.html>

From rosuav at gmail.com  Wed Nov 19 04:37:28 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 19 Nov 2014 14:37:28 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAP7+vJ+_3Dj6iSuFdErUDha21Mb3VWgqK31neSKU+23sVOx1iA@mail.gmail.com>
References: <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <546B848E.6040406@biologie.uni-freiburg.de>
 <m4giog$gko$1@ger.gmane.org>
 <CAPTjJmq3-iMsKqRfVCsW0PtAHqLcRD3ECe6Es2G1jr+zW1syUQ@mail.gmail.com>
 <20141119011548.GZ2748@ando.pearwood.info>
 <CAPTjJmok5ahEx=mi07JDM9bLfC3rxzhWhoxf4ah3yhEbiJWrHQ@mail.gmail.com>
 <CAP7+vJ+_3Dj6iSuFdErUDha21Mb3VWgqK31neSKU+23sVOx1iA@mail.gmail.com>
Message-ID: <CAPTjJmpALYNHcXGc606wi94vVWKhRjStbDesiS_AQmXmbCH3ow@mail.gmail.com>

On Wed, Nov 19, 2014 at 1:56 PM, Guido van Rossum <guido at python.org> wrote:
> The only thing standing in the way of fixing this is the recognition that
> there may be a fair amount of code out there that depends on this hack, and
> which will have to be rewritten.

Has anyone come across any more non-trivial examples? We have
contextlib (in the standard library) and contextlib2 (third-party),
plus a number of StackOverflow posts and such. Are there any other
known pieces of code that would be seriously hampered by this change?

ChrisA

From guido at python.org  Wed Nov 19 05:22:41 2014
From: guido at python.org (Guido van Rossum)
Date: Tue, 18 Nov 2014 20:22:41 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmpALYNHcXGc606wi94vVWKhRjStbDesiS_AQmXmbCH3ow@mail.gmail.com>
References: <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <546B848E.6040406@biologie.uni-freiburg.de> <m4giog$gko$1@ger.gmane.org>
 <CAPTjJmq3-iMsKqRfVCsW0PtAHqLcRD3ECe6Es2G1jr+zW1syUQ@mail.gmail.com>
 <20141119011548.GZ2748@ando.pearwood.info>
 <CAPTjJmok5ahEx=mi07JDM9bLfC3rxzhWhoxf4ah3yhEbiJWrHQ@mail.gmail.com>
 <CAP7+vJ+_3Dj6iSuFdErUDha21Mb3VWgqK31neSKU+23sVOx1iA@mail.gmail.com>
 <CAPTjJmpALYNHcXGc606wi94vVWKhRjStbDesiS_AQmXmbCH3ow@mail.gmail.com>
Message-ID: <CAP7+vJJcLL63ix8ZOLhrx4sm5exE4790TAGwkvysN4uwQkjsVQ@mail.gmail.com>

On Tue, Nov 18, 2014 at 7:37 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Wed, Nov 19, 2014 at 1:56 PM, Guido van Rossum <guido at python.org>
> wrote:
> > The only thing standing in the way of fixing this is the recognition that
> > there may be a fair amount of code out there that depends on this hack,
> and
> > which will have to be rewritten.
>
> Has anyone come across any more non-trivial examples? We have
> contextlib (in the standard library) and contextlib2 (third-party),
> plus a number of StackOverflow posts and such. Are there any other
> known pieces of code that would be seriously hampered by this change?
>

One possible way to find out would be to write a simple version of a patch
(maybe one that doesn't use __future__ but just always converts
StopIteration to RuntimeError when it bubbling out of a generator frame)
and run the stdlib tests, then see how many tests this breaks. (I
understand if you don't want to write it. But maybe someone does. :-)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141118/83261a11/attachment-0001.html>

From storchaka at gmail.com  Wed Nov 19 17:01:29 2014
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 19 Nov 2014 18:01:29 +0200
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
 <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
Message-ID: <m4iev3$7cl$1@ger.gmane.org>

On 18.11.14 06:50, Guido van Rossum wrote:
> Separate from this special case, I am also worried about backward
> compatibility, and I have yet to get a good idea for how widespread code
> is that depends on StopIteration bubbling out from generators. I also
> don't have a good idea how often this issue bites users, but I have a
> feeling it does bite.

E.g. current difflib implementation depends on this behavior.

     ...
     while True:
         ...
         while(found_diff is False):
             from_line, to_line, found_diff = next(line_pair_iterator)
             ...



From ncoghlan at gmail.com  Wed Nov 19 17:03:27 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 20 Nov 2014 02:03:27 +1000
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
 <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
Message-ID: <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>

On 18 November 2014 14:50, Guido van Rossum <guido at python.org> wrote:

> Nick,
>
> I think we've gone through enough clarifications of the PEP now to be
> clear on the proposal. I saw in one of your earliest replies (right after
> Chris posted his first draft) that you're hesitant to support the PEP
> because of what would have to change to contextlib. What I couldn't quite
> read is whether you think that the proposal by itself is not an
> improvement, or whether you're just worried about compatibility.
>

I think it's an improvement - I really like the fact that it brings
generators into line with your reasoning in the with statement PEP that
flow control constructs should be locally visible. At the moment, "raise
StopIteration" in a generator context is effectively non-local flow
control, as it means any function call (explicit or implicit) or yield
point may gracefully stop generator execution, rather than only return
statements.

>From the point of view of a generator *consumer*, I'm now happy that the
backwards incompatibility is limited to cases where you throw in
StopIteration and want to check if you got the same StopIteration instance
back - you'll now also need to check for RuntimeError with __cause__ set to
the StopIteration instance you threw in. You can construct scenarios where
such a check will give a false positive, but they're getting seriously
contrived at that point.

That's obscure enough that I think it's on par with other behavioural
tweaks we've included in the "Porting to Python X.Y" guides in the past.


> Apparently you know of a large group of users who use an older 3rd party
> version of contextlib, and for whom that older, 3rd party contextlib should
> keep working with future versions of Python 3 without updating their
> version of contextlib -- did I get that right? What exactly is the
> constraint there that makes their version of contextlib immutable even
> though the version of Python they are using may move forward?
>

I don't even remember how that came up now, but it was entirely
hypothetical, and I think it can be ignored as a concern. As you say, being
able to update to Python 3.5 without also being able to update to a new
version of contextlib2 would just be weird.

Even if such a strange scenario did somehow come up, it would still be
possible to switch to a conditional import where they used the stdlib
version if available, and only fell back to contextlib2 on earlier versions
of Python.


> Separate from this special case, I am also worried about backward
> compatibility, and I have yet to get a good idea for how widespread code is
> that depends on StopIteration bubbling out from generators. I also don't
> have a good idea how often this issue bites users, but I have a feeling it
> does bite. E.g. this quote from c.l.py (
> https://mail.python.org/pipermail/python-list/2014-November/680775.html):
>
> """
> I did find it annoying occasionally that raising StopIteration inside a
> generator expression conveys a different behavior than elsewhere. It did
> take me quite a while to understand why that is so, but after that it
> did not cause me much of a headache anymore.
> """
>

One advantage of needing a __future__ import on the generator author side
is that you always have the option of changing your mind, and *never*
making the new behaviour the default.

That wouldn't be a *good* outcome, but I don't think it would be
intolerable.

OTOH, I'm also not sure the status quo is sufficiently problematic to be
worth changing. Yes, it's a little weird, but is it *that* much weirder
than the unavoidable issues with exceptions thrown in __next__,
__getitem__, __getattr__ and other special methods where a particular type
of exception is handled directly by the interpreter?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141120/472b8ddf/attachment.html>

From rosuav at gmail.com  Wed Nov 19 17:24:07 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 20 Nov 2014 03:24:07 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
 <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>
Message-ID: <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>

On Thu, Nov 20, 2014 at 3:03 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> OTOH, I'm also not sure the status quo is sufficiently problematic to be
> worth changing. Yes, it's a little weird, but is it *that* much weirder than
> the unavoidable issues with exceptions thrown in __next__, __getitem__,
> __getattr__ and other special methods where a particular type of exception
> is handled directly by the interpreter?

If you write __next__, you write in a "raise StopIteration" when it's
done. If you write __getattr__, you write in "raise AttributeError" if
the attribute shouldn't exist. Those are sufficiently explicit that it
should be reasonably clear that the exception is the key. But when you
write a generator, you don't explicitly raise:

def gen():
    yield 1
    yield 2
    yield 3
    return 4

The distinction in __next__ is between returning something and raising
something. The distinction in a generator is between "yield" and
"return". Why should a generator author have to be concerned about one
particular exception having magical meaning?

Imagine this scenario:

def producer():
    """Return user input, or raise KeyboardInterrupt"""
    return input("Enter the next string: ")

def consumer():
    """Process the user's input"""
    while True:
        try:
            command = producer()
        except KeyboardInterrupt:
            break
        dispatch(command)


Okay, now let's make a mock producer:

strings = ["do stuff","do more stuff","blah blah"]
def mock_producer()
    if strings: return strings.pop(0)
    raise KeyboardInterrupt

That's how __next__ works, only with a different exception, and I
think people would agree that this is NOT a good use of
KeyboardInterrupt. If you put a few extra layers in between the
producer and consumer, you'd be extremely surprised that an unexpected
KeyboardInterrupt just quietly terminated a loop. Yet this is exactly
what the generator-and-for-loop model creates: a situation in which
StopIteration, despite not being seen at either end of the code, now
has magical properties. Without the generator, *only* __next__ has
this effect, and that's exactly where it's documented to be.

Does that make for more justification? Unexpected exceptions bubbling
up is better than unexpected exceptions quietly terminating loops?

ChrisA

From ncoghlan at gmail.com  Wed Nov 19 17:45:27 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 20 Nov 2014 02:45:27 +1000
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
 <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>
 <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
Message-ID: <CADiSq7c1bQePno=bfVqP3JtWieJXm19==FrS4onuHeSQYZYk8g@mail.gmail.com>

On 20 November 2014 02:24, Chris Angelico <rosuav at gmail.com> wrote:

> Does that make for more justification? Unexpected exceptions bubbling
> up is better than unexpected exceptions quietly terminating loops?
>

The part I found most compelling was when you pointed out that in the
special method implementations, the normal return path was always spelled
with "return", while the "value missing" result was indicated with a
special kind of exception (StopIteration, AttributeError, IndexError or
KeyError), and then any other exception was consider unexpected.

Generators add the third notion of being able to suspend execution via
"yield", which then left them with two different ways of spelling
termination inside the frame: "return" OR "raise StopIteration". The second
spelling ("raise StopIteration") is then inherently surprising, as it's
entirely redundant, *except* in that it allows you to effectively have a
"hidden return" in a generator frame that can't be done anywhere else.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141120/8e2221e2/attachment-0001.html>

From tjreedy at udel.edu  Wed Nov 19 23:46:39 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 19 Nov 2014 17:46:39 -0500
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
 <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>
 <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
Message-ID: <m4j6l1$nts$1@ger.gmane.org>

On 11/19/2014 11:24 AM, Chris Angelico wrote:
> On Thu, Nov 20, 2014 at 3:03 AM, Nick Coghlan <ncoghlan-Re5JQEeQqe8AvxtiuMwx3w at public.gmane.org> wrote:
>> OTOH, I'm also not sure the status quo is sufficiently problematic to be
>> worth changing. Yes, it's a little weird, but is it *that* much weirder than
>> the unavoidable issues with exceptions thrown in __next__, __getitem__,
>> __getattr__ and other special methods where a particular type of exception
>> is handled directly by the interpreter?
>
> If you write __next__, you write in a "raise StopIteration" when it's
> done. If you write __getattr__, you write in "raise AttributeError" if
> the attribute shouldn't exist. Those are sufficiently explicit that it
> should be reasonably clear that the exception is the key. But when you
> write a generator, you don't explicitly raise:
>
> def gen():
>      yield 1
>      yield 2
>      yield 3
>      return 4
>
> The distinction in __next__ is between returning something and raising
> something. The distinction in a generator is between "yield" and
> "return".

Which, as I said a week ago, is why there is no need for "raise 
StopIteration" in a generator function.  The doc clearly states the 
limited intended use of StopIteration.
'''
exception StopIteration
     Raised by built-in function next() and an iterator?s __next__() 
method to signal that there are no further items produced by the iterator.
'''
StopIteration is exposed so it can be raised in user coded __next__() 
and caught when using explicit next().  If it was only used for builtins 
and for loops, it would not need to be visible.

> Why should a generator author have to be concerned about one
> particular exception having magical meaning?

I am not sure of your intent with this rhetorical (?) question.

> Imagine this scenario:
>
> def producer():
>      """Return user input, or raise KeyboardInterrupt"""
>      return input("Enter the next string: ")

The prompt should be "Enter the next string or hit ^C to quit: ".

> def consumer():
>      """Process the user's input"""
>      while True:
>          try:
>              command = producer()
>          except KeyboardInterrupt:
>              break
>          dispatch(command)

> Okay, now let's make a mock producer:
>
> strings = ["do stuff","do more stuff","blah blah"]
> def mock_producer()
>      if strings: return strings.pop(0)
>      raise KeyboardInterrupt

> That's how __next__ works, only with a different exception, and I
> think people would agree that this is NOT a good use of
> KeyboardInterrupt.

It is avoidable because the return type of producer is limited to 
strings.  Therefore, producer could (and perhaps should) itself catch 
KeyboardInterrupt and return None, which is intended for such use. 
Consumer would then be simplified by replacing 3 lines with "if command 
is None: break".

 > If you put a few extra layers in between the
> producer and consumer, you'd be extremely surprised that an unexpected
> KeyboardInterrupt just quietly terminated a loop. Yet this is exactly
> what the generator-and-for-loop model creates: a situation in which
> StopIteration, despite not being seen at either end of the code, now
> has magical properties. Without the generator, *only* __next__ has
> this effect, and that's exactly where it's documented to be.
>
> Does that make for more justification? Unexpected exceptions bubbling
> up is better than unexpected exceptions quietly terminating loops?


-- 
Terry Jan Reedy



From rosuav at gmail.com  Wed Nov 19 23:57:43 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 20 Nov 2014 09:57:43 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <m4j6l1$nts$1@ger.gmane.org>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
 <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>
 <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
 <m4j6l1$nts$1@ger.gmane.org>
Message-ID: <CAPTjJmoiF0HB++tphkvYef1JN8o+T-Xet9jOCbykxCsTXJD0Jw@mail.gmail.com>

On Thu, Nov 20, 2014 at 9:46 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 11/19/2014 11:24 AM, Chris Angelico wrote:
>> The distinction in __next__ is between returning something and raising
>> something. The distinction in a generator is between "yield" and
>> "return".
>
>
> Which, as I said a week ago, is why there is no need for "raise
> StopIteration" in a generator function.  The doc clearly states the limited
> intended use of StopIteration.
> '''
> exception StopIteration
>     Raised by built-in function next() and an iterator?s __next__() method
> to signal that there are no further items produced by the iterator.
> '''
> StopIteration is exposed so it can be raised in user coded __next__() and
> caught when using explicit next().  If it was only used for builtins and for
> loops, it would not need to be visible.
>
>> Why should a generator author have to be concerned about one
>> particular exception having magical meaning?
>
> I am not sure of your intent with this rhetorical (?) question.

Yes, rhetorical. Basically saying the same as you are: that
StopIteration is a part of __next__, not generators.

>> Imagine this scenario:
>>
>> def producer():
>>      """Return user input, or raise KeyboardInterrupt"""
>>      return input("Enter the next string: ")
>
> The prompt should be "Enter the next string or hit ^C to quit: ".

Yeah, the point is about its interaction with the rest of the program,
not the human.

>> def consumer():
>>      """Process the user's input"""
>>      while True:
>>          try:
>>              command = producer()
>>          except KeyboardInterrupt:
>>              break
>>          dispatch(command)
>
>
>> Okay, now let's make a mock producer:
>>
>> strings = ["do stuff","do more stuff","blah blah"]
>> def mock_producer()
>>      if strings: return strings.pop(0)
>>      raise KeyboardInterrupt
>
>
>> That's how __next__ works, only with a different exception, and I
>> think people would agree that this is NOT a good use of
>> KeyboardInterrupt.
>
>
> It is avoidable because the return type of producer is limited to strings.
> Therefore, producer could (and perhaps should) itself catch
> KeyboardInterrupt and return None, which is intended for such use. Consumer
> would then be simplified by replacing 3 lines with "if command is None:
> break".

Sure it does. But suppose it does some parsing on the string first,
and that parsing might return literally any object. The structure of
the program is the same, but now it really does need to signal "no
more stuff" in some way other than return value.

Just trying to concoct a situation similar to generators/for loops,
using a different exception. I'm fairly sure there's no way to make
the above system seem truly plausible, because KeyboardInterrupt is a
bad exception for the purpose; but it's still broadly similar, and I
think the same applies: StopException should be *only* inside
__next__() and next(). Since generators can distinguish yield from
return, they don't need to distinguish return from raise.

ChrisA

From rosuav at gmail.com  Wed Nov 19 23:58:43 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 20 Nov 2014 09:58:43 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmoiF0HB++tphkvYef1JN8o+T-Xet9jOCbykxCsTXJD0Jw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <CABicbJJFsu3U-+r4EBkF9PTpOHZw0g=sfcp=ZZ=2wO-5-VJctQ@mail.gmail.com>
 <CAPTjJmqwUuk1-+7JW4GEFSCWk1m44xrd9Ojr7NpkCY60gJTYPA@mail.gmail.com>
 <CAP7+vJKn5P1jzfMq+GHSBRza_63L2AocDOftNE2bCZFSWV0=EA@mail.gmail.com>
 <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>
 <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
 <m4j6l1$nts$1@ger.gmane.org>
 <CAPTjJmoiF0HB++tphkvYef1JN8o+T-Xet9jOCbykxCsTXJD0Jw@mail.gmail.com>
Message-ID: <CAPTjJmoq_OLSuwx0NA0kSG37kXafTjnyTDSYZ6DHVijrU33+XQ@mail.gmail.com>

On Thu, Nov 20, 2014 at 9:57 AM, Chris Angelico <rosuav at gmail.com> wrote:
> Since generators can distinguish yield from
> return, they don't need to distinguish return from raise.

Bad grammar, should edit before posting. Since generators can
distinguish value from no value by using yield and return, they don't
need to use yield and raise.

ChrisA

From steve at pearwood.info  Thu Nov 20 01:25:47 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 20 Nov 2014 11:25:47 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CADiSq7c1bQePno=bfVqP3JtWieJXm19==FrS4onuHeSQYZYk8g@mail.gmail.com>
References: <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>
 <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
 <CADiSq7c1bQePno=bfVqP3JtWieJXm19==FrS4onuHeSQYZYk8g@mail.gmail.com>
Message-ID: <20141120002546.GH2748@ando.pearwood.info>

On Thu, Nov 20, 2014 at 02:45:27AM +1000, Nick Coghlan wrote:

> The part I found most compelling was when you pointed out that in the
> special method implementations, the normal return path was always spelled
> with "return", while the "value missing" result was indicated with a
> special kind of exception (StopIteration, AttributeError, IndexError or
> KeyError), and then any other exception was consider unexpected.
> 
> Generators add the third notion of being able to suspend execution via
> "yield", which then left them with two different ways of spelling
> termination inside the frame: "return" OR "raise StopIteration". The second
> spelling ("raise StopIteration") is then inherently surprising, as it's
> entirely redundant, *except* in that it allows you to effectively have a
> "hidden return" in a generator frame that can't be done anywhere else.

I'm not sure that many people outside of this and the python-dev mailing 
lists would find the use of "raise StopIteration" surprising. Rather, I 
expect that they will find the use of an explicit "return" inside a 
generator surprising. People are informally taught that generators use 
yield *instead of* return, so seeing both in the same function is a 
surprise. (Most generators quitely fall out the bottom with no explicit 
end.)


I don't claim that doing so is Pythonic or even good practice, but I am 
sure that there are a lot of people who believe that raising 
StopIteration to exit a generator is (1) supported and (2) preferred.

Examples of code in the wild using StopIteration to exit:


http://code.openhub.net/file?fid=ezlejSoT2q7PWrhgNkpdU55MWOA&cid=jVcYOxnQhvU&s=raise%20StopIteration&fp=301369&mp&projSelected=true#L0

http://code.openhub.net/file?fid=M0gWWCpn-avqHO_jnsYcG2T81lg&cid=VKn_M0_GgKM&s=raise%20StopIteration&fp=301283&mp&projSelected=true#L0

http://code.openhub.net/file?fid=pDrrTI8lyh0LO_6rTCk9npC96SE&cid=Y8jg8v1AyqU&s=raise%20StopIteration&fp=41191&mp&projSelected=true#L0

http://code.openhub.net/file?fid=PTjGrE_5rOhyZhL1CUrPBtRk7n8&cid=tWtPpAs4E1g&s=raise%20StopIteration&fp=210789&mp&projSelected=true#L0

http://code.openhub.net/file?fid=WzkucGktJhjsP8cj4BO6Wcnbx-0&cid=fsj7E8vdVMA&s=raise%20StopIteration&fp=401086&mp&projSelected=true#L0

http://stackoverflow.com/questions/6784934/python-yield-and-stopiteration-in-one-loop

http://stackoverflow.com/questions/14183803/in-pythons-generators-what-is-the-difference-between-raise-stopiteration-and

That last example not only uses raise to exit the generator, but the 
author actually guesses that it is the more Pythonic way to do so.

Here is a description of the generator protocol which could easily lead 
the reader to conclude that raising StopIteration is the correct way to 
exit a generator:

    To support this protocol, functions with yield statement 
    are compiled specially as generators. They return a generator 
    object when they are called. The returned object supports the 
    iteration interface with an automatically created __next__() 
    method to resume execution. Generator functions may have a 
    return simply terminates the generation of values by raising 
    a StopIteration exceptions after any normal function exit. 

http://www.bogotobogo.com/python/python_generators.php


At this point, I'm convinced that there is a good argument for a 
__future__ import changing this behaviour. But I suspect that making 
this the default behaviour in the future will break a lot of code.



-- 
Steven

From rosuav at gmail.com  Thu Nov 20 02:34:08 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 20 Nov 2014 12:34:08 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <20141120002546.GH2748@ando.pearwood.info>
References: <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>
 <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
 <CADiSq7c1bQePno=bfVqP3JtWieJXm19==FrS4onuHeSQYZYk8g@mail.gmail.com>
 <20141120002546.GH2748@ando.pearwood.info>
Message-ID: <CAPTjJmp36yWuFWW2MVr2itkGoo-m=0HAV2D8gO8vw-E4fq6jgw@mail.gmail.com>

On Thu, Nov 20, 2014 at 11:25 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> I'm not sure that many people outside of this and the python-dev mailing
> lists would find the use of "raise StopIteration" surprising. Rather, I
> expect that they will find the use of an explicit "return" inside a
> generator surprising. People are informally taught that generators use
> yield *instead of* return, so seeing both in the same function is a
> surprise. (Most generators quitely fall out the bottom with no explicit
> end.)

Interesting. But "yield instead of return" doesn't automatically say
"and then raise StopIteration to early-abort"; I'd say the informal
description is fine, it just needs to be modified differently once
people actually want an early abort. ("You can still use 'return' for
its other purpose, terminating a function before reaching the end.")

> Examples of code in the wild using StopIteration to exit:
>
> http://code.openhub.net/file?fid=PTjGrE_5rOhyZhL1CUrPBtRk7n8&cid=tWtPpAs4E1g&s=raise%20StopIteration&fp=210789&mp&projSelected=true#L0

Trivially unnecessary, and as soon as there's a bug report, the
"What's New In 3.7" page will explain that it needs to be removed.

> http://code.openhub.net/file?fid=WzkucGktJhjsP8cj4BO6Wcnbx-0&cid=fsj7E8vdVMA&s=raise%20StopIteration&fp=401086&mp&projSelected=true#L0

I have no idea what this one is doing, but it looks like it's half way
to what's wanted here. Catch the exception and deal with it... this
proposal just means the "deal with it" part needs to be reworded into
a return statement.

All it needs is for "What's New in 3.5" to recommend use of 'return'
instead of 'raise StopIteration', and all these cases will be easily
compatible with all [1] versions of Python.

> http://stackoverflow.com/questions/6784934/python-yield-and-stopiteration-in-one-loop

The accepted answer correctly advises the function be written to
simply return. This will work fine. The other answer has a local
"raise StopIteration", which can be translated into a simple "return".

> http://stackoverflow.com/questions/14183803/in-pythons-generators-what-is-the-difference-between-raise-stopiteration-and
>
> That last example not only uses raise to exit the generator, but the
> author actually guesses that it is the more Pythonic way to do so.

The question's author does, but the accepted answer recommends "return".

This may result in the odd question here or there, but it's not a
major problem. Any time a generator has "raise StopIteration" in its
own body, it can simply become "return". That's easy. The issue comes
up when it's not raising that itself, but is letting it bubble up -
maybe from a next() call.

def takewhiletrue(iter):
    while True: # coincidental with the function name
        # try:
        val = next(iter)
        # except StopIteration: return
        if not val: break
        yield val

This won't come up in a simple search for "raise StopIteration", and
if you have something like this where the condition is almost always
going to be reached eventually, you might not notice the problem for a
long time. How would you know to add the commented-out lines? What
kind of code search would you use to detect this?

> Here is a description of the generator protocol which could easily lead
> the reader to conclude that raising StopIteration is the correct way to
> exit a generator:
>
>     To support this protocol, functions with yield statement
>     are compiled specially as generators. They return a generator
>     object when they are called. The returned object supports the
>     iteration interface with an automatically created __next__()
>     method to resume execution. Generator functions may have a
>     return simply terminates the generation of values by raising
>     a StopIteration exceptions after any normal function exit.
>
> http://www.bogotobogo.com/python/python_generators.php

Even that does recommend 'return'. If anyone reads that, writes "raise
StopIteration", sees code bombing with RuntimeError, and then comes to
python-list, we can explain that the recommended method is "return". I
have no problem with this. There are plenty of much-worse practices
that people pick up - mixing bytes and text, using backslashes in
Windows path names without doubling them or using raw literals, etc,
etc, etc. In lots of cases they'll seem to work ("C:\Program Files\New
Stuff\Testing" will work, until you lower-case the name), but when
they break, you just have to fix them. This wouldn't be the first
Python minor version to tighten up requirements to remove bug magnets.

> At this point, I'm convinced that there is a good argument for a
> __future__ import changing this behaviour. But I suspect that making
> this the default behaviour in the future will break a lot of code.

I suspect that a huge percentage of the code so broken can be
trivially fixed just by search/replacing "raise StopIteration" with
"return". There'll be only a very few where the StopIteration is
raised from some other function and needs to be caught and dealt with
- and fixing those is just as likely to reveal bugs needing fixing.

ChrisA

From steve at pearwood.info  Thu Nov 20 03:06:15 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 20 Nov 2014 13:06:15 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
References: <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>
 <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
Message-ID: <20141120020615.GI2748@ando.pearwood.info>

On Thu, Nov 20, 2014 at 03:24:07AM +1100, Chris Angelico wrote:

> If you write __next__, you write in a "raise StopIteration" when it's
> done. If you write __getattr__, you write in "raise AttributeError" if
> the attribute shouldn't exist. Those are sufficiently explicit that it
> should be reasonably clear that the exception is the key. But when you
> write a generator, you don't explicitly raise:

That's not true in practice. See my reply to Nick, there is lots of code 
out there which uses StopIteration to exit generators. Some of that code 
isn't very good code -- I've seen "raise StopIteration" immediately 
before falling out the bottom of the generator -- but until now it has 
worked and the impression some people have gotten is that it is actually 
preferred.


> def gen():
>     yield 1
>     yield 2
>     yield 3
>     return 4

Until 3.2, that was a syntax error. For the majority of people who are 
still using Python 2.7, it is *still* a syntax error. To write this in a 
backwards-compatible way, you have to exit the generator with:

    raise StopIteration(2)


> The distinction in __next__ is between returning something and raising
> something. The distinction in a generator is between "yield" and
> "return". Why should a generator author have to be concerned about one
> particular exception having magical meaning?

I would put it another way: informally, the distinction between a 
generator and a function is that generators use yield where functions 
use return. Most people are happy with that informal definition, a full 
pedantic explanation of co-routines will just confuse them or bore them. 
The rule they will learn is:

* use return in functions
* use yield in generators

That makes generators that use both surprising. Since most generators 
either run forever or fall out the bottom when they are done, I expect 
that seeing a generator with a return in it is likely to surprise a lot 
of people. I've known that return works for many years, and I still 
give a double-take whenever I see it in a generator.


> Imagine this scenario:
> 
> def producer():
>     """Return user input, or raise KeyboardInterrupt"""
>     return input("Enter the next string: ")
> 
> def consumer():
>     """Process the user's input"""
>     while True:
>         try:
>             command = producer()
>         except KeyboardInterrupt:
>             break
>         dispatch(command)
> 
> 
> Okay, now let's make a mock producer:
> 
> strings = ["do stuff","do more stuff","blah blah"]
> def mock_producer()
>     if strings: return strings.pop(0)
>     raise KeyboardInterrupt
> 
> That's how __next__ works, only with a different exception, and I
> think people would agree that this is NOT a good use of
> KeyboardInterrupt.

Why not? How else are you going to communicate something out of band to 
the consumer except via an exception? 

We can argue about whether KeyboardInterrupt is the right exception to 
use or not, but if you insist that this is a bad protocol then you're 
implicitly saying that the iterator protocol is also a bad protocol.


> If you put a few extra layers in between the
> producer and consumer, you'd be extremely surprised that an unexpected
> KeyboardInterrupt just quietly terminated a loop.

You might be, but since I've paid attention to the protocol rules, I 
won't be. Sorry to be harsh, but how clear do we have to be? 
StopIteration terminates iterators, and generators are iterators. That 
rule may or may not be inconvenient, it might be annoying (but sometimes 
useful), it might hide bugs, it might even be something that we can 
easily forget until reminded, but if it comes as a "surprise" that just means 
you don't know how the iterator protocol works.

There are good reasons for changing this behaviour, but pandering to 
people who don't know how the iterator protocol works is not one of 
them.


> Yet this is exactly
> what the generator-and-for-loop model creates: a situation in which
> StopIteration, despite not being seen at either end of the code, now
> has magical properties.

That's exactly how the protocol works. Even if you write "return" in 
your generator, it still raises StopIteration.


> Without the generator, *only* __next__ has
> this effect, and that's exactly where it's documented to be.

The documentation states that __next__ raises StopIteration, it doesn't 
say that *only* __next__ should raise StopIteration.

https://docs.python.org/3/library/stdtypes.html#iterator.__next__

I trust that we all expect to be able to factor out the raise into a 
helper function or method, yes? It truly would be surprising if this 
failed:


class MyIterator:
    def __iter__(self):
        return self
    def __next__(self):
        return something()


def something():
    # Toy helper function.
    if random.random() < 0.5:
        return "Spam!"
    raise StopIteration



Now let's write this as a generator:

def gen():
    while True:
        yield something()


which is much nicer than:

def gen():
    while True:
        try:
            yield something()
        except StopIteration:
            return   # converted by Python into raise StopIteration



-- 
Steven

From rosuav at gmail.com  Thu Nov 20 03:24:02 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 20 Nov 2014 13:24:02 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <20141120020615.GI2748@ando.pearwood.info>
References: <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>
 <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
 <20141120020615.GI2748@ando.pearwood.info>
Message-ID: <CAPTjJmoM0my6HLta_7Kdey51vj7BUR2YPd2BMQO_DcPaRfUn-w@mail.gmail.com>

On Thu, Nov 20, 2014 at 1:06 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Thu, Nov 20, 2014 at 03:24:07AM +1100, Chris Angelico wrote:
>
>> If you write __next__, you write in a "raise StopIteration" when it's
>> done. If you write __getattr__, you write in "raise AttributeError" if
>> the attribute shouldn't exist. Those are sufficiently explicit that it
>> should be reasonably clear that the exception is the key. But when you
>> write a generator, you don't explicitly raise:
>
> That's not true in practice. See my reply to Nick, there is lots of code
> out there which uses StopIteration to exit generators. Some of that code
> isn't very good code -- I've seen "raise StopIteration" immediately
> before falling out the bottom of the generator -- but until now it has
> worked and the impression some people have gotten is that it is actually
> preferred.

Yes, I thought it was rare. I stand corrected. Reword that to "you
don't *need to* explicitly raise", since you can simply return, and it
becomes true again, though.

>> def gen():
>>     yield 1
>>     yield 2
>>     yield 3
>>     return 4
>
> Until 3.2, that was a syntax error. For the majority of people who are
> still using Python 2.7, it is *still* a syntax error. To write this in a
> backwards-compatible way, you have to exit the generator with:
>
>     raise StopIteration(2)

In most cases you won't need to put a value on it, so bare "return"
will work just fine. I just put a return value onto it so it wouldn't
look trivially useless.

>> The distinction in __next__ is between returning something and raising
>> something. The distinction in a generator is between "yield" and
>> "return". Why should a generator author have to be concerned about one
>> particular exception having magical meaning?
>
> I would put it another way: informally, the distinction between a
> generator and a function is that generators use yield where functions
> use return. Most people are happy with that informal definition, a full
> pedantic explanation of co-routines will just confuse them or bore them.
> The rule they will learn is:
>
> * use return in functions
> * use yield in generators
>
> That makes generators that use both surprising. Since most generators
> either run forever or fall out the bottom when they are done, I expect
> that seeing a generator with a return in it is likely to surprise a lot
> of people. I've known that return works for many years, and I still
> give a double-take whenever I see it in a generator.

But it's just as surprising to put "raise StopIteration" into it. It's
normal to put that into __next__, it's not normal to need it in a
generator. Either way, it's something unusual; so let's go with the
unusual "return" rather than the unusual "raise".

>> That's how __next__ works, only with a different exception, and I
>> think people would agree that this is NOT a good use of
>> KeyboardInterrupt.
>
> Why not? How else are you going to communicate something out of band to
> the consumer except via an exception?
>
> We can argue about whether KeyboardInterrupt is the right exception to
> use or not, but if you insist that this is a bad protocol then you're
> implicitly saying that the iterator protocol is also a bad protocol.

Well, that's exactly what I do mean. KeyboardInterrupt is not a good
way for two parts of a program to communicate with each other, largely
because it can be raised unexpectedly. Which is the point of this PEP:
raising StopIteration unexpectedly should also result in a noisy
traceback.

> I trust that we all expect to be able to factor out the raise into a
> helper function or method, yes? It truly would be surprising if this
> failed:
>
>
> class MyIterator:
>     def __iter__(self):
>         return self
>     def __next__(self):
>         return something()
>
>
> def something():
>     # Toy helper function.
>     if random.random() < 0.5:
>         return "Spam!"
>     raise StopIteration
>
>
>
> Now let's write this as a generator:
>
> def gen():
>     while True:
>         yield something()
>
>
> which is much nicer than:
>
> def gen():
>     while True:
>         try:
>             yield something()
>         except StopIteration:
>             return   # converted by Python into raise StopIteration

Sure. There was a suggestion that "return yield from something()"
would work, though, which - I can't confirm that this works, but
assuming it does - would be a lot tidier. But there's still a
difference. Your first helper function was specifically a __next__
helper. It was tied intrinsically to the iterator protocol. If you
want to call a __next__ helper (or actually call next(iter) on
something) inside a generator, you'll have to - if this change goes
through - cope with the fact that generator protocol says "return"
where __next__ protocol says "raise StopIteration". If you want a
generator helper, it'd look like this:

def something():
     # Toy helper function.
     if random.random() < 0.5:
         yield "Spam!"

def gen():
    yield from something()

Voila! Now it's a generator helper, following generator protocol.
Every bit as tidy as the original. Let's write a __getitem__ helper:

def something(x):
     # Toy helper function.
     if random.random() < 0.5:
         return "Spam!"
    raise KeyError(x)

class X:
    def __getitem__(self, x): return something(x)

Same thing. As soon as you get into raising these kinds of exceptions,
you're tying your helper to a specific protocol. All that's happening
with PEP 479 is that generator and iterator protocol are being
distinguished slightly.

ChrisA

From rosuav at gmail.com  Thu Nov 20 04:44:25 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 20 Nov 2014 14:44:25 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAP7+vJJcLL63ix8ZOLhrx4sm5exE4790TAGwkvysN4uwQkjsVQ@mail.gmail.com>
References: <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <546B848E.6040406@biologie.uni-freiburg.de>
 <m4giog$gko$1@ger.gmane.org>
 <CAPTjJmq3-iMsKqRfVCsW0PtAHqLcRD3ECe6Es2G1jr+zW1syUQ@mail.gmail.com>
 <20141119011548.GZ2748@ando.pearwood.info>
 <CAPTjJmok5ahEx=mi07JDM9bLfC3rxzhWhoxf4ah3yhEbiJWrHQ@mail.gmail.com>
 <CAP7+vJ+_3Dj6iSuFdErUDha21Mb3VWgqK31neSKU+23sVOx1iA@mail.gmail.com>
 <CAPTjJmpALYNHcXGc606wi94vVWKhRjStbDesiS_AQmXmbCH3ow@mail.gmail.com>
 <CAP7+vJJcLL63ix8ZOLhrx4sm5exE4790TAGwkvysN4uwQkjsVQ@mail.gmail.com>
Message-ID: <CAPTjJmqJn8ho=W1vD91zWjK9QMjgFcrbnO=EyiWTJx8btEX2Ag@mail.gmail.com>

On Wed, Nov 19, 2014 at 3:22 PM, Guido van Rossum <guido at python.org> wrote:
> One possible way to find out would be to write a simple version of a patch
> (maybe one that doesn't use __future__ but just always converts
> StopIteration to RuntimeError when it bubbling out of a generator frame) and
> run the stdlib tests, then see how many tests this breaks. (I understand if
> you don't want to write it. But maybe someone does. :-)

I poked around a bit in the code and managed to come up with this. It
doesn't chain the previous exception, so the traceback is a little
scanty, but it does turn a StopIteration into a RuntimeError. (It
might also leak the original StopIteration. I'm not sure.) Prior to
this patch, I had 377 of 390 tests passing flawlessly and no failures
(just skips and warnings); with this applied, six failures.

diff -r 23ab1197df0b Objects/genobject.c
--- a/Objects/genobject.c Wed Nov 19 13:21:40 2014 +0200
+++ b/Objects/genobject.c Thu Nov 20 13:43:44 2014 +1100
@@ -130,6 +130,14 @@
         }
         Py_CLEAR(result);
     }
+    else if (!result)
+    {
+        if (PyErr_ExceptionMatches(PyExc_StopIteration))
+        {
+            PyErr_SetString(PyExc_RuntimeError,
+                "generator raised StopIteration");
+        }
+    }

     if (!result || f->f_stacktop == NULL) {
         /* generator can't be rerun, so release the frame */

However, I'm not sure about setting the context. In errors.c is a
function _PyErr_ChainExceptions which appears to do a similar job, so
I imitated its code. Here's the result:

    else if (!result)
    {
        if (PyErr_ExceptionMatches(PyExc_StopIteration))
        {
            PyObject *exc, *val, *val2, *tb;
            PyErr_Fetch(&exc, &val, &tb);
            PyErr_NormalizeException(&exc, &val, &tb);
            Py_DECREF(exc);
            Py_XDECREF(tb);
            PyErr_SetString(PyExc_RuntimeError,
                "generator raised StopIteration");
            PyErr_Fetch(&exc, &val2, &tb);
            PyErr_NormalizeException(&exc, &val2, &tb);
            PyException_SetContext(val2, val);
            PyErr_Restore(exc, val2, tb);
        }
    }

The context is being set, but without a traceback.

#############

def good_gen():
    yield 1
    return 2

def evil_gen():
    yield 1
    raise StopIteration(2)

# In absence of PEP 479 changes, the above two should be virtually
indistinguishable.

print("Starting.")
good = tuple(good_gen())
print("Good:", good, good == (1,))
evil = tuple(evil_gen())
print("Evil:", evil, evil == (1,))

#############

Starting.
Good: (1,) True
StopIteration: 2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "../test_pep0479.py", line 14, in <module>
    evil = tuple(evil_gen())
RuntimeError: generator raised StopIteration



What am I missing here? Do I need to force something to construct a
full traceback before it can show the line number that actually raised
StopIteration?

ChrisA

From ethan at stoneleaf.us  Thu Nov 20 05:19:41 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 19 Nov 2014 20:19:41 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
 generators
In-Reply-To: <20141120002546.GH2748@ando.pearwood.info>
References: <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>
 <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
 <CADiSq7c1bQePno=bfVqP3JtWieJXm19==FrS4onuHeSQYZYk8g@mail.gmail.com>
 <20141120002546.GH2748@ando.pearwood.info>
Message-ID: <546D6BDD.4040007@stoneleaf.us>

On 11/19/2014 04:25 PM, Steven D'Aprano wrote:
> 
> Here is a description of the generator protocol which could easily lead 
> the reader to conclude that raising StopIteration is the correct way to 
> exit a generator:
> 
>     To support this protocol, functions with yield statement 
>     are compiled specially as generators. They return a generator 
>     object when they are called. The returned object supports the 
>     iteration interface with an automatically created __next__() 
>     method to resume execution. Generator functions may have a 
>     return simply terminates the generation of values by raising 
>     a StopIteration exceptions after any normal function exit. 
> 
> http://www.bogotobogo.com/python/python_generators.php

We are not, however, responsible for third-party documentation.


> At this point, I'm convinced that there is a good argument for a 
> __future__ import changing this behaviour. But I suspect that making 
> this the default behaviour in the future will break a lot of code.

Isn't that the case with every __future__ directive that becomes the standard?  Folks have an entire minor release to
make the adjustment.

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141119/9e55654d/attachment.sig>

From steve at pearwood.info  Thu Nov 20 06:30:51 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 20 Nov 2014 16:30:51 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmp36yWuFWW2MVr2itkGoo-m=0HAV2D8gO8vw-E4fq6jgw@mail.gmail.com>
References: <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>
 <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
 <CADiSq7c1bQePno=bfVqP3JtWieJXm19==FrS4onuHeSQYZYk8g@mail.gmail.com>
 <20141120002546.GH2748@ando.pearwood.info>
 <CAPTjJmp36yWuFWW2MVr2itkGoo-m=0HAV2D8gO8vw-E4fq6jgw@mail.gmail.com>
Message-ID: <20141120053050.GA20200@ando.pearwood.info>

On Thu, Nov 20, 2014 at 12:34:08PM +1100, Chris Angelico wrote:
> On Thu, Nov 20, 2014 at 11:25 AM, Steven D'Aprano <steve at pearwood.info> wrote:

> > Examples of code in the wild using StopIteration to exit:
> >
> > http://code.openhub.net/file?fid=PTjGrE_5rOhyZhL1CUrPBtRk7n8&cid=tWtPpAs4E1g&s=raise%20StopIteration&fp=210789&mp&projSelected=true#L0
> 
> Trivially unnecessary, and as soon as there's a bug report, the
> "What's New In 3.7" page will explain that it needs to be removed.

The point isn't that it is easy to fix. I'm sure that there will be 
cases of code that are not easy to fix. The point is that we're breaking 
working code and causing code churn.

We're not fixing a bug. We're changing behaviour people rely on. That 
ought to make us more conservative about breaking their code.


> This may result in the odd question here or there, but it's not a
> major problem. 

And neither is the existing behaviour. We're weighing up whether the 
small benefit in fixing this wart is worth the pain. The PEP isn't 
approved yet, and right from the beginning Guido said that he feared 
that fixing this might be too disruptive. I'm trying to get a feel for 
how disruptive it will be.

I did a quick and informal survey of the developers I work with. The 
dreaded lurgy has gone through our office, so most of them are away ill, 
but of those still here (all two of them) one of them couldn't remember 
whether you exit a generator with "yield nil" or "return nil" (he's a 
Ruby and Objective-C guy when we're not paying him to write Python) and 
the other one said that the whole problem is that generators exist in 
the first place, Python should get rid of them and allow people to 
define their own using macros (he likes to think of himself as a Lisp 
and Scheme guru :-)

Make of that what you will.



-- 
Steven

From rosuav at gmail.com  Thu Nov 20 06:38:01 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 20 Nov 2014 16:38:01 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <20141120053050.GA20200@ando.pearwood.info>
References: <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>
 <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
 <CADiSq7c1bQePno=bfVqP3JtWieJXm19==FrS4onuHeSQYZYk8g@mail.gmail.com>
 <20141120002546.GH2748@ando.pearwood.info>
 <CAPTjJmp36yWuFWW2MVr2itkGoo-m=0HAV2D8gO8vw-E4fq6jgw@mail.gmail.com>
 <20141120053050.GA20200@ando.pearwood.info>
Message-ID: <CAPTjJmrpZCT87_tbjPoqxB7Ym=AsLxF3zjAj9QWiDRjTGTi5bg@mail.gmail.com>

On Thu, Nov 20, 2014 at 4:30 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> the other one said that the whole problem is that generators exist in
> the first place, Python should get rid of them and allow people to
> define their own using macros (he likes to think of himself as a Lisp
> and Scheme guru :-)

There's a language that lets you define anything you like. It's called
"file on disk". If you don't like how it runs, you just put a little
shebang at the top and the whole rest of the file is interpreted
differently...

On one side of the balance is code breakage. On the other side is
currently broken code where bugs will be found. Which is the stronger
argument? I'm inclined toward the latter, but neither has a huge body
of code to back it.

ChrisA

From steve at pearwood.info  Thu Nov 20 06:50:03 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 20 Nov 2014 16:50:03 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <546D6BDD.4040007@stoneleaf.us>
References: <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>
 <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
 <CADiSq7c1bQePno=bfVqP3JtWieJXm19==FrS4onuHeSQYZYk8g@mail.gmail.com>
 <20141120002546.GH2748@ando.pearwood.info> <546D6BDD.4040007@stoneleaf.us>
Message-ID: <20141120055003.GB20200@ando.pearwood.info>

On Wed, Nov 19, 2014 at 08:19:41PM -0800, Ethan Furman wrote:

> We are not, however, responsible for third-party documentation.

No, of course not, but we should be aware that:

* some people believe that raising StopIteration is an acceptable 
  way to exit a generator; and

* doing so has worked fine since generators were introduced back
  in Python 2.2.

I wonder whether people who learned about generators back in the 2.2 
days will have stronger opinions about raising StopIteration than more 
recent users? I remember learning that an explicit raise was the way to 
exit a generator, and sure enough the 2.2 What's New says this:

  Inside a generator function, the return statement can only be used 
  without a value, and signals the end of the procession of values; 
  afterwards the generator cannot return any further values. return with 
  a value, such as return 5, is a syntax error inside a generator 
  function. The end of the generator?s results can also be indicated by 
  raising StopIteration manually, or by just letting the flow of 
  execution fall off the bottom of the function.

https://docs.python.org/3/whatsnew/2.2.html#pep-255-simple-generators

That's not to say that we can't change the behaviour, but neither can we 
say it is undocumented or blame third parties.


> > At this point, I'm convinced that there is a good argument for a 
> > __future__ import changing this behaviour. But I suspect that making 
> > this the default behaviour in the future will break a lot of code.
> 
> Isn't that the case with every __future__ directive that becomes the 
> standard?  Folks have an entire minor release to make the adjustment.

Huh, you say that like it's a long time :-)




-- 
Steven

From rosuav at gmail.com  Thu Nov 20 06:54:20 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 20 Nov 2014 16:54:20 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmqJn8ho=W1vD91zWjK9QMjgFcrbnO=EyiWTJx8btEX2Ag@mail.gmail.com>
References: <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <546B848E.6040406@biologie.uni-freiburg.de>
 <m4giog$gko$1@ger.gmane.org>
 <CAPTjJmq3-iMsKqRfVCsW0PtAHqLcRD3ECe6Es2G1jr+zW1syUQ@mail.gmail.com>
 <20141119011548.GZ2748@ando.pearwood.info>
 <CAPTjJmok5ahEx=mi07JDM9bLfC3rxzhWhoxf4ah3yhEbiJWrHQ@mail.gmail.com>
 <CAP7+vJ+_3Dj6iSuFdErUDha21Mb3VWgqK31neSKU+23sVOx1iA@mail.gmail.com>
 <CAPTjJmpALYNHcXGc606wi94vVWKhRjStbDesiS_AQmXmbCH3ow@mail.gmail.com>
 <CAP7+vJJcLL63ix8ZOLhrx4sm5exE4790TAGwkvysN4uwQkjsVQ@mail.gmail.com>
 <CAPTjJmqJn8ho=W1vD91zWjK9QMjgFcrbnO=EyiWTJx8btEX2Ag@mail.gmail.com>
Message-ID: <CAPTjJmowewoTnfnnAy4d3BzpqFoqr4T=qhyOMLiTAdrzX2rL2w@mail.gmail.com>

On Thu, Nov 20, 2014 at 2:44 PM, Chris Angelico <rosuav at gmail.com> wrote:
>
> I poked around a bit in the code and managed to come up with this. It
> doesn't chain the previous exception, so the traceback is a little
> scanty, but it does turn a StopIteration into a RuntimeError. (It
> might also leak the original StopIteration. I'm not sure.) Prior to
> this patch, I had 377 of 390 tests passing flawlessly and no failures
> (just skips and warnings); with this applied, six failures.
>

With the attached demo patch, all tests pass except test_generators,
which explicitly tests stuff about the correlation between return and
StopIteration. There's the contextlib changes, a couple of places that
were raising StopIteration and should be returning, and a couple that
were letting StopIteration bubble and now need to catch it and return.
I've deliberately not followed PEP 8 here, in the interests of
minimizing diff size; in several cases, blocks of code ought to be
indented a level, but I cheated and created a half-indentation to show
how little actually changes.

If anyone would care to try this on their own codebases, that'd be helpful.

ChrisA
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pep0479.patch
Type: text/x-patch
Size: 4590 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141120/9549ff50/attachment.bin>

From ncoghlan at gmail.com  Thu Nov 20 11:01:40 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 20 Nov 2014 20:01:40 +1000
Subject: [Python-ideas] Return expressions
Message-ID: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>

This idea occurred to me in context of PEP 479 and the "or stop()" hack for
generator expressions. I'm not a big enough fan of the idea to pursue it
myself, but if anyone is bothered by the prospect of PEP 479 taking the "or
stop()" technique away, this may be worth pursuing further.

As a reminder of how the hack works, the following generator:

>>> def takewhile(iterable, pred):
...     for x in iter(iterable):
...         if not pred(x):
...             return
...         yield x
...
>>> list(takewhile(range(10), (lambda x: x < 5)))
[0, 1, 2, 3, 4]

Can currently be converted to a generator expression with the aid of a
helper function:

>>> def stop():
...     raise StopIteration
...
>>> list(x for x in range(10) if x < 5 or stop())
[0, 1, 2, 3, 4]

Under PEP 479, that will raise RuntimeError instead. If return was an
expression rather than a statement (ala the yield statement -> expression
conversion in PEP 342), then the following would be functionally equivalent
to the current "or stop()" trick:

>>> list(x for x in range(10) if x < 5 or return)
[0, 1, 2, 3, 4]

This is emphatically *not* executable pseudocode - it only makes sense in
terms of the translation to the corresponding full generator function
definition. However, the translation remains exact in terms of the normal
semantics of the "return" keyword, unlike various other proposals to use
the "while" keyword to provide comparable functionality.

For comprehensions, the meaning of a bare return would likely need to be
tweaked slightly to be the same as reaching the end of the frame: returning
the object being constructed by the comprehension, rather than None.

>>> [x for x in range(10) if x < 5 or return]
[0, 1, 2, 3, 4]
>>> {x for x in range(10) if x < 5 or return}
{0, 1, 2, 3, 4}
>>> {x:x for x in range(10) if x < 5 or return}
{0:0, 1:1, 2:2, 3:3, 4:4}

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141120/c7b22350/attachment.html>

From ncoghlan at gmail.com  Thu Nov 20 11:13:52 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 20 Nov 2014 20:13:52 +1000
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmowewoTnfnnAy4d3BzpqFoqr4T=qhyOMLiTAdrzX2rL2w@mail.gmail.com>
References: <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <546B848E.6040406@biologie.uni-freiburg.de>
 <m4giog$gko$1@ger.gmane.org>
 <CAPTjJmq3-iMsKqRfVCsW0PtAHqLcRD3ECe6Es2G1jr+zW1syUQ@mail.gmail.com>
 <20141119011548.GZ2748@ando.pearwood.info>
 <CAPTjJmok5ahEx=mi07JDM9bLfC3rxzhWhoxf4ah3yhEbiJWrHQ@mail.gmail.com>
 <CAP7+vJ+_3Dj6iSuFdErUDha21Mb3VWgqK31neSKU+23sVOx1iA@mail.gmail.com>
 <CAPTjJmpALYNHcXGc606wi94vVWKhRjStbDesiS_AQmXmbCH3ow@mail.gmail.com>
 <CAP7+vJJcLL63ix8ZOLhrx4sm5exE4790TAGwkvysN4uwQkjsVQ@mail.gmail.com>
 <CAPTjJmqJn8ho=W1vD91zWjK9QMjgFcrbnO=EyiWTJx8btEX2Ag@mail.gmail.com>
 <CAPTjJmowewoTnfnnAy4d3BzpqFoqr4T=qhyOMLiTAdrzX2rL2w@mail.gmail.com>
Message-ID: <CADiSq7fOeHkx0tqwZ-=HWZV4b75khJ+OqokdoBnHiHLNXy0mcA@mail.gmail.com>

On 20 November 2014 15:54, Chris Angelico <rosuav at gmail.com> wrote:

> On Thu, Nov 20, 2014 at 2:44 PM, Chris Angelico <rosuav at gmail.com> wrote:
> >
> > I poked around a bit in the code and managed to come up with this. It
> > doesn't chain the previous exception, so the traceback is a little
> > scanty, but it does turn a StopIteration into a RuntimeError. (It
> > might also leak the original StopIteration. I'm not sure.) Prior to
> > this patch, I had 377 of 390 tests passing flawlessly and no failures
> > (just skips and warnings); with this applied, six failures.
> >
>
> With the attached demo patch, all tests pass except test_generators,
> which explicitly tests stuff about the correlation between return and
> StopIteration. There's the contextlib changes, a couple of places that
> were raising StopIteration and should be returning, and a couple that
> were letting StopIteration bubble and now need to catch it and return.
> I've deliberately not followed PEP 8 here, in the interests of
> minimizing diff size; in several cases, blocks of code ought to be
> indented a level, but I cheated and created a half-indentation to show
> how little actually changes.
>

Thanks Chris - could you make a tracker issue and attach this there?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141120/f2d0aafa/attachment-0001.html>

From rosuav at gmail.com  Thu Nov 20 11:43:54 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 20 Nov 2014 21:43:54 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CADiSq7fOeHkx0tqwZ-=HWZV4b75khJ+OqokdoBnHiHLNXy0mcA@mail.gmail.com>
References: <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <546B848E.6040406@biologie.uni-freiburg.de>
 <m4giog$gko$1@ger.gmane.org>
 <CAPTjJmq3-iMsKqRfVCsW0PtAHqLcRD3ECe6Es2G1jr+zW1syUQ@mail.gmail.com>
 <20141119011548.GZ2748@ando.pearwood.info>
 <CAPTjJmok5ahEx=mi07JDM9bLfC3rxzhWhoxf4ah3yhEbiJWrHQ@mail.gmail.com>
 <CAP7+vJ+_3Dj6iSuFdErUDha21Mb3VWgqK31neSKU+23sVOx1iA@mail.gmail.com>
 <CAPTjJmpALYNHcXGc606wi94vVWKhRjStbDesiS_AQmXmbCH3ow@mail.gmail.com>
 <CAP7+vJJcLL63ix8ZOLhrx4sm5exE4790TAGwkvysN4uwQkjsVQ@mail.gmail.com>
 <CAPTjJmqJn8ho=W1vD91zWjK9QMjgFcrbnO=EyiWTJx8btEX2Ag@mail.gmail.com>
 <CAPTjJmowewoTnfnnAy4d3BzpqFoqr4T=qhyOMLiTAdrzX2rL2w@mail.gmail.com>
 <CADiSq7fOeHkx0tqwZ-=HWZV4b75khJ+OqokdoBnHiHLNXy0mcA@mail.gmail.com>
Message-ID: <CAPTjJmrv27TyKwH3Cdqv4CcuQ-2k6JD7sk9nhPPNDGOb8FrJcA@mail.gmail.com>

On Thu, Nov 20, 2014 at 9:13 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 20 November 2014 15:54, Chris Angelico <rosuav at gmail.com> wrote:
>>
>> On Thu, Nov 20, 2014 at 2:44 PM, Chris Angelico <rosuav at gmail.com> wrote:
>> >
>> > I poked around a bit in the code and managed to come up with this. It
>> > doesn't chain the previous exception, so the traceback is a little
>> > scanty, but it does turn a StopIteration into a RuntimeError. (It
>> > might also leak the original StopIteration. I'm not sure.) Prior to
>> > this patch, I had 377 of 390 tests passing flawlessly and no failures
>> > (just skips and warnings); with this applied, six failures.
>> >
>>
>> With the attached demo patch, all tests pass except test_generators,
>> which explicitly tests stuff about the correlation between return and
>> StopIteration. There's the contextlib changes, a couple of places that
>> were raising StopIteration and should be returning, and a couple that
>> were letting StopIteration bubble and now need to catch it and return.
>> I've deliberately not followed PEP 8 here, in the interests of
>> minimizing diff size; in several cases, blocks of code ought to be
>> indented a level, but I cheated and created a half-indentation to show
>> how little actually changes.
>
>
> Thanks Chris - could you make a tracker issue and attach this there?

http://bugs.python.org/issue22906

ChrisA

From python at mrabarnett.plus.com  Thu Nov 20 13:54:36 2014
From: python at mrabarnett.plus.com (MRAB)
Date: Thu, 20 Nov 2014 12:54:36 +0000
Subject: [Python-ideas] Return expressions
In-Reply-To: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
Message-ID: <546DE48C.5000805@mrabarnett.plus.com>

On 2014-11-20 10:01, Nick Coghlan wrote:
> This idea occurred to me in context of PEP 479 and the "or stop()" hack
> for generator expressions. I'm not a big enough fan of the idea to
> pursue it myself, but if anyone is bothered by the prospect of PEP 479
> taking the "or stop()" technique away, this may be worth pursuing further.
>
> As a reminder of how the hack works, the following generator:
>
>  >>> def takewhile(iterable, pred):
> ...     for x in iter(iterable):
> ...         if not pred(x):
> ...             return
> ...         yield x
> ...
>  >>> list(takewhile(range(10), (lambda x: x < 5)))
> [0, 1, 2, 3, 4]
>
> Can currently be converted to a generator expression with the aid of a
> helper function:
>
>  >>> def stop():
> ...     raise StopIteration
> ...
>  >>> list(x for x in range(10) if x < 5 or stop())
> [0, 1, 2, 3, 4]
>
> Under PEP 479, that will raise RuntimeError instead. If return was an
> expression rather than a statement (ala the yield statement ->
> expression conversion in PEP 342), then the following would be
> functionally equivalent to the current "or stop()" trick:
>
>  >>> list(x for x in range(10) if x < 5 or return)
> [0, 1, 2, 3, 4]
>
> This is emphatically *not* executable pseudocode - it only makes sense
> in terms of the translation to the corresponding full generator function
> definition. However, the translation remains exact in terms of the
> normal semantics of the "return" keyword, unlike various other proposals
> to use the "while" keyword to provide comparable functionality.
>
> For comprehensions, the meaning of a bare return would likely need to be
> tweaked slightly to be the same as reaching the end of the frame:
> returning the object being constructed by the comprehension, rather than
> None.
>
>  >>> [x for x in range(10) if x < 5 or return]
> [0, 1, 2, 3, 4]
>  >>> {x for x in range(10) if x < 5 or return}
> {0, 1, 2, 3, 4}
>  >>> {x:x for x in range(10) if x < 5 or return}
> {0:0, 1:1, 2:2, 3:3, 4:4}
>
I'd prefer 'while' instead:

 >>> [x for x in range(10) while x < 5]
[0, 1, 2, 3, 4]
 >>> {x for x in range(10) while x < 5}
{0, 1, 2, 3, 4}
 >>> {x:x for x in range(10) while x < 5}
{0:0, 1:1, 2:2, 3:3, 4:4}


From njs at pobox.com  Thu Nov 20 14:02:23 2014
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 20 Nov 2014 13:02:23 +0000
Subject: [Python-ideas] Return expressions
In-Reply-To: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
Message-ID: <CAPJVwB=jGDPGS3uEhmEhAKyJmMSptJEonSe3L7EAkQ6F+eAv+g@mail.gmail.com>

On Thu, Nov 20, 2014 at 10:01 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >>> list(x for x in range(10) if x < 5 or return)
>  [0, 1, 2, 3, 4]

I'd be concerned about the confusion engendered by 'return' having
totally different semantics within very small regions, e.g. we'd have

def takewhile(iterable, pred):
    return (x for x in iterable if pred(x) or return)

Esp. because normally 'return' is very powerful -- right now any
mention of that token anywhere in a function body jumps out of the
entire function, so you really have to scan for them actively when
trying to understand a function's flow control.

Not a big fan of a flow control expression in general, really; yield
expressions are kinda flow-controlly, but they just pause execution,
they don't disrupt the logical through-line of a block of code.

-n

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org

From ncoghlan at gmail.com  Thu Nov 20 14:36:26 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 20 Nov 2014 23:36:26 +1000
Subject: [Python-ideas] Return expressions
In-Reply-To: <546DE48C.5000805@mrabarnett.plus.com>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
 <546DE48C.5000805@mrabarnett.plus.com>
Message-ID: <CADiSq7cWybgFx7YtP+xYin3wLLX3kh3gjQxghUh7jqc=LVs8dQ@mail.gmail.com>

On 20 November 2014 22:54, MRAB <python at mrabarnett.plus.com> wrote:

> I'd prefer 'while' instead:
>
> >>> [x for x in range(10) while x < 5]
> [0, 1, 2, 3, 4]
> >>> {x for x in range(10) while x < 5}
> {0, 1, 2, 3, 4}
> >>> {x:x for x in range(10) while x < 5}
> {0:0, 1:1, 2:2, 3:3, 4:4}
>

That's been suggested many times, and always fails on the grounds of being
completely incompatible with the expansions of comprehensions and generator
expressions as nested compound statements.

The return expression idea is one that reflects the actual deep structure
of the nested scopes that underlie the syntactic sugar, so it's compatible
with the expansion.

This idea was mainly aimed at folks that might be inclined to worry about
the possible loss of the "or stop()" generator expression trick in PEP 479,
though. If that trick wasn't currently possible, and wasn't being proposed
for removal, I never would have posted this idea.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141120/a5af1969/attachment.html>

From encukou at gmail.com  Thu Nov 20 15:43:04 2014
From: encukou at gmail.com (Petr Viktorin)
Date: Thu, 20 Nov 2014 15:43:04 +0100
Subject: [Python-ideas] Return expressions
In-Reply-To: <CADiSq7cWybgFx7YtP+xYin3wLLX3kh3gjQxghUh7jqc=LVs8dQ@mail.gmail.com>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
 <546DE48C.5000805@mrabarnett.plus.com>
 <CADiSq7cWybgFx7YtP+xYin3wLLX3kh3gjQxghUh7jqc=LVs8dQ@mail.gmail.com>
Message-ID: <CA+=+wqCN4Q+poFEkSwRyXsx1eHTpeJjPwqQiSKz5BfkoyZReYw@mail.gmail.com>

On Thu, Nov 20, 2014 at 2:36 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 20 November 2014 22:54, MRAB <python at mrabarnett.plus.com> wrote:
>>
>> I'd prefer 'while' instead:
>>
>> >>> [x for x in range(10) while x < 5]
>> [0, 1, 2, 3, 4]
>> >>> {x for x in range(10) while x < 5}
>> {0, 1, 2, 3, 4}
>> >>> {x:x for x in range(10) while x < 5}
>> {0:0, 1:1, 2:2, 3:3, 4:4}
>
>
> That's been suggested many times, and always fails on the grounds of being
> completely incompatible with the expansions of comprehensions and generator
> expressions as nested compound statements.
>
> The return expression idea is one that reflects the actual deep structure of
> the nested scopes that underlie the syntactic sugar, so it's compatible with
> the expansion.
>
> This idea was mainly aimed at folks that might be inclined to worry about
> the possible loss of the "or stop()" generator expression trick in PEP 479,
> though. If that trick wasn't currently possible, and wasn't being proposed
> for removal, I never would have posted this idea.
>

The "while" expresses intent, "return" caters to low-level mechanics.
The "nested compound statements" explanation is already not that good:
why does the value come first, not last? For readability*. The
difference between function/nested compound statements syntax and
comprehension syntax is already so big, and the low-level "while
x:"?"if not x: break" translation is so easy, that between the two the
readability of "while" should win.

Outside of comprehensions, an expression that never has a value seems
quite backwards (and dangerous, in this case).

* (or you might say, for similarity to math notation)

From python at mrabarnett.plus.com  Thu Nov 20 16:49:53 2014
From: python at mrabarnett.plus.com (MRAB)
Date: Thu, 20 Nov 2014 15:49:53 +0000
Subject: [Python-ideas] Return expressions
In-Reply-To: <CA+=+wqCN4Q+poFEkSwRyXsx1eHTpeJjPwqQiSKz5BfkoyZReYw@mail.gmail.com>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
 <546DE48C.5000805@mrabarnett.plus.com>
 <CADiSq7cWybgFx7YtP+xYin3wLLX3kh3gjQxghUh7jqc=LVs8dQ@mail.gmail.com>
 <CA+=+wqCN4Q+poFEkSwRyXsx1eHTpeJjPwqQiSKz5BfkoyZReYw@mail.gmail.com>
Message-ID: <546E0DA1.10609@mrabarnett.plus.com>


On 2014-11-20 14:43, Petr Viktorin wrote:
> On Thu, Nov 20, 2014 at 2:36 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > On 20 November 2014 22:54, MRAB <python at mrabarnett.plus.com> wrote:
> >>
> >> I'd prefer 'while' instead:
> >>
> >> >>> [x for x in range(10) while x < 5]
> >> [0, 1, 2, 3, 4]
> >> >>> {x for x in range(10) while x < 5}
> >> {0, 1, 2, 3, 4}
> >> >>> {x:x for x in range(10) while x < 5}
> >> {0:0, 1:1, 2:2, 3:3, 4:4}
> >
> >
> > That's been suggested many times, and always fails on the grounds of being
> > completely incompatible with the expansions of comprehensions and generator
> > expressions as nested compound statements.
> >
> > The return expression idea is one that reflects the actual deep structure of
> > the nested scopes that underlie the syntactic sugar, so it's compatible with
> > the expansion.
> >
> > This idea was mainly aimed at folks that might be inclined to worry about
> > the possible loss of the "or stop()" generator expression trick in PEP 479,
> > though. If that trick wasn't currently possible, and wasn't being proposed
> > for removal, I never would have posted this idea.
> >
>
> The "while" expresses intent, "return" caters to low-level mechanics.
> The "nested compound statements" explanation is already not that good:
> why does the value come first, not last? For readability*. The
> difference between function/nested compound statements syntax and
> comprehension syntax is already so big, and the low-level "while
> x:"?"if not x: break" translation is so easy, that between the two the
> readability of "while" should win.
I can see the problem with 'while': if there are multiple 'for' parts, 
they are equivalent to nested
'for' loops, so you might assume that a 'while' part would also be 
equivalent to a nested 'while'
loop, whereas it would, in fact, be controlling the preceding 'for' part.

In other words, it would be:

     for x in range(10) while x < 5:
         ....

but could be seen as:

     for x in range(10):
         while x < 5:
             ....
> Outside of comprehensions, an expression that never has a value seems
> quite backwards (and dangerous, in this case).
>
> * (or you might say, for similarity to math notation)
>

From ron3200 at gmail.com  Thu Nov 20 17:11:30 2014
From: ron3200 at gmail.com (Ron Adam)
Date: Thu, 20 Nov 2014 10:11:30 -0600
Subject: [Python-ideas] Return expressions
In-Reply-To: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
Message-ID: <m4l3rj$r0f$1@ger.gmane.org>



On 11/20/2014 04:01 AM, Nick Coghlan wrote:
> This idea occurred to me in context of PEP 479 and the "or stop()" hack for
> generator expressions. I'm not a big enough fan of the idea to pursue it
> myself, but if anyone is bothered by the prospect of PEP 479 taking the "or
> stop()" technique away, this may be worth pursuing further.
>
> As a reminder of how the hack works, the following generator:
>
>  >>> def takewhile(iterable, pred):
> ...     for x in iter(iterable):
> ...         if not pred(x):
> ...             return
> ...         yield x
> ...
>  >>> list(takewhile(range(10), (lambda x: x < 5)))
> [0, 1, 2, 3, 4]

Just a note:

If this generator was used on an iterator more than once it would drop 
items between groups.

Takewhile fits the peek-ahead pattern I mentioned in an earlier thread that 
groupby matches.  I think groupby manages to hide the looked ahead value in 
the yield path, but it still does look ahead, just not in an obvious way.

It's my understanding looking ahead is problomatic for genrators because 
they may have side effects when next() is called on them.  Like reading 
input or writing output, or accessing some other object outside the generator.

It seams to me that most (if not all) partial iterations will either want 
to yield the "last" value tested, or stop "before" the last value tested. 
The generator expression version of the above has issues with both of those.

Cheers,
    Ron

















From ncoghlan at gmail.com  Thu Nov 20 17:21:53 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 21 Nov 2014 02:21:53 +1000
Subject: [Python-ideas] Return expressions
In-Reply-To: <546E0DA1.10609@mrabarnett.plus.com>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
 <546DE48C.5000805@mrabarnett.plus.com>
 <CADiSq7cWybgFx7YtP+xYin3wLLX3kh3gjQxghUh7jqc=LVs8dQ@mail.gmail.com>
 <CA+=+wqCN4Q+poFEkSwRyXsx1eHTpeJjPwqQiSKz5BfkoyZReYw@mail.gmail.com>
 <546E0DA1.10609@mrabarnett.plus.com>
Message-ID: <CADiSq7d1E3fXq6ShBUybHOcHzLzdzqoXV-BSzzaMvLXo09-2wg@mail.gmail.com>

On 21 November 2014 01:49, MRAB <python at mrabarnett.plus.com> wrote:
>
>
> On 2014-11-20 14:43, Petr Viktorin wrote:
>>
>> The "while" expresses intent, "return" caters to low-level mechanics.
>> The "nested compound statements" explanation is already not that good:
>> why does the value come first, not last? For readability*. The
>> difference between function/nested compound statements syntax and
>> comprehension syntax is already so big, and the low-level "while
>> x:"?"if not x: break" translation is so easy, that between the two the
>> readability of "while" should win.
>
> I can see the problem with 'while': if there are multiple 'for' parts, they are equivalent to nested
> 'for' loops, so you might assume that a 'while' part would also be equivalent to a nested 'while'
> loop, whereas it would, in fact, be controlling the preceding 'for' part.
>
> In other words, it would be:
>
>     for x in range(10) while x < 5:
>         ....
>
> but could be seen as:
>
>     for x in range(10):
>         while x < 5:


PEP 3142 was the last attempt at asking this question - Guido isn't
keen on the idea, so he rejected it the last time we did a pass
through the PEPs that weren't being actively worked on.

So while the odds still aren't good, a proposal that kept the
statement and expression forms consistent might still have some
chance, by proposing that the statement example above:

    for x in range(10) while x < 5:
        ....

Be equivalent to:

    for x in range(10):
        if not x < 5:
            break
        ....

That is, there'd only be one loop, rather than two.

At that point, a "while" clause in a comprehension or generator
expression would be part of that enhanced syntax, rather than a
standalone while loop.

That does also suggest another possible alternative to the "or stop()"
trick - allow *break* as an expression.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From rosuav at gmail.com  Thu Nov 20 17:36:16 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 21 Nov 2014 03:36:16 +1100
Subject: [Python-ideas] Return expressions
In-Reply-To: <CADiSq7d1E3fXq6ShBUybHOcHzLzdzqoXV-BSzzaMvLXo09-2wg@mail.gmail.com>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
 <546DE48C.5000805@mrabarnett.plus.com>
 <CADiSq7cWybgFx7YtP+xYin3wLLX3kh3gjQxghUh7jqc=LVs8dQ@mail.gmail.com>
 <CA+=+wqCN4Q+poFEkSwRyXsx1eHTpeJjPwqQiSKz5BfkoyZReYw@mail.gmail.com>
 <546E0DA1.10609@mrabarnett.plus.com>
 <CADiSq7d1E3fXq6ShBUybHOcHzLzdzqoXV-BSzzaMvLXo09-2wg@mail.gmail.com>
Message-ID: <CAPTjJmrqAoJp=-4sbpocXZ7XJaN7DAdAR8L3NbKmbCq3G0zZ3w@mail.gmail.com>

On Fri, Nov 21, 2014 at 3:21 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> That does also suggest another possible alternative to the "or stop()"
> trick - allow *break* as an expression.

Or just as a keyword component of a comprehension, in the same way
'for' is. Since 'if' already has meaning, [x for x in range(10) if x
>= 5 break] would be confusing, so probably it'd have to be the
slightly-awkward-to-read [x for x in range(10) break x >= 5], adding a
termination condition to the preceding loop.

ChrisA

From random832 at fastmail.us  Thu Nov 20 18:35:47 2014
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Thu, 20 Nov 2014 12:35:47 -0500
Subject: [Python-ideas] Return expressions
In-Reply-To: <546E0DA1.10609@mrabarnett.plus.com>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
 <546DE48C.5000805@mrabarnett.plus.com>
 <CADiSq7cWybgFx7YtP+xYin3wLLX3kh3gjQxghUh7jqc=LVs8dQ@mail.gmail.com>
 <CA+=+wqCN4Q+poFEkSwRyXsx1eHTpeJjPwqQiSKz5BfkoyZReYw@mail.gmail.com>
 <546E0DA1.10609@mrabarnett.plus.com>
Message-ID: <1416504947.266514.193437377.1DE3BBF5@webmail.messagingengine.com>



On Thu, Nov 20, 2014, at 10:49, MRAB wrote:

> I can see the problem with 'while': if there are multiple 'for' parts, 
> they are equivalent to nested
> 'for' loops, so you might assume that a 'while' part would also be 
> equivalent to a nested 'while'
> loop, whereas it would, in fact, be controlling the preceding 'for' part.
> 
> In other words, it would be:
> 
>      for x in range(10) while x < 5:
>          ....
> 
> but could be seen as:
> 
>      for x in range(10):
>          while x < 5:
>              ....

I've always thought of comprehensions as a sort of "inside-out" loop,
due to the value at the beginning: (f(a) for a in range) is roughly
equivalent to for a in range: yield f(a).

Due to this, I was actually somewhat surprised to find that multiple for
clauses don't work as if they were nested inside-out loops

>>> [(a, b) for a in range(3) for b in range(3)]
[(0, 0), (0, 1), (0, 2), (1, 0), (1, 1), (1, 2), (2, 0), (2, 1), (2, 2)]
>>> [[(a, b) for a in range(3)] for b in range(3)]
[[(0, 0), (1, 0), (2, 0)], [(0, 1), (1, 1), (2, 1)], [(0, 2), (1, 2),
(2, 2)]]

I think people trying to use "while" for this are coming from an
english-grammar point of view.

From njs at pobox.com  Thu Nov 20 19:02:31 2014
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 20 Nov 2014 18:02:31 +0000
Subject: [Python-ideas] Return expressions
In-Reply-To: <CAPTjJmrqAoJp=-4sbpocXZ7XJaN7DAdAR8L3NbKmbCq3G0zZ3w@mail.gmail.com>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
 <546DE48C.5000805@mrabarnett.plus.com>
 <CADiSq7cWybgFx7YtP+xYin3wLLX3kh3gjQxghUh7jqc=LVs8dQ@mail.gmail.com>
 <CA+=+wqCN4Q+poFEkSwRyXsx1eHTpeJjPwqQiSKz5BfkoyZReYw@mail.gmail.com>
 <546E0DA1.10609@mrabarnett.plus.com>
 <CADiSq7d1E3fXq6ShBUybHOcHzLzdzqoXV-BSzzaMvLXo09-2wg@mail.gmail.com>
 <CAPTjJmrqAoJp=-4sbpocXZ7XJaN7DAdAR8L3NbKmbCq3G0zZ3w@mail.gmail.com>
Message-ID: <CAPJVwBnCP3vMznUbTe6HL0AWWdF8vD_dqhUqio1bG0hFgjG1ZQ@mail.gmail.com>

On Thu, Nov 20, 2014 at 4:36 PM, Chris Angelico <rosuav at gmail.com> wrote:
> On Fri, Nov 21, 2014 at 3:21 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> That does also suggest another possible alternative to the "or stop()"
>> trick - allow *break* as an expression.
>
> Or just as a keyword component of a comprehension, in the same way
> 'for' is. Since 'if' already has meaning, [x for x in range(10) if x
>>= 5 break] would be confusing, so probably it'd have to be the
> slightly-awkward-to-read [x for x in range(10) break x >= 5], adding a
> termination condition to the preceding loop.

'break' is certainly better than 'return' -- currently 'break' means
"look for the nearest enclosing loop", not "look for the enclosing
function", so it'd preserve that at least.

[x for x in range(10) if x >= 5 else break]?

with 'else break' handled in the grammar as a non-compositional phrase
like 'is not'? Not sure how this interacts with multiple mixed for and
if clauses -- I've never wrapped my head around those semantics.

-n

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org

From antony.lee at berkeley.edu  Thu Nov 20 19:24:12 2014
From: antony.lee at berkeley.edu (Antony Lee)
Date: Thu, 20 Nov 2014 10:24:12 -0800
Subject: [Python-ideas] Return expressions
In-Reply-To: <CAPJVwBnCP3vMznUbTe6HL0AWWdF8vD_dqhUqio1bG0hFgjG1ZQ@mail.gmail.com>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
 <546DE48C.5000805@mrabarnett.plus.com>
 <CADiSq7cWybgFx7YtP+xYin3wLLX3kh3gjQxghUh7jqc=LVs8dQ@mail.gmail.com>
 <CA+=+wqCN4Q+poFEkSwRyXsx1eHTpeJjPwqQiSKz5BfkoyZReYw@mail.gmail.com>
 <546E0DA1.10609@mrabarnett.plus.com>
 <CADiSq7d1E3fXq6ShBUybHOcHzLzdzqoXV-BSzzaMvLXo09-2wg@mail.gmail.com>
 <CAPTjJmrqAoJp=-4sbpocXZ7XJaN7DAdAR8L3NbKmbCq3G0zZ3w@mail.gmail.com>
 <CAPJVwBnCP3vMznUbTe6HL0AWWdF8vD_dqhUqio1bG0hFgjG1ZQ@mail.gmail.com>
Message-ID: <CAGRr6BH41awv-uZs4q52_NjO8XRBcaMshzOTXGJEG21C1JO46A@mail.gmail.com>

This is fairly similar to a recent thread suggesting an "or raise"
construct ("x = f() or raise ValueError" as a shortcut for "x = f(); if not
x: raise ValueError").  I suggested that "raise" be made a value-less
expression, so that "x = f() or raise ..." can be intepreted normally.  As
a side effect, this would make lambdas slightly more powerful too.

Having "return" and "break" as value-less expressions would be nice too.
The interaction with nested loops in generators would be the natural one,
without any generator-specific syntax:

((x, y) if stay_in_inner(x, y) else break for x in X for y in Y)

is the same as

for x in X:
    for y in Y:
        if stay_in_inner(x, y):
            yield x, y
        else:
            break

or, with break-expression everywhere, as a separate generator:

for x in X:
    for y in Y:
        (yield x, y) if stay_in_inner(x, y) else break

whereas

((x, y) if stay_in_gen(x, y) else return for x in X for y in Y)

is the same as

for x in X:
    for y in Y:
        if stay_in_gen(x, y):
            yield x, y
        else:
            return

or

for x in X:
    for y in Y:
        (yield x, y) if stay_in_gen(x, y) else return

i.e. the intepretation of a genexp is still to take the nested "for"s and
"if"s after the leading expression, add suitable colons, newlines and
indentation, and insert the leading expression at the end.

Antony

2014-11-20 10:02 GMT-08:00 Nathaniel Smith <njs at pobox.com>:

> On Thu, Nov 20, 2014 at 4:36 PM, Chris Angelico <rosuav at gmail.com> wrote:
> > On Fri, Nov 21, 2014 at 3:21 AM, Nick Coghlan <ncoghlan at gmail.com>
> wrote:
> >> That does also suggest another possible alternative to the "or stop()"
> >> trick - allow *break* as an expression.
> >
> > Or just as a keyword component of a comprehension, in the same way
> > 'for' is. Since 'if' already has meaning, [x for x in range(10) if x
> >>= 5 break] would be confusing, so probably it'd have to be the
> > slightly-awkward-to-read [x for x in range(10) break x >= 5], adding a
> > termination condition to the preceding loop.
>
> 'break' is certainly better than 'return' -- currently 'break' means
> "look for the nearest enclosing loop", not "look for the enclosing
> function", so it'd preserve that at least.
>
> [x for x in range(10) if x >= 5 else break]?
>
> with 'else break' handled in the grammar as a non-compositional phrase
> like 'is not'? Not sure how this interacts with multiple mixed for and
> if clauses -- I've never wrapped my head around those semantics.
>
> -n
>
> --
> Nathaniel J. Smith
> Postdoctoral researcher - Informatics - University of Edinburgh
> http://vorpus.org
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141120/1d4db85c/attachment-0001.html>

From greg.ewing at canterbury.ac.nz  Thu Nov 20 22:13:48 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 21 Nov 2014 10:13:48 +1300
Subject: [Python-ideas] Return expressions
In-Reply-To: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
Message-ID: <546E598C.8000905@canterbury.ac.nz>

Nick Coghlan wrote:
>  >>> list(x for x in range(10) if x < 5 or return)
> [0, 1, 2, 3, 4]
> 
> This is emphatically *not* executable pseudocode - it only makes sense 
> in terms of the translation to the corresponding full generator function

-1 on messing with the semantics of 'return' (which would
have implications beyond this use case) just so you can write
something in one particular context that doesn't read intuitively.

If early exit from comprehensions is really a desirable feature
(and Guido doesn't seem to think so) it would be better addressed
with an explicit language feature, e.g.

    list(x for x in range(10) while x < 5)

    [x for x in range(10) while x < 5]

This makes the intent perfectly clear and works just as well
in all kinds of comprehensions.

It breaks the strict equivalence between comprehension syntax
and the corresponding nested loop code, but you can't have
everything.

-- 
Greg

From wolfgang.maier at biologie.uni-freiburg.de  Thu Nov 20 23:39:58 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Thu, 20 Nov 2014 23:39:58 +0100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmoM0my6HLta_7Kdey51vj7BUR2YPd2BMQO_DcPaRfUn-w@mail.gmail.com>
References: <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>
 <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
 <20141120020615.GI2748@ando.pearwood.info>
 <CAPTjJmoM0my6HLta_7Kdey51vj7BUR2YPd2BMQO_DcPaRfUn-w@mail.gmail.com>
Message-ID: <m4lqju$tj5$1@ger.gmane.org>

On 20.11.2014 03:24, Chris Angelico wrote:
> On Thu, Nov 20, 2014 at 1:06 PM, Steven D'Aprano <steve at pearwood.info> wrote:
>
>> I trust that we all expect to be able to factor out the raise into a
>> helper function or method, yes? It truly would be surprising if this
>> failed:
>>
>>
>> class MyIterator:
>>      def __iter__(self):
>>          return self
>>      def __next__(self):
>>          return something()
>>
>>
>> def something():
>>      # Toy helper function.
>>      if random.random() < 0.5:
>>          return "Spam!"
>>      raise StopIteration
>>
>>
>>
>> Now let's write this as a generator:
>>
>> def gen():
>>      while True:
>>          yield something()
>>
>>
>> which is much nicer than:
>>
>> def gen():
>>      while True:
>>          try:
>>              yield something()
>>          except StopIteration:
>>              return   # converted by Python into raise StopIteration
>
> Sure. There was a suggestion that "return yield from something()"
> would work, though, which - I can't confirm that this works, but
> assuming it does - would be a lot tidier. But there's still a
> difference. Your first helper function was specifically a __next__
> helper. It was tied intrinsically to the iterator protocol. If you
> want to call a __next__ helper (or actually call next(iter) on
> something) inside a generator, you'll have to - if this change goes
> through - cope with the fact that generator protocol says "return"
> where __next__ protocol says "raise StopIteration". If you want a
> generator helper, it'd look like this:
>
> def something():
>       # Toy helper function.
>       if random.random() < 0.5:
>           yield "Spam!"
>
> def gen():
>      yield from something()
>
> Voila! Now it's a generator helper, following generator protocol.
> Every bit as tidy as the original. Let's write a __getitem__ helper:
>

Hmm, I'm not convinced by these toy examples, but I did inspect some of 
my own code for incompatibility with the proposed change. I found that 
there really is only one recurring pattern I use that I'd have to change 
and that is how I've implemented several file parsers. I tend to write 
them like this:

def parser (file_object):
     while True:
         title_line = next(file_object) # will terminate after the last 
record

         try:
             # read and process the rest of the record here
         except StopIteration:
             # this record is incomplete
             raise OSError('Invalid file format')
         yield processed_record

So I'm catching StopIteration raised by the underlying IOWrapper only if 
it occurs in illegal places (with regard to the file format the parser 
expects), but not when it indicates the end of a correct file.
I always thought of letting the Error bubble up as a way to keep the 
parser transparent.
Now in this case, I think, I would have to change this to:

def parser (io_object):
     while True:
         try:
             title_line = next(io_object)
         except StopIteration:
             return
...

which I could certainly do without too much effort, but could this be 
one of the more widespread sources of incompatibility that Steve imagines ?

Wolfgang



From ron3200 at gmail.com  Thu Nov 20 23:50:03 2014
From: ron3200 at gmail.com (Ron Adam)
Date: Thu, 20 Nov 2014 16:50:03 -0600
Subject: [Python-ideas] Return expressions
In-Reply-To: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
Message-ID: <m4lr6s$7b1$1@ger.gmane.org>



On 11/20/2014 04:01 AM, Nick Coghlan wrote:
> Can currently be converted to a generator expression with the aid of a
> helper function:
>
>  >>> def stop():
> ...     raise StopIteration
> ...
>  >>> list(x for x in range(10) if x < 5 or stop())
> [0, 1, 2, 3, 4]

How about making a stop_generator() function builtin, just for this feature?

It could raise a specific StopIteration instance, possibly the same 
subclass/instance as return does in generators, so the generator would know 
to catch it instead of complaining.  As a builting, all the mechanics are 
kept out of the way as implementation details.

For those who already use stop() in many places, if there are any one who 
does that often, they could just do stop = stop_generator at the top of 
their programs.

I think "stop" is too general a name in this case.  It's fine when the 
function is defined right above where it is used, but as a builtin a more 
specific name would be better.

Cheers,
    Ron





From wolfgang.maier at biologie.uni-freiburg.de  Thu Nov 20 23:58:51 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Thu, 20 Nov 2014 23:58:51 +0100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <20141120020615.GI2748@ando.pearwood.info>
References: <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>
 <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
 <20141120020615.GI2748@ando.pearwood.info>
Message-ID: <m4lrnb$fa0$1@ger.gmane.org>

On 20.11.2014 03:06, Steven D'Aprano wrote:
> On Thu, Nov 20, 2014 at 03:24:07AM +1100, Chris Angelico wrote:
>
>> Without the generator, *only* __next__ has
>> this effect, and that's exactly where it's documented to be.
>
> The documentation states that __next__ raises StopIteration, it doesn't
> say that *only* __next__ should raise StopIteration.
>
> https://docs.python.org/3/library/stdtypes.html#iterator.__next__
>
> I trust that we all expect to be able to factor out the raise into a
> helper function or method, yes? It truly would be surprising if this
> failed:
>
>
> class MyIterator:
>      def __iter__(self):
>          return self
>      def __next__(self):
>          return something()
>
>
> def something():
>      # Toy helper function.
>      if random.random() < 0.5:
>          return "Spam!"
>      raise StopIteration
>
>
>
> Now let's write this as a generator:
>
> def gen():
>      while True:
>          yield something()
>
>
> which is much nicer than:
>
> def gen():
>      while True:
>          try:
>              yield something()
>          except StopIteration:
>              return   # converted by Python into raise StopIteration
>

I find this example a compelling argument against the PEP. Personally, 
I'm dealing a lot more often with refactoring a generator function into 
a iterator class than I'm rewriting generator expressions into 
comprehensions (at least the exotic kinds that would reveal their 
inequality).
So for me at least, the burden of having to remember that I can let (and 
should let) StopIteration bubble up inside __next__, but not in 
generator functions weighs in heavier than the equality argument and the 
protection against hard-to-diagnose (but rarely occurring) bugs in 
nested generator functions.


From rosuav at gmail.com  Fri Nov 21 00:30:50 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 21 Nov 2014 10:30:50 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <m4lrnb$fa0$1@ger.gmane.org>
References: <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>
 <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
 <20141120020615.GI2748@ando.pearwood.info>
 <m4lrnb$fa0$1@ger.gmane.org>
Message-ID: <CAPTjJmoWmcQL0Os3+Yr1TabNweA9fYiGAY1-vu5mv0OGnAYrZg@mail.gmail.com>

On Fri, Nov 21, 2014 at 9:58 AM, Wolfgang Maier
<wolfgang.maier at biologie.uni-freiburg.de> wrote:
> I find this example a compelling argument against the PEP. Personally, I'm
> dealing a lot more often with refactoring a generator function into a
> iterator class than I'm rewriting generator expressions into comprehensions
> (at least the exotic kinds that would reveal their inequality).
> So for me at least, the burden of having to remember that I can let (and
> should let) StopIteration bubble up inside __next__, but not in generator
> functions weighs in heavier than the equality argument and the protection
> against hard-to-diagnose (but rarely occurring) bugs in nested generator
> functions.

Compare my earlier response to Steven, though: it's not difficult to
refactor a generator into a helper-generator, rather than refactor a
generator into a helper-__next__. This proposal would force a
decoupling of generator protocol from __next__ protocol. The ugliness
in Steven's examples comes from trying to use a __next__ helper in a
generator. It'd be just as ugly trying to refactor __getitem__ to make
use of a __getattr__ helper - you'd have to catch AttributeError and
turn it into KeyError at the boundary between the two protocols.

ChrisA

From greg.ewing at canterbury.ac.nz  Thu Nov 20 22:28:32 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 21 Nov 2014 10:28:32 +1300
Subject: [Python-ideas] Return expressions
In-Reply-To: <m4l3rj$r0f$1@ger.gmane.org>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
 <m4l3rj$r0f$1@ger.gmane.org>
Message-ID: <546E5D00.6040005@canterbury.ac.nz>

Ron Adam wrote:

> It's my understanding looking ahead is problomatic for genrators because 
> they may have side effects when next() is called on them.

Not just generators, but any iterator.

Any algorithm that requires lookahead has problems if you
apply it more than once to the same iterator. Something has
to store the lookahead value in between times, and the
iterator protocol just doesn't allow for that.

-- 
Greg

From guido at python.org  Fri Nov 21 00:51:07 2014
From: guido at python.org (Guido van Rossum)
Date: Thu, 20 Nov 2014 15:51:07 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <m4lqju$tj5$1@ger.gmane.org>
References: <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>
 <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
 <20141120020615.GI2748@ando.pearwood.info>
 <CAPTjJmoM0my6HLta_7Kdey51vj7BUR2YPd2BMQO_DcPaRfUn-w@mail.gmail.com>
 <m4lqju$tj5$1@ger.gmane.org>
Message-ID: <CAP7+vJKeiWNW5Oo7FJmRVCoZ_u8vzg_GOwV-kytmMFdQozWVXg@mail.gmail.com>

On Thu, Nov 20, 2014 at 2:39 PM, Wolfgang Maier <
wolfgang.maier at biologie.uni-freiburg.de> wrote:

> [...]
> Hmm, I'm not convinced by these toy examples, but I did inspect some of my
> own code for incompatibility with the proposed change. I found that there
> really is only one recurring pattern I use that I'd have to change and that
> is how I've implemented several file parsers. I tend to write them like
> this:
>
> def parser (file_object):
>     while True:
>         title_line = next(file_object) # will terminate after the last
> record
>
>         try:
>             # read and process the rest of the record here
>         except StopIteration:
>             # this record is incomplete
>             raise OSError('Invalid file format')
>         yield processed_record
>
> So I'm catching StopIteration raised by the underlying IOWrapper only if
> it occurs in illegal places (with regard to the file format the parser
> expects), but not when it indicates the end of a correct file.
> I always thought of letting the Error bubble up as a way to keep the
> parser transparent.
> Now in this case, I think, I would have to change this to:
>
> def parser (io_object):
>     while True:
>         try:
>             title_line = next(io_object)
>         except StopIteration:
>             return
> ...
>
> which I could certainly do without too much effort, but could this be one
> of the more widespread sources of incompatibility that Steve imagines ?


There's probably something important missing from your examples. The above
while-loop is equivalent to

    for title_line in io_object:
        ...

If you're okay with getting RuntimeError instead of OSError for an
undesirable StopIteration, you can just drop the except clause altogether.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141120/8c3568d2/attachment.html>

From tjreedy at udel.edu  Fri Nov 21 06:00:31 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 21 Nov 2014 00:00:31 -0500
Subject: [Python-ideas] Return expressions
In-Reply-To: <546E0DA1.10609@mrabarnett.plus.com>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
 <546DE48C.5000805@mrabarnett.plus.com>
 <CADiSq7cWybgFx7YtP+xYin3wLLX3kh3gjQxghUh7jqc=LVs8dQ@mail.gmail.com>
 <CA+=+wqCN4Q+poFEkSwRyXsx1eHTpeJjPwqQiSKz5BfkoyZReYw@mail.gmail.com>
 <546E0DA1.10609@mrabarnett.plus.com>
Message-ID: <m4mgu1$vm2$3@ger.gmane.org>

On 11/20/2014 10:49 AM, MRAB wrote:

> I can see the problem with 'while': if there are multiple 'for' parts,
> they are equivalent to nested
> 'for' loops,

x for x in A if x=B
is also equivalent to nested compound statements.

> so you might assume that a 'while' part would also be
> equivalent to a nested 'while'
> loop, whereas it would, in fact, be controlling the preceding 'for' part.

-- 
Terry Jan Reedy


From wolfgang.maier at biologie.uni-freiburg.de  Fri Nov 21 11:19:40 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Fri, 21 Nov 2014 11:19:40 +0100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAP7+vJKeiWNW5Oo7FJmRVCoZ_u8vzg_GOwV-kytmMFdQozWVXg@mail.gmail.com>
References: <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>
 <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
 <20141120020615.GI2748@ando.pearwood.info>
 <CAPTjJmoM0my6HLta_7Kdey51vj7BUR2YPd2BMQO_DcPaRfUn-w@mail.gmail.com>
 <m4lqju$tj5$1@ger.gmane.org>
 <CAP7+vJKeiWNW5Oo7FJmRVCoZ_u8vzg_GOwV-kytmMFdQozWVXg@mail.gmail.com>
Message-ID: <m4n3js$tso$1@ger.gmane.org>

On 21.11.2014 00:51, Guido van Rossum wrote:
> On Thu, Nov 20, 2014 at 2:39 PM, Wolfgang Maier
> <wolfgang.maier at biologie.uni-freiburg.de
> <mailto:wolfgang.maier at biologie.uni-freiburg.de>>
> wrote:
>
>     [...]
>     Hmm, I'm not convinced by these toy examples, but I did inspect some
>     of my own code for incompatibility with the proposed change. I found
>     that there really is only one recurring pattern I use that I'd have
>     to change and that is how I've implemented several file parsers. I
>     tend to write them like this:
>
>     def parser (file_object):
>          while True:
>              title_line = next(file_object) # will terminate after the
>     last record
>
>              try:
>                  # read and process the rest of the record here
>              except StopIteration:
>                  # this record is incomplete
>                  raise OSError('Invalid file format')
>              yield processed_record
>
> There's probably something important missing from your examples. The
> above while-loop is equivalent to
>
>      for title_line in io_object:
>          ...
>

My reason for not using a for loop here is that I'm trying to read from 
a file where several lines form a record, so I'm reading the title line 
of a record (and if there is no record in the file any more I want the 
parser generator to terminate/return. If a title line is read 
successfully then I'm reading the record's body lines inside a 
try/except, i.e. where it says "# read and process the rest of the 
record here" in my shortened code I am actually calling next several 
times again to retrieve the body lines (and while reading these lines an 
unexpected StopIteration in the IOWrapper is considered a file format 
error).
I realize that I could also use a for loop and still call 
next(file_object) inside it, but I find this a potentially confusing 
pattern that I'm trying to avoid by using the while loop and all 
explicit next(). Compare:

for title_line in file_object:
     record_body = next(file_object)
     # in reality record_body is generated using several next calls
     # depending on the content found in the record body while it's read
     yield (title_line, record_body)

vs

while True:
     title_line = next(file_object)
     body = next(file_object)
     yield (title_line, body)

To me, the for loop version suggests to me that the content of 
file_object is read in line by line by the loop (even though the name 
title_line tries to hint at this being not true). Only when I inspect 
the loop body I see that further items are retrieved with next() and, 
thus, skipped in the for iteration. The while loop, on the other hand, 
makes the number of iterations very clear by showing all of them in the 
loop body.

Would you agree that this is justification enough for while instead of 
for or is it only me who thinks that a for loop makes the code read 
awkward ?


> If you're okay with getting RuntimeError instead of OSError for an
> undesirable StopIteration, you can just drop the except clause altogether.

Right, I could do this if the PEP-described behavior was in effect today.



From rosuav at gmail.com  Fri Nov 21 11:26:24 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 21 Nov 2014 21:26:24 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <m4n3js$tso$1@ger.gmane.org>
References: <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>
 <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
 <20141120020615.GI2748@ando.pearwood.info>
 <CAPTjJmoM0my6HLta_7Kdey51vj7BUR2YPd2BMQO_DcPaRfUn-w@mail.gmail.com>
 <m4lqju$tj5$1@ger.gmane.org>
 <CAP7+vJKeiWNW5Oo7FJmRVCoZ_u8vzg_GOwV-kytmMFdQozWVXg@mail.gmail.com>
 <m4n3js$tso$1@ger.gmane.org>
Message-ID: <CAPTjJmo0pspQXNX_7bPEw1rYXKHhAKK-6Q09zn0iY01i5NiJww@mail.gmail.com>

On Fri, Nov 21, 2014 at 9:19 PM, Wolfgang Maier
<wolfgang.maier at biologie.uni-freiburg.de> wrote:
> My reason for not using a for loop here is that I'm trying to read from a
> file where several lines form a record, so I'm reading the title line of a
> record (and if there is no record in the file any more I want the parser
> generator to terminate/return. If a title line is read successfully then I'm
> reading the record's body lines inside a try/except, i.e. where it says "#
> read and process the rest of the record here" in my shortened code I am
> actually calling next several times again to retrieve the body lines (and
> while reading these lines an unexpected StopIteration in the IOWrapper is
> considered a file format error).
> I realize that I could also use a for loop and still call next(file_object)
> inside it, but I find this a potentially confusing pattern that I'm trying
> to avoid by using the while loop and all explicit next().

I agree. The last example in the PEP is a cut-down form of your
parser, and I raise the exact same concern:

https://www.python.org/dev/peps/pep-0479/#examples

The use of the for loop strongly implies that the loop body will be
executed once for each thing in the iterable, which isn't true if you
next() it in the body. Legal? Sure. Confusing? Definitely.

ChrisA

From wolfgang.maier at biologie.uni-freiburg.de  Fri Nov 21 11:38:40 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Fri, 21 Nov 2014 11:38:40 +0100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmo0pspQXNX_7bPEw1rYXKHhAKK-6Q09zn0iY01i5NiJww@mail.gmail.com>
References: <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>
 <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
 <20141120020615.GI2748@ando.pearwood.info>
 <CAPTjJmoM0my6HLta_7Kdey51vj7BUR2YPd2BMQO_DcPaRfUn-w@mail.gmail.com>
 <m4lqju$tj5$1@ger.gmane.org>
 <CAP7+vJKeiWNW5Oo7FJmRVCoZ_u8vzg_GOwV-kytmMFdQozWVXg@mail.gmail.com>
 <m4n3js$tso$1@ger.gmane.org>
 <CAPTjJmo0pspQXNX_7bPEw1rYXKHhAKK-6Q09zn0iY01i5NiJww@mail.gmail.com>
Message-ID: <m4n4ng$ehp$1@ger.gmane.org>

On 21.11.2014 11:26, Chris Angelico wrote:
>
> I agree. The last example in the PEP is a cut-down form of your
> parser, and I raise the exact same concern:
>
> https://www.python.org/dev/peps/pep-0479/#examples
>
> The use of the for loop strongly implies that the loop body will be
> executed once for each thing in the iterable, which isn't true if you
> next() it in the body. Legal? Sure. Confusing? Definitely.
>

Yes, that example illustrates the exact same thing I tried to describe. 
Nice!



From raymond.hettinger at gmail.com  Fri Nov 21 12:24:40 2014
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Fri, 21 Nov 2014 03:24:40 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
Message-ID: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>


> On Nov 15, 2014, at 1:29 AM, Chris Angelico <rosuav at gmail.com> wrote:
> 
> Abstract
> ========
> 
> This PEP proposes a semantic change to ``StopIteration`` when raised
> inside a generator, unifying the behaviour of list comprehensions and
> generator expressions somewhat.


Please let me know if I'm reading the PEP correctly.
Does the proposal break all existing code in generators
that uses next() to raise StopIteration or that raises
StopIteration explicitly?

For example, here is the pure python recipe for itertools.accumulate()
show in the docs at https://docs.python.org/3/library/itertools.html#itertool-functions <https://docs.python.org/3/library/itertools.html#itertool-functions> :


    def accumulate(iterable, func=operator.add):
        'Return running totals'
        # accumulate([1,2,3,4,5]) --> 1 3 6 10 15
        # accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120
        it = iter(iterable)
        total = next(it)
        yield total
        for element in it:
            total = func(total, element)
            yield total

Or would it break the traditional examples of how to write something
like izip() using a generator?

    def izip(iterable1, iterable2):
        it1 = iter(iterable1)
        it2 = iter(iterable2)
        while True:
            v1 = next(it1)
            v2 = next(it2)
            yield v1, v2

    assert list(izip('ab', 'cde')) == [('a', 'c'), ('b', 'd')]
    assert list(izip('abc', 'cd')) == [('a', 'c'), ('b', 'd')]

My initial reading of the PEP was a bit unsettling because the listed
examples (such as unwrap() and parser()) were a series of cases 
where code that was currently working just fine for the last decade
would break and need be changed to less pleasant looking code.

Also, the PEP motivation seemed somewhat weak.  Instead of listing
known bugs or real-world development difficulties, it seems to hinge
almost entirely  some "being surprised" that list comprehensions and
generator expressions aren't the same in every regard (they aren't).

AFAICT, that suggestion is that an incorrect expectation of perfect symmetry
warrants a number of what the author calls "consequences for existing code".

It seems that if the real problem is one of false expectations or surprises,
the direct solution would be to provide clearer examples of how things
actually work and to disabuse the idea that list comprehensions and
generator expressions are more interchangeable than they actually are.


Raymond


P.S.   On a more general note, I think that our biggest problem
in the Python world is getting people to switch to Python 3.
If we really want that to happen, we should develop a strong
aversion to proposals that further increase the semantic
difference between Python 2 and Python 3.









-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141121/ecf60e6d/attachment.html>

From rosuav at gmail.com  Fri Nov 21 12:50:52 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 21 Nov 2014 22:50:52 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
Message-ID: <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>

On Fri, Nov 21, 2014 at 10:24 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
>
> Please let me know if I'm reading the PEP correctly.
> Does the proposal break all existing code in generators
> that uses next() to raise StopIteration or that raises
> StopIteration explicitly?
>
> For example, here is the pure python recipe for itertools.accumulate()
> show in the docs at
> https://docs.python.org/3/library/itertools.html#itertool-functions :
>
>
>     def accumulate(iterable, func=operator.add):
>         'Return running totals'
>         # accumulate([1,2,3,4,5]) --> 1 3 6 10 15
>         # accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120
>         it = iter(iterable)
>         total = next(it)
>         yield total
>         for element in it:
>             total = func(total, element)
>             yield total

The case where the iterable is empty would now raise, yes.

> Or would it break the traditional examples of how to write something
> like izip() using a generator?
>
>     def izip(iterable1, iterable2):
>         it1 = iter(iterable1)
>         it2 = iter(iterable2)
>         while True:
>             v1 = next(it1)
>             v2 = next(it2)
>             yield v1, v2
>
>     assert list(izip('ab', 'cde')) == [('a', 'c'), ('b', 'd')]
>     assert list(izip('abc', 'cd')) == [('a', 'c'), ('b', 'd')]

Yes, this would be affected. This proposal causes a separation of
generators and iterators, so it's no longer possible to pretend that
they're the same thing.

> Also, the PEP motivation seemed somewhat weak.  Instead of listing
> known bugs or real-world development difficulties, it seems to hinge
> almost entirely  some "being surprised" that list comprehensions and
> generator expressions aren't the same in every regard (they aren't).

The main point is one of exceptions being silently suppressed.
Iterator protocol involves the StopIteration exception; generator
protocol doesn't, yet currently a generator that raises StopIteration
will quietly terminate. It's as if every generator is wrapped inside
"try: ..... except StopIteration: pass". Would you accept any function
being written with that kind of implicit suppression of any other
exception?

> P.S.   On a more general note, I think that our biggest problem
> in the Python world is getting people to switch to Python 3.
> If we really want that to happen, we should develop a strong
> aversion to proposals that further increase the semantic
> difference between Python 2 and Python 3.

The recommended form of the code will work exactly the same way in
both versions: explicitly catching StopIteration and using it as a
signal that the function should terminate. The only difference is the
behaviour of the non-recommended practice of allowing an exception to
bubble part-way and then be implicitly caught.

ChrisA

From ncoghlan at gmail.com  Fri Nov 21 13:55:34 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 21 Nov 2014 22:55:34 +1000
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
Message-ID: <CADiSq7ch3ZRrGV2FBYtRb3uQb=yQBRJPtHv4jHpPSHz089a3YA@mail.gmail.com>

On 21 November 2014 21:50, Chris Angelico <rosuav at gmail.com> wrote:
> The recommended form of the code will work exactly the same way in
> both versions: explicitly catching StopIteration and using it as a
> signal that the function should terminate. The only difference is the
> behaviour of the non-recommended practice of allowing an exception to
> bubble part-way and then be implicitly caught.

Raymond's point is that for a long time, the equivalence between
"return" and "raise StopIteration" in a generator function has been
explicit. The dissatisfaction with the "non-local flow control"
aspects of the latter really only started to creep in around Python
2.5 (based on the explicit decision to avoid non-local flow control
behaviour in the definition of the with statement in PEP 343), and
this PEP is the first time this longstanding behaviour of generators
has been seriously questioned at the python-dev level.

Guido also didn't add himself as a co-author on the PEP, so it isn't
clear on first reading that *he's* the one considering the change,
rather than it being an independent suggestion on your part :)

I suspect enough evidence of breakage is accumulating to tip the
balance back to "not worth the hassle", but it would also be possible
to just *add* the "from __future__ import generator_stop" feature, and
postpone a decision on making that the only available behaviour.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From wolfgang.maier at biologie.uni-freiburg.de  Fri Nov 21 14:31:33 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Fri, 21 Nov 2014 14:31:33 +0100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
Message-ID: <m4nerl$7g7$1@ger.gmane.org>

On 21.11.2014 12:24, Raymond Hettinger wrote:
>
>> On Nov 15, 2014, at 1:29 AM, Chris Angelico
>> <rosuav at gmail.com
>> <mailto:rosuav at gmail.com>> wrote:
>>
>> Abstract
>> ========
>>
>> This PEP proposes a semantic change to ``StopIteration`` when raised
>> inside a generator, unifying the behaviour of list comprehensions and
>> generator expressions somewhat.
>
>
> Please let me know if I'm reading the PEP correctly.
> Does the proposal break all existing code in generators
> that uses next() to raise StopIteration or that raises
> StopIteration explicitly?
>
> For example, here is the pure python recipe for itertools.accumulate()
> show in the docs at
> https://docs.python.org/3/library/itertools.html#itertool-functions :
>
>
>      def accumulate(iterable, func=operator.add):
>          'Return running totals'
>          # accumulate([1,2,3,4,5]) --> 1 3 6 10 15
>          # accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120
>          it = iter(iterable)
>          total = next(it)
>          yield total
>          for element in it:
>              total = func(total, element)
>              yield total
>
> Or would it break the traditional examples of how to write something
> like izip() using a generator?
>
>      def izip(iterable1, iterable2):
>          it1 = iter(iterable1)
>          it2 = iter(iterable2)
>          while True:
>              v1 = next(it1)
>              v2 = next(it2)
>              yield v1, v2
>
>      assert list(izip('ab', 'cde')) == [('a', 'c'), ('b', 'd')]
>      assert list(izip('abc', 'cd')) == [('a', 'c'), ('b', 'd')]
>

Since I already learnt quite a lot from following this thread:
I checked yesterday what the docs have to say about the pure-python 
equivalent of python3's zip() because I expected it to look like the 
above izip recipe (making it incompatible with the PEP behavior). 
However, I found that the given equivalent code is:

def zip(*iterables):
     # zip('ABCD', 'xy') --> Ax By
     sentinel = object()
     iterators = [iter(it) for it in iterables]
     while iterators:
         result = []
         for it in iterators:
             elem = next(it, sentinel)
             if elem is sentinel:
                 return
             result.append(elem)
         yield tuple(result)

i.e., there is no unprotected next call in this example.

What surprised me though is that the protection here is done via the 
default argument of next() while more typically you'd use a try/except 
clause. So what's the difference between the two ? Specifically, with a 
default value given will next just catch StopIteration, which you could 
do less verbosely yourself and/or is there some speed gain from the fact 
that the Error has to be propagated up one level less ?
Is there a guideline when to use try/except vs. next with a default value ?

Thanks,
Wolfgang


From guido at python.org  Fri Nov 21 16:39:40 2014
From: guido at python.org (Guido van Rossum)
Date: Fri, 21 Nov 2014 07:39:40 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <m4n3js$tso$1@ger.gmane.org>
References: <CAPTjJmqZXke=dsxHyMq8D_2jkRyRiR+hkdE84V-NwkFvJ5=8=w@mail.gmail.com>
 <CAP7+vJJcZeAa0_wmoTAuDhGfAijDWLNK_UtBPxqGw9M6tbweZw@mail.gmail.com>
 <CAPTjJmpULUyyKQDwfs_nLsSxYjUSzKL0cR4Qh0Gczbxz5hnELg@mail.gmail.com>
 <546A3DEE.5040206@stoneleaf.us>
 <CAP7+vJLcrH7o=1AT8qODtneBpCCaxWJ8zhaoJ14_o8Kkyxj5CA@mail.gmail.com>
 <20141117235641.GW2748@ando.pearwood.info>
 <CAP7+vJLyNqqsW0pUWavWHXpctne9tLyx0pE8vZ-5F_QS0_zA3Q@mail.gmail.com>
 <CAP7+vJKGULT5sMyB4251X695LUoSwo0hEVQdMUFLCnxe0CN9Ag@mail.gmail.com>
 <CADiSq7feB_ufekGU23e0T6doUwbgiox5Ukq4hoN2noAruhJEHg@mail.gmail.com>
 <CAPTjJmq+eJGz6DrVKP5y1HnKnXxJgGbMDLfKpLnv9s0POA2QVA@mail.gmail.com>
 <20141120020615.GI2748@ando.pearwood.info>
 <CAPTjJmoM0my6HLta_7Kdey51vj7BUR2YPd2BMQO_DcPaRfUn-w@mail.gmail.com>
 <m4lqju$tj5$1@ger.gmane.org>
 <CAP7+vJKeiWNW5Oo7FJmRVCoZ_u8vzg_GOwV-kytmMFdQozWVXg@mail.gmail.com>
 <m4n3js$tso$1@ger.gmane.org>
Message-ID: <CAP7+vJJ5FWxs8pARPGvVmUZc09_NOUVUXC3qXyMHE1AZOhg4MA@mail.gmail.com>

On Fri, Nov 21, 2014 at 2:19 AM, Wolfgang Maier <
wolfgang.maier at biologie.uni-freiburg.de> wrote:

> On 21.11.2014 00:51, Guido van Rossum wrote:
>
>> On Thu, Nov 20, 2014 at 2:39 PM, Wolfgang Maier
>> <wolfgang.maier at biologie.uni-freiburg.de
>> <mailto:wolfgang.maier at biologie.uni-freiburg.de>>
>> wrote:
>>
>>     [...]
>>     Hmm, I'm not convinced by these toy examples, but I did inspect some
>>     of my own code for incompatibility with the proposed change. I found
>>     that there really is only one recurring pattern I use that I'd have
>>     to change and that is how I've implemented several file parsers. I
>>     tend to write them like this:
>>
>>     def parser (file_object):
>>          while True:
>>              title_line = next(file_object) # will terminate after the
>>     last record
>>
>>              try:
>>                  # read and process the rest of the record here
>>              except StopIteration:
>>                  # this record is incomplete
>>                  raise OSError('Invalid file format')
>>              yield processed_record
>>
>> There's probably something important missing from your examples. The
>> above while-loop is equivalent to
>>
>>      for title_line in io_object:
>>          ...
>>
>>
> My reason for not using a for loop here is that I'm trying to read from a
> file where several lines form a record, so I'm reading the title line of a
> record (and if there is no record in the file any more I want the parser
> generator to terminate/return. If a title line is read successfully then
> I'm reading the record's body lines inside a try/except, i.e. where it says
> "# read and process the rest of the record here" in my shortened code I am
> actually calling next several times again to retrieve the body lines (and
> while reading these lines an unexpected StopIteration in the IOWrapper is
> considered a file format error).
> I realize that I could also use a for loop and still call
> next(file_object) inside it, but I find this a potentially confusing
> pattern that I'm trying to avoid by using the while loop and all explicit
> next(). Compare:
>
> for title_line in file_object:
>     record_body = next(file_object)
>     # in reality record_body is generated using several next calls
>     # depending on the content found in the record body while it's read
>     yield (title_line, record_body)
>
> vs
>
> while True:
>     title_line = next(file_object)
>     body = next(file_object)
>     yield (title_line, body)
>
> To me, the for loop version suggests to me that the content of file_object
> is read in line by line by the loop (even though the name title_line tries
> to hint at this being not true). Only when I inspect the loop body I see
> that further items are retrieved with next() and, thus, skipped in the for
> iteration. The while loop, on the other hand, makes the number of
> iterations very clear by showing all of them in the loop body.
>
> Would you agree that this is justification enough for while instead of for
> or is it only me who thinks that a for loop makes the code read awkward ?
>

Now that you have explained it I see your point.


>  If you're okay with getting RuntimeError instead of OSError for an
>> undesirable StopIteration, you can just drop the except clause altogether.
>>
>
> Right, I could do this if the PEP-described behavior was in effect today.
>

 So shouldn't you be voting *for* the PEP?

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141121/75e0c003/attachment.html>

From barry at python.org  Fri Nov 21 16:53:15 2014
From: barry at python.org (Barry Warsaw)
Date: Fri, 21 Nov 2014 10:53:15 -0500
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <CADiSq7ch3ZRrGV2FBYtRb3uQb=yQBRJPtHv4jHpPSHz089a3YA@mail.gmail.com>
Message-ID: <20141121105315.2ef5545e@anarchist.wooz.org>

On Nov 21, 2014, at 10:55 PM, Nick Coghlan wrote:

>I suspect enough evidence of breakage is accumulating to tip the
>balance back to "not worth the hassle", but it would also be possible
>to just *add* the "from __future__ import generator_stop" feature, and
>postpone a decision on making that the only available behaviour.

I have no opinion on the actual PEP, but I'm not sure the above is a good
resolution.  future imports should be for things that have a clear path to
default behavior in some future release.  I don't think we should incur
technical debt to future-ize a feature that won't eventually get adopted.
Such a thing will just be another wart that will be difficult to remove for
backward compatibility.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141121/f177f66d/attachment.sig>

From abarnert at yahoo.com  Fri Nov 21 17:22:55 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 21 Nov 2014 08:22:55 -0800
Subject: [Python-ideas] Return expressions
In-Reply-To: <CADiSq7d1E3fXq6ShBUybHOcHzLzdzqoXV-BSzzaMvLXo09-2wg@mail.gmail.com>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
 <546DE48C.5000805@mrabarnett.plus.com>
 <CADiSq7cWybgFx7YtP+xYin3wLLX3kh3gjQxghUh7jqc=LVs8dQ@mail.gmail.com>
 <CA+=+wqCN4Q+poFEkSwRyXsx1eHTpeJjPwqQiSKz5BfkoyZReYw@mail.gmail.com>
 <546E0DA1.10609@mrabarnett.plus.com>
 <CADiSq7d1E3fXq6ShBUybHOcHzLzdzqoXV-BSzzaMvLXo09-2wg@mail.gmail.com>
Message-ID: <CEE296EB-7DE2-4EF6-9831-A0FEB0F47FB7@yahoo.com>



Sent from a random iPhone

On Nov 20, 2014, at 8:21, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 21 November 2014 01:49, MRAB <python at mrabarnett.plus.com> wrote:
>> 
>> 
>> On 2014-11-20 14:43, Petr Viktorin wrote:
>>> 
>>> The "while" expresses intent, "return" caters to low-level mechanics.
>>> The "nested compound statements" explanation is already not that good:
>>> why does the value come first, not last? For readability*. The
>>> difference between function/nested compound statements syntax and
>>> comprehension syntax is already so big, and the low-level "while
>>> x:"?"if not x: break" translation is so easy, that between the two the
>>> readability of "while" should win.
>> 
>> I can see the problem with 'while': if there are multiple 'for' parts, they are equivalent to nested
>> 'for' loops, so you might assume that a 'while' part would also be equivalent to a nested 'while'
>> loop, whereas it would, in fact, be controlling the preceding 'for' part.
>> 
>> In other words, it would be:
>> 
>>    for x in range(10) while x < 5:
>>        ....
>> 
>> but could be seen as:
>> 
>>    for x in range(10):
>>        while x < 5:
> 
> 
> PEP 3142 was the last attempt at asking this question - Guido isn't
> keen on the idea, so he rejected it the last time we did a pass
> through the PEPs that weren't being actively worked on.

The fact few times this has came up on -ideas or -dev, people suggested a variety of alternative syntaxes for this idea, just as is happening this time, and so far, they've always been subsets of the same ideas.

There are syntaxes that map nicely to the nested statement translation but don't read well, syntaxes that read well but make no sense as nested statements, syntaxes that sort of meet both those criteria but are so verbose nobody would use them, syntaxes that only make sense if everyone knows that comprehensions (including genexprs) are run inside their own hidden functions, syntaxes that only make sense if you pretend they aren't, syntaxes that require a change to normal for statements that wouldn't ever realistically be used there, and syntaxes that change the basic principle behind how comprehensions work. (And maybe my suggestion to make or stop() work in listcomps the same way fits in here, even though the goal was to unify the semantics of all genexprs and other comprehensions, rather than to provide this functionality.)

So far, Guido hasn't liked any of them, and none of them have gotten the kind of widespread buy-in from the community that seem likely to convince him to take another look. So it seems like a waste of time to rehash them every year or so.

Maybe what we need to do is update PEP 3142, gathering all of the alternative ideas, and expanding on the rationale, so Guido can reject that, and next time this comes up, everyone will have something to read before having the same arguments about the same proposals?

From steve at pearwood.info  Fri Nov 21 17:30:32 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 22 Nov 2014 03:30:32 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
Message-ID: <20141121163032.GN2748@ando.pearwood.info>

On Fri, Nov 21, 2014 at 10:50:52PM +1100, Chris Angelico wrote:
> On Fri, Nov 21, 2014 at 10:24 PM, Raymond Hettinger
> <raymond.hettinger at gmail.com> wrote:

> > Or would it break the traditional examples of how to write something
> > like izip() using a generator?
> >
> >     def izip(iterable1, iterable2):
> >         it1 = iter(iterable1)
> >         it2 = iter(iterable2)
> >         while True:
> >             v1 = next(it1)
> >             v2 = next(it2)
> >             yield v1, v2
> >
> >     assert list(izip('ab', 'cde')) == [('a', 'c'), ('b', 'd')]
> >     assert list(izip('abc', 'cd')) == [('a', 'c'), ('b', 'd')]
> 
> Yes, this would be affected. This proposal causes a separation of
> generators and iterators, so it's no longer possible to pretend that
> they're the same thing.

But generators and iterators *are the same thing*. (Generator functions 
are not iterators, but generators themselves are.) Iterators don't have 
a specific type, but they obey the iterator protocol:


py> def gen():
...     yield 1
...     yield 2
...
py> it = gen()
py> iter(it) is it
True
py> hasattr(it, '__next__')
True

`it` is an iterator.


> > Also, the PEP motivation seemed somewhat weak.  Instead of listing
> > known bugs or real-world development difficulties, it seems to hinge
> > almost entirely  some "being surprised" that list comprehensions and
> > generator expressions aren't the same in every regard (they aren't).
> 
> The main point is one of exceptions being silently suppressed.
> Iterator protocol involves the StopIteration exception; generator
> protocol doesn't, yet currently a generator that raises StopIteration
> will quietly terminate. It's as if every generator is wrapped inside
> "try: ..... except StopIteration: pass". Would you accept any function
> being written with that kind of implicit suppression of any other
> exception?

Yes.

That's how the classic pre-iterator iteration protocol works:

py> class K:
...     def __getitem__(self, i):
...             if i == 5: raise IndexError
...             return i
...
py> x = K()
py> list(x)
[0, 1, 2, 3, 4]


Context managers support suppressing any exception which occurs:

    If the suite was exited due to an exception, and the return 
    value from the __exit__() method was false, the exception is 
    reraised. If the return value was true, the exception is 
    suppressed, and execution continues with the statement
    following the with statement.

https://docs.python.org/3/reference/compound_stmts.html#the-with-statement


So there's two examples, one of the oldest going back to Python 1 days, 
and one of the newest. There may be others.



-- 
Steven

From ethan at stoneleaf.us  Fri Nov 21 17:36:04 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 21 Nov 2014 08:36:04 -0800
Subject: [Python-ideas] Return expressions
In-Reply-To: <CEE296EB-7DE2-4EF6-9831-A0FEB0F47FB7@yahoo.com>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
 <546DE48C.5000805@mrabarnett.plus.com>
 <CADiSq7cWybgFx7YtP+xYin3wLLX3kh3gjQxghUh7jqc=LVs8dQ@mail.gmail.com>
 <CA+=+wqCN4Q+poFEkSwRyXsx1eHTpeJjPwqQiSKz5BfkoyZReYw@mail.gmail.com>
 <546E0DA1.10609@mrabarnett.plus.com>
 <CADiSq7d1E3fXq6ShBUybHOcHzLzdzqoXV-BSzzaMvLXo09-2wg@mail.gmail.com>
 <CEE296EB-7DE2-4EF6-9831-A0FEB0F47FB7@yahoo.com>
Message-ID: <546F69F4.9000301@stoneleaf.us>

On 11/21/2014 08:22 AM, Andrew Barnert wrote:
> 
> Maybe what we need to do is update PEP 3142, gathering all of the alternative ideas,
> and expanding on the rationale, so Guido can reject that, and next time this comes
> up, everyone will have something to read before having the same arguments about the
> same proposals?

Are we accepting nominations?  I have the perfect fellow in mind.  :)

(And in case Chris decides to nominate me in retaliation, I decline. ;)

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141121/1ec14f07/attachment.sig>

From rosuav at gmail.com  Fri Nov 21 17:43:38 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 22 Nov 2014 03:43:38 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <20141121163032.GN2748@ando.pearwood.info>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info>
Message-ID: <CAPTjJmqia+KrGsNGQPy8o98U4Mm0qZbaWRVXf_Y6rkFz9x7-zA@mail.gmail.com>

On Sat, Nov 22, 2014 at 3:30 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Fri, Nov 21, 2014 at 10:50:52PM +1100, Chris Angelico wrote:
>> Yes, this would be affected. This proposal causes a separation of
>> generators and iterators, so it's no longer possible to pretend that
>> they're the same thing.
>
> But generators and iterators *are the same thing*. (Generator functions
> are not iterators, but generators themselves are.) Iterators don't have
> a specific type, but they obey the iterator protocol:

I can write many other factory functions which return iterators. They
are not, themselves, iterators, and therefore should not be expected
to follow iterator protocol.

def gen():
    return iter([1,2])


> py> it = gen()
> py> iter(it) is it
> True
> py> hasattr(it, '__next__')
> True
>
> `it` is an iterator.

The above function works with those tests, too. Generator functions
are functions that return iterators, and the __next__ method of the
returned object is what follows iterator protocol.

>> The main point is one of exceptions being silently suppressed.
>> Iterator protocol involves the StopIteration exception; generator
>> protocol doesn't, yet currently a generator that raises StopIteration
>> will quietly terminate. It's as if every generator is wrapped inside
>> "try: ..... except StopIteration: pass". Would you accept any function
>> being written with that kind of implicit suppression of any other
>> exception?
>
> Yes.
>
> That's how the classic pre-iterator iteration protocol works:
>
> py> class K:
> ...     def __getitem__(self, i):
> ...             if i == 5: raise IndexError
> ...             return i
> ...
> py> x = K()
> py> list(x)
> [0, 1, 2, 3, 4]

That's following getitem protocol, and it's part of that protocol for
the raising of IndexError to be the way of not returning any value.
But what's more surprising is that raising StopIteration will also
silently halt iteration, which I think is not good:

>>> class K:
    def __getitem__(self, i):
        if i == 5: raise StopIteration
        return i

>>> list(K())
[0, 1, 2, 3, 4]

> Context managers support suppressing any exception which occurs:
>
>     If the suite was exited due to an exception, and the return
>     value from the __exit__() method was false, the exception is
>     reraised. If the return value was true, the exception is
>     suppressed, and execution continues with the statement
>     following the with statement.
>
> https://docs.python.org/3/reference/compound_stmts.html#the-with-statement

Context managers get a chance to function like a try/except block. If
one silently and unexpectedly suppresses an exception, it's going to
be surprising; but more likely, it's as clear and explicit as an
actual try/except block. This isn't "as soon as you use a 'with'
block, any XyzError will jump to the end of the block and keep going".

ChrisA

From rosuav at gmail.com  Fri Nov 21 17:49:15 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 22 Nov 2014 03:49:15 +1100
Subject: [Python-ideas] Return expressions
In-Reply-To: <546F69F4.9000301@stoneleaf.us>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
 <546DE48C.5000805@mrabarnett.plus.com>
 <CADiSq7cWybgFx7YtP+xYin3wLLX3kh3gjQxghUh7jqc=LVs8dQ@mail.gmail.com>
 <CA+=+wqCN4Q+poFEkSwRyXsx1eHTpeJjPwqQiSKz5BfkoyZReYw@mail.gmail.com>
 <546E0DA1.10609@mrabarnett.plus.com>
 <CADiSq7d1E3fXq6ShBUybHOcHzLzdzqoXV-BSzzaMvLXo09-2wg@mail.gmail.com>
 <CEE296EB-7DE2-4EF6-9831-A0FEB0F47FB7@yahoo.com>
 <546F69F4.9000301@stoneleaf.us>
Message-ID: <CAPTjJmosQVXr4qbbewYAuQRtYKv-LpADRKfy2XukN5dhyt5p7A@mail.gmail.com>

On Sat, Nov 22, 2014 at 3:36 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 11/21/2014 08:22 AM, Andrew Barnert wrote:
>>
>> Maybe what we need to do is update PEP 3142, gathering all of the alternative ideas,
>> and expanding on the rationale, so Guido can reject that, and next time this comes
>> up, everyone will have something to read before having the same arguments about the
>> same proposals?
>
> Are we accepting nominations?  I have the perfect fellow in mind.  :)
>
> (And in case Chris decides to nominate me in retaliation, I decline. ;)

Heh! Great, I'm going to get a name for writing the PEPs that exist
solely to be rejected... I was hoping PEP 479 would be accepted
easily, given who was advocating it, but now it's starting to look
like I doomed the proposal :)

ChrisA

From ethan at stoneleaf.us  Fri Nov 21 17:49:47 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 21 Nov 2014 08:49:47 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
 generators
In-Reply-To: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
Message-ID: <546F6D2B.1040007@stoneleaf.us>

On 11/21/2014 03:24 AM, Raymond Hettinger wrote:
> 
> Also, the PEP motivation seemed somewhat weak.  Instead of listing
> known bugs or real-world development difficulties, it seems to hinge
> almost entirely  some "being surprised" that list comprehensions and
> generator expressions aren't the same in every regard (they aren't).

I believe the motivation is more along the lines of the difficulty and time wasted in debugging a malfunctioning program
when a generator stops early because a StopIteration escaped instead of having some other exception raised.

This would be along the same lines as not allowing sum to work with str -- a more valid case, IMO, because the sum
restriction is performance based, while this change would actually prevent breakage... or more accurately, put the
breakage at the cause and make it much easier to fix.

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141121/9ef1057a/attachment.sig>

From ethan at stoneleaf.us  Fri Nov 21 17:52:59 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 21 Nov 2014 08:52:59 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
 generators
In-Reply-To: <20141121163032.GN2748@ando.pearwood.info>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info>
Message-ID: <546F6DEB.50906@stoneleaf.us>

On 11/21/2014 08:30 AM, Steven D'Aprano wrote:
> 
> But generators and iterators *are the same thing*. (Generator functions 
> are not iterators, but generators themselves are.) Iterators don't have 
> a specific type, but they obey the iterator protocol:

Um, no, they aren't.  One cannot 'send()' into any ol' iterator.

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141121/d75aba43/attachment-0001.sig>

From steve at pearwood.info  Fri Nov 21 18:16:23 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 22 Nov 2014 04:16:23 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <546F6DEB.50906@stoneleaf.us>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info> <546F6DEB.50906@stoneleaf.us>
Message-ID: <20141121171622.GP2748@ando.pearwood.info>

On Fri, Nov 21, 2014 at 08:52:59AM -0800, Ethan Furman wrote:
> On 11/21/2014 08:30 AM, Steven D'Aprano wrote:
> > 
> > But generators and iterators *are the same thing*. (Generator functions 
> > are not iterators, but generators themselves are.) Iterators don't have 
> > a specific type, but they obey the iterator protocol:
> 
> Um, no, they aren't.  One cannot 'send()' into any ol' iterator.

"Must not support send()" has never been part of the definition of 
iterators.

The `Iterator` ABC also recognises generators as iterators:

py> def gen():
...     yield 1
...
py> from collections import Iterator
py> isinstance(gen(), Iterator)
True


and they are documented as iterators:

    Python?s generators provide a convenient way to implement
    the iterator protocol. If a container object?s __iter__() 
    method is implemented as a generator, it will automatically 
    return an iterator object (technically, a generator object) 
    supplying the __iter__() and __next__() methods.

https://docs.python.org/3/library/stdtypes.html#generator-types


I don't understand where this idea that generators aren't iterators has 
come from, unless it is confusion between the generator *function* and 
the generator object itself.


-- 
Steven

From abarnert at yahoo.com  Fri Nov 21 18:18:29 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 21 Nov 2014 09:18:29 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <546F6DEB.50906@stoneleaf.us>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info> <546F6DEB.50906@stoneleaf.us>
Message-ID: <592A1ABA-41FE-4643-B6B2-CFD733982EB6@yahoo.com>

On Nov 21, 2014, at 8:52, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 11/21/2014 08:30 AM, Steven D'Aprano wrote:
>> 
>> But generators and iterators *are the same thing*. (Generator functions 
>> are not iterators, but generators themselves are.) Iterators don't have 
>> a specific type, but they obey the iterator protocol:
> 
> Um, no, they aren't.  One cannot 'send()' into any ol' iterator.

Generators are a subtype of iterators. They support the iterator protocol completely, and in the same way as any other iterator. They also support extensions to that protocol--e.g., send(). But they also have a relationship to a generator function or generator expression, which you could call a "protocol" but if so it's not one expressible at the level of the language.

I think that leads to a bit of confusion when speaking loosely. When someone says "the generator protocol vs. the iterator protocol" the "obviously correct" meaning is send and throw, but it's not what people always mean.

Then again, the word "generator" itself leads to confusion when speaking loosely. Maybe it would be clearer if "generator" had no meaning; generator functions return generator iterators. But I don't think this confusion has caused serious problems over the decades, so I doubt the more minor confusion at issue here is likely to be serious.


From antony.lee at berkeley.edu  Fri Nov 21 18:41:31 2014
From: antony.lee at berkeley.edu (Antony Lee)
Date: Fri, 21 Nov 2014 09:41:31 -0800
Subject: [Python-ideas] Return expressions
In-Reply-To: <CAPTjJmosQVXr4qbbewYAuQRtYKv-LpADRKfy2XukN5dhyt5p7A@mail.gmail.com>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
 <546DE48C.5000805@mrabarnett.plus.com>
 <CADiSq7cWybgFx7YtP+xYin3wLLX3kh3gjQxghUh7jqc=LVs8dQ@mail.gmail.com>
 <CA+=+wqCN4Q+poFEkSwRyXsx1eHTpeJjPwqQiSKz5BfkoyZReYw@mail.gmail.com>
 <546E0DA1.10609@mrabarnett.plus.com>
 <CADiSq7d1E3fXq6ShBUybHOcHzLzdzqoXV-BSzzaMvLXo09-2wg@mail.gmail.com>
 <CEE296EB-7DE2-4EF6-9831-A0FEB0F47FB7@yahoo.com>
 <546F69F4.9000301@stoneleaf.us>
 <CAPTjJmosQVXr4qbbewYAuQRtYKv-LpADRKfy2XukN5dhyt5p7A@mail.gmail.com>
Message-ID: <CAGRr6BFn9imDjyMDgveD+n18sHb9nSzKTCww37yMuAh83tju9g@mail.gmail.com>

I would like to believe that "break"-as-a-expression solves most of these
issues.  It is reasonable, though, to drop the "return"-as-an-expression
part of the proposal, because you need to know that the comprehension is
run in a separate function to understand it (it's a bit ironic that I am
dropping the original proposal to defend my own, now...).

Consider

((x, y) for x in l1 if f1(x) or break for y in l2 if f2(y) or break)

This maps directly to

for x in l1:
    if f1(x) or break:
        for y in l2:
            if f2(y) or break:
                yield x, y

which is literally a copy-paste of the second part followed by yielding the
first part.  I think this reads reasonably well but this is obviously a
subjective issue.

Antony

2014-11-21 8:49 GMT-08:00 Chris Angelico <rosuav at gmail.com>:

> On Sat, Nov 22, 2014 at 3:36 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> > On 11/21/2014 08:22 AM, Andrew Barnert wrote:
> >>
> >> Maybe what we need to do is update PEP 3142, gathering all of the
> alternative ideas,
> >> and expanding on the rationale, so Guido can reject that, and next time
> this comes
> >> up, everyone will have something to read before having the same
> arguments about the
> >> same proposals?
> >
> > Are we accepting nominations?  I have the perfect fellow in mind.  :)
> >
> > (And in case Chris decides to nominate me in retaliation, I decline. ;)
>
> Heh! Great, I'm going to get a name for writing the PEPs that exist
> solely to be rejected... I was hoping PEP 479 would be accepted
> easily, given who was advocating it, but now it's starting to look
> like I doomed the proposal :)
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141121/6bddb23d/attachment.html>

From abarnert at yahoo.com  Fri Nov 21 19:06:46 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 21 Nov 2014 10:06:46 -0800
Subject: [Python-ideas] Return expressions
In-Reply-To: <CAGRr6BFn9imDjyMDgveD+n18sHb9nSzKTCww37yMuAh83tju9g@mail.gmail.com>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
 <546DE48C.5000805@mrabarnett.plus.com>
 <CADiSq7cWybgFx7YtP+xYin3wLLX3kh3gjQxghUh7jqc=LVs8dQ@mail.gmail.com>
 <CA+=+wqCN4Q+poFEkSwRyXsx1eHTpeJjPwqQiSKz5BfkoyZReYw@mail.gmail.com>
 <546E0DA1.10609@mrabarnett.plus.com>
 <CADiSq7d1E3fXq6ShBUybHOcHzLzdzqoXV-BSzzaMvLXo09-2wg@mail.gmail.com>
 <CEE296EB-7DE2-4EF6-9831-A0FEB0F47FB7@yahoo.com>
 <546F69F4.9000301@stoneleaf.us>
 <CAPTjJmosQVXr4qbbewYAuQRtYKv-LpADRKfy2XukN5dhyt5p7A@mail.gmail.com>
 <CAGRr6BFn9imDjyMDgveD+n18sHb9nSzKTCww37yMuAh83tju9g@mail.gmail.com>
Message-ID: <C6F16739-2AA1-475B-8CF9-61042352F343@yahoo.com>

On Nov 21, 2014, at 9:41, Antony Lee <antony.lee at berkeley.edu> wrote:

> I would like to believe that "break"-as-a-expression solves most of these issues. 

This one has been raised every time in the past. Since we don't have the PEP, I'll try to summarize the problems. But first:

> It is reasonable, though, to drop the "return"-as-an-expression part of the proposal, because you need to know that the comprehension is run in a separate function to understand it (it's a bit ironic that I am dropping the original proposal to defend my own, now...).

There's a more significant difference between the two proposals. A break expression allows you to break out of a single loop, but gives you no way to break out of the whole thing (as your example shows). A return expression allows you to break out of the whole thing, but gives you no way to break out of a single loop. That's why they're not interchangeable in explicit loop code. (Also, notice that every so often, someone proposed a numbered or labeled break, and the answer is always "Why would you need that? If your code has enough loops that you need to break out of some but not all, refactor those some into a separate function and then use return.")

This also raises the question of why one of break/continue/return should be an expression but not the others. If your answer is "because you only need continue when there's code not controlled by the if, which is impossible in a comprehension" then you're pretty much admitting that the flow control expression thing is not really a general purpose thing, but a hack made for comprehensions.

> Consider
> 
> ((x, y) for x in l1 if f1(x) or break for y in l2 if f2(y) or break)
> 
> This maps directly to
> 
> for x in l1:
>     if f1(x) or break:
>         for y in l2:
>             if f2(y) or break:
>                 yield x, y

When you write it this way, it's pretty clear that you're abusing or as an else, and that it comes in the wrong place, and that you're cramming side effects into an expression with no value. Would you put anything else with side effects here, even a call to a logging function?

You're also introducing unnecessary indentation, and making the code more verbose by comparison with the obvious alternative:

for x in l1:
    if not f1(x): break
    for y in l2:
        if not f2(y): break
        yield x, y

Yes, that alternative can't be written as a comprehension. But that doesn't mean we should come up with a less obvious, more verbose, and harder to reason about alternative just because it can be written as a comprehension, even though we'd never write it as an explicit loop. If you want to go that route, we might as well just enshrine the "or stop()" hack instead of trying to break it.

Also, can you imagine using break as an expression in any other context besides as an or operand in an if statement directly under a for statement?



From chris.barker at noaa.gov  Fri Nov 21 18:51:41 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 21 Nov 2014 09:51:41 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <592A1ABA-41FE-4643-B6B2-CFD733982EB6@yahoo.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info> <546F6DEB.50906@stoneleaf.us>
 <592A1ABA-41FE-4643-B6B2-CFD733982EB6@yahoo.com>
Message-ID: <CALGmxEJoVRVN1t4eQvTGjSsRecZVamgYaM9=BqSM-GnttVXvBg@mail.gmail.com>

On Fri, Nov 21, 2014 at 9:18 AM, Andrew Barnert <
abarnert at yahoo.com.dmarc.invalid> wrote:

> Generators are a subtype of iterators. They support the iterator protocol
> completely, and in the same way as any other iterator. They also support
> extensions to that protocol--e.g., send(). But they also have a
> relationship to a generator function or generator expression,
>

interesting -- I've always called those "generator comprehensions" -- but
anyway, -- do they have a special relationship?

I can put any iterable in a generator expression:

gen_exp = (i for i in [3,4,5,6])

the result is a generator:

In [5]: type(gen_exp)
Out[5]: generator

so I guess you could call that a "special relationship" -- but it looks to
me kind of like an alternate constructor.

But in any case, you can use a generator created by a generator expression
or a generator function the same way you can use a iterable or an iterator
class.

Then again, the word "generator" itself leads to confusion when speaking
> loosely. Maybe it would be clearer if "generator" had no meaning; generator
> functions return generator iterators.


not sure how that would help -- a generator is a type, and it is created by
either calling a generator function or a generator expression.

if there is confusion, it's when folks call a generator function a
"generator"

Anyway, I just went back and read the PEP, and I'm still confused -- would
the PEP make generators behave  more like iterator classes, or less like
them?

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141121/fe4fa18d/attachment-0001.html>

From jeanpierreda at gmail.com  Fri Nov 21 19:25:14 2014
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Fri, 21 Nov 2014 10:25:14 -0800
Subject: [Python-ideas] Return expressions
In-Reply-To: <CAGRr6BFn9imDjyMDgveD+n18sHb9nSzKTCww37yMuAh83tju9g@mail.gmail.com>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
 <546DE48C.5000805@mrabarnett.plus.com>
 <CADiSq7cWybgFx7YtP+xYin3wLLX3kh3gjQxghUh7jqc=LVs8dQ@mail.gmail.com>
 <CA+=+wqCN4Q+poFEkSwRyXsx1eHTpeJjPwqQiSKz5BfkoyZReYw@mail.gmail.com>
 <546E0DA1.10609@mrabarnett.plus.com>
 <CADiSq7d1E3fXq6ShBUybHOcHzLzdzqoXV-BSzzaMvLXo09-2wg@mail.gmail.com>
 <CEE296EB-7DE2-4EF6-9831-A0FEB0F47FB7@yahoo.com>
 <546F69F4.9000301@stoneleaf.us>
 <CAPTjJmosQVXr4qbbewYAuQRtYKv-LpADRKfy2XukN5dhyt5p7A@mail.gmail.com>
 <CAGRr6BFn9imDjyMDgveD+n18sHb9nSzKTCww37yMuAh83tju9g@mail.gmail.com>
Message-ID: <CABicbJJN7FB7zpS0cY-S-gDwVWiV=SBAd-mhDwbEV2Jay5k3VA@mail.gmail.com>

On Fri, Nov 21, 2014 at 9:41 AM, Antony Lee <antony.lee at berkeley.edu> wrote:
> I would like to believe that "break"-as-a-expression solves most of these
> issues.  It is reasonable, though, to drop the "return"-as-an-expression
> part of the proposal, because you need to know that the comprehension is run
> in a separate function to understand it (it's a bit ironic that I am
> dropping the original proposal to defend my own, now...).
>
> Consider
>
> ((x, y) for x in l1 if f1(x) or break for y in l2 if f2(y) or break)
>
> This maps directly to
>
> for x in l1:
>     if f1(x) or break:
>         for y in l2:
>             if f2(y) or break:
>                 yield x, y
>
> which is literally a copy-paste of the second part followed by yielding the
> first part.  I think this reads reasonably well but this is obviously a
> subjective issue.

I really hope "or break" doesn't become an idiom if break is turned
into an expression. I find [x if x < N else break for x in ...] so
much more readable than [x for x in ... if x < N or break]

This is very near (but not directly equivalent) to the relatively idiomatic:

for x in ...:
    if x < N:
        yield x
    else:
        break

-- Devin

From antony.lee at berkeley.edu  Fri Nov 21 19:26:06 2014
From: antony.lee at berkeley.edu (Antony Lee)
Date: Fri, 21 Nov 2014 10:26:06 -0800
Subject: [Python-ideas] Return expressions
In-Reply-To: <C6F16739-2AA1-475B-8CF9-61042352F343@yahoo.com>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
 <546DE48C.5000805@mrabarnett.plus.com>
 <CADiSq7cWybgFx7YtP+xYin3wLLX3kh3gjQxghUh7jqc=LVs8dQ@mail.gmail.com>
 <CA+=+wqCN4Q+poFEkSwRyXsx1eHTpeJjPwqQiSKz5BfkoyZReYw@mail.gmail.com>
 <546E0DA1.10609@mrabarnett.plus.com>
 <CADiSq7d1E3fXq6ShBUybHOcHzLzdzqoXV-BSzzaMvLXo09-2wg@mail.gmail.com>
 <CEE296EB-7DE2-4EF6-9831-A0FEB0F47FB7@yahoo.com>
 <546F69F4.9000301@stoneleaf.us>
 <CAPTjJmosQVXr4qbbewYAuQRtYKv-LpADRKfy2XukN5dhyt5p7A@mail.gmail.com>
 <CAGRr6BFn9imDjyMDgveD+n18sHb9nSzKTCww37yMuAh83tju9g@mail.gmail.com>
 <C6F16739-2AA1-475B-8CF9-61042352F343@yahoo.com>
Message-ID: <CAGRr6BEEHn1NUchuPmLbb-fYxfEk+cS8gGw7uaU7tKR4JccvBg@mail.gmail.com>

2014-11-21 10:06 GMT-08:00 Andrew Barnert <abarnert at yahoo.com>:

> On Nov 21, 2014, at 9:41, Antony Lee <antony.lee at berkeley.edu> wrote:
>
> > I would like to believe that "break"-as-a-expression solves most of
> these issues.
>
> This one has been raised every time in the past. Since we don't have the
> PEP, I'll try to summarize the problems. But first:
>
> > It is reasonable, though, to drop the "return"-as-an-expression part of
> the proposal, because you need to know that the comprehension is run in a
> separate function to understand it (it's a bit ironic that I am dropping
> the original proposal to defend my own, now...).
>
> There's a more significant difference between the two proposals. A break
> expression allows you to break out of a single loop, but gives you no way
> to break out of the whole thing (as your example shows). A return
> expression allows you to break out of the whole thing, but gives you no way
> to break out of a single loop. That's why they're not interchangeable in
> explicit loop code. (Also, notice that every so often, someone proposed a
> numbered or labeled break, and the answer is always "Why would you need
> that? If your code has enough loops that you need to break out of some but
> not all, refactor those some into a separate function and then use return.")
>
> This also raises the question of why one of break/continue/return should
> be an expression but not the others. If your answer is "because you only
> need continue when there's code not controlled by the if, which is
> impossible in a comprehension" then you're pretty much admitting that the
> flow control expression thing is not really a general purpose thing, but a
> hack made for comprehensions.
>

 I guess you're going to make me flip-flop and go back to defend having all
three of "break", "return" and "raise" as expressions (I can't see a single
use of "continue" as an expression but would be happy to be proven wrong).
The issue of "return" as an expression is obviously that it exposes the
internals of generators (i.e. they are in their own function) but I can
live with that.  ("raise" as expressions is a separate issue, the main
application being to stuff it into lambdas.)

>
> > Consider
> >
> > ((x, y) for x in l1 if f1(x) or break for y in l2 if f2(y) or break)
> >
> > This maps directly to
> >
> > for x in l1:
> >     if f1(x) or break:
> >         for y in l2:
> >             if f2(y) or break:
> >                 yield x, y
>
> When you write it this way, it's pretty clear that you're abusing or as an
> else, and that it comes in the wrong place, and that you're cramming side
> effects into an expression with no value. Would you put anything else with
> side effects here, even a call to a logging function?
>
> You're also introducing unnecessary indentation, and making the code more
> verbose by comparison with the obvious alternative:
>
> for x in l1:
>     if not f1(x): break
>     for y in l2:
>         if not f2(y): break
>         yield x, y
>
> Yes, that alternative can't be written as a comprehension. But that
> doesn't mean we should come up with a less obvious, more verbose, and
> harder to reason about alternative just because it can be written as a
> comprehension, even though we'd never write it as an explicit loop. If you
> want to go that route, we might as well just enshrine the "or stop()" hack
> instead of trying to break it.
>

 The introduction of unnecessary indentation is honestly not there to make
the non-generator example look bad, but simply to show how the proposal
addresses the issue of translating nested loops.

For the non-nested loop case, a simpler and arguably more readable way to
write it would be

(x if f(x) else break for x in X)

Perhaps giving the nested case first was poor PR; yes, the nested case is
slightly hackish but I am fairly sure anybody can figure out the meaning of
the above.

>
> Also, can you imagine using break as an expression in any other context
> besides as an or operand in an if statement directly under a for statement?
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141121/47ea8f9a/attachment.html>

From rosuav at gmail.com  Fri Nov 21 19:28:58 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 22 Nov 2014 05:28:58 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CALGmxEJoVRVN1t4eQvTGjSsRecZVamgYaM9=BqSM-GnttVXvBg@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info>
 <546F6DEB.50906@stoneleaf.us>
 <592A1ABA-41FE-4643-B6B2-CFD733982EB6@yahoo.com>
 <CALGmxEJoVRVN1t4eQvTGjSsRecZVamgYaM9=BqSM-GnttVXvBg@mail.gmail.com>
Message-ID: <CAPTjJmr27PTa2Tz1ZUhcuG-Cwbpux0HzDwm9Vv1qKW7=4TUFyQ@mail.gmail.com>

On Sat, Nov 22, 2014 at 4:51 AM, Chris Barker <chris.barker at noaa.gov> wrote:
> Anyway, I just went back and read the PEP, and I'm still confused -- would
> the PEP make generators behave  more like iterator classes, or less like
> them?

Neutral. A generator function, an iterator class, etc, etc, etc,
exists solely to construct an iterator. That iterator has a __next__
method, which either returns a value or raises StopIteration, or
raises some other exception (which bubbles up).

There are two easy ways to write iterators. One is to construct a class:

class Iter:
    def __init__(self): self.x = 0
    def __iter__(self): return self
    def __next__(self):
        if self.x == 3: raise StopIteration
        self.x += 1
        return self.x

Another is to write a generator function:

def gen():
    yield 1
    yield 2
    yield 3

Both Iter and gen are callables which return iterators. Both of them
will produce three integers and then raise StopIteration. Both will,
as is good form for iterators, continue to raise StopIteration
thereafter.

And neither Iter nor gen is, itself, an iterator. One is a class which
constructs iterators. The other is a generator function, which also
constructs iterators. That's all. In Iter.__next__, I wrote code which
chose between "return" and "raise StopIteration" to define its result;
in gen(), I wrote code which chose between "yield" and "return" (in
this case, the implicit return at the end of the function) to define
its result.

The only change made by this proposal is that StopIteration becomes,
in a generator, like any other unexpected exception. It creates a
separation between "iterator protocol" (which is implemented by
__next__) and "generator protocol" (which is written in the body of a
function with 'yield' in it).

ChrisA

From ethan at stoneleaf.us  Fri Nov 21 18:27:03 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 21 Nov 2014 09:27:03 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
 generators
In-Reply-To: <20141121171622.GP2748@ando.pearwood.info>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info> <546F6DEB.50906@stoneleaf.us>
 <20141121171622.GP2748@ando.pearwood.info>
Message-ID: <546F75E7.9040409@stoneleaf.us>

On 11/21/2014 09:16 AM, Steven D'Aprano wrote:
> On Fri, Nov 21, 2014 at 08:52:59AM -0800, Ethan Furman wrote:
>> On 11/21/2014 08:30 AM, Steven D'Aprano wrote:
>>>
>>> But generators and iterators *are the same thing*. (Generator functions 
>>> are not iterators, but generators themselves are.) Iterators don't have 
>>> a specific type, but they obey the iterator protocol:
>>
>> Um, no, they aren't.  One cannot 'send()' into any ol' iterator.
> 
> "Must not support send()" has never been part of the definition of 
> iterators.

I suspect a language use issue: *are the same thing* implies to me a type check, not an isinstance check.

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141121/a7111216/attachment-0001.sig>

From abarnert at yahoo.com  Fri Nov 21 19:48:05 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 21 Nov 2014 10:48:05 -0800
Subject: [Python-ideas] Return expressions
In-Reply-To: <546F69F4.9000301@stoneleaf.us>
References: <CADiSq7dgTqV=uakgWEzhC3b+B69uF9c-vD9B7_9c1uZvmv7oQQ@mail.gmail.com>
 <546DE48C.5000805@mrabarnett.plus.com>
 <CADiSq7cWybgFx7YtP+xYin3wLLX3kh3gjQxghUh7jqc=LVs8dQ@mail.gmail.com>
 <CA+=+wqCN4Q+poFEkSwRyXsx1eHTpeJjPwqQiSKz5BfkoyZReYw@mail.gmail.com>
 <546E0DA1.10609@mrabarnett.plus.com>
 <CADiSq7d1E3fXq6ShBUybHOcHzLzdzqoXV-BSzzaMvLXo09-2wg@mail.gmail.com>
 <CEE296EB-7DE2-4EF6-9831-A0FEB0F47FB7@yahoo.com>
 <546F69F4.9000301@stoneleaf.us>
Message-ID: <B17415AD-9922-4675-AA4E-C4664F1BEE38@yahoo.com>

If someone _is_ going to do this, I have one more suggestion that I didn't raise last time around, because I don't want to argue for it, but for completeness:

Briefly: add if and while clauses to the for statement; redefine comprehensions as nested for loops.

The basic problem is that Python comprehensions map to arbitrarily nested for and if statements, and there's no way to add "while" semantics cleanly with another nested statement.

If you look at the languages that have while clauses, this is not how their comprehensions work.

Some of them, they have nested loops only, but each loop has optional modifiers that Python's don't: an if or when clause that filters, a while clause that terminates early, maybe unless and until clauses that do the same but with negated conditions (although until has a choice between doing the last iteration or not doing it). This allows for everything you can do in Python except multiple if clauses on the same for clause (which are usually better rewritten with and--or, when that's too unwieldy, you probably shouldn't have been writing it as a one-liner).

In others, there's a single loop, but it takes nested iterable expressions, so you only get one set of modifiers for all of the iterables.

Others just don't have any syntax for nested iteration at all; you have to turn it upside-down and use the outer loop as the iterable for the inner loop.

This would have another minor arguable benefit. Every once in a while I see some code like this:

    for x in (x for x in xs if f(x)):

... because the writer didn't think long enough to realize he doesn't need the comprehension. This code is obviously a lot better as:

    for x in xs if f(x):

But that example shows exactly why I don't think this is worth arguing for. The same code is even better as:

    for x in xs:
        if f(x):

And that brings us back to the reason Clojure, Racket, etc. have clauses on their for loops: Because they don't have loop statements, they have a loop function. (Well, usually it's a macro that quotes the arguments so you don't have to lambda all your expressions into functions, and to allow you to avoid at least one excess repetition, but let's ignore that.) The when and while clauses are just keyword arguments to the function. More generally, they encourage or require you to write everything as a complex expression, which is the exact opposite of what Python encourages.

Still, just because the idea comes from languages that are very different from Python doesn't necessarily mean it wouldn't fit; after all, that's how we got comprehensions in the first place.

Sent from a random iPhone

On Nov 21, 2014, at 8:36, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 11/21/2014 08:22 AM, Andrew Barnert wrote:
>> 
>> Maybe what we need to do is update PEP 3142, gathering all of the alternative ideas,
>> and expanding on the rationale, so Guido can reject that, and next time this comes
>> up, everyone will have something to read before having the same arguments about the
>> same proposals?
> 
> Are we accepting nominations?  I have the perfect fellow in mind.  :)
> 
> (And in case Chris decides to nominate me in retaliation, I decline. ;)
> 
> --
> ~Ethan~
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From chris.barker at noaa.gov  Fri Nov 21 20:33:03 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 21 Nov 2014 11:33:03 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
Message-ID: <CALGmxE+=F09nr=THwfViX4TtDDMZo0H64xOSnePdp2rEj_zAXg@mail.gmail.com>

I htink I'm getting closer to clarity here, but:

let's say I have this silly  helper function (similar to the one in the
PEP):

## (sorry -- python2)

def helper(x):
    if x > 2:
        raise StopIteration
    else:
        return x

# I call that helper from both an iterator class and a generator function.

class Iter(object):
    def __init__(self): self.x = 0
    def __iter__(self): return self
    def next(self):
        if self.x == 3: raise StopIteration
        self.x += 1
        return helper(self.x)


def gen():
    yield helper(1)
    yield helper(2)
    yield helper(3)

# first just loop through them:

for i in Iter():
    print i

for i in gen():
    print i

# they act exactly the same

# then put them inside generator expression and loop through that:

for j in ( i for i in gen() ):
    print j

for j in ( i for i in Iter() ):
    print j

# they still act exactly the same.

Would this PEP chance that? would that StopIteration bubble up through the
iterator class, but not through the generator function? If so then I'm -1
on the PEP.

The only change made by this proposal is that StopIteration becomes,
> in a generator, like any other unexpected exception.


but from the user's perspective, an iterator and a generator should look
the same.

And I think from the perspective of the author of a given generator
function, they should look as much the same as possible-- i.e. if a
StopIteration is raised internally, it will behave the same way as it would
in a iterator class.

It creates a
> separation between "iterator protocol" (which is implemented by
> __next__) and "generator protocol" (which is written in the body of a
> function with 'yield' in it).
>

terminology messing me up here -- I understood the "iterator protocol" top
mean the __iter__ and __next__ -- and a generator satisfies that protocol.
Is there a "generator protocol" that means somethign different?

Maybe that's what this whole sub-thread is about:

I never thought there was such a thing as a "generator protocol", and I
don't see why there should be.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141121/a4fbcce3/attachment.html>

From rosuav at gmail.com  Fri Nov 21 22:55:53 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 22 Nov 2014 08:55:53 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CALGmxE+=F09nr=THwfViX4TtDDMZo0H64xOSnePdp2rEj_zAXg@mail.gmail.com>
References: <CALGmxE+=F09nr=THwfViX4TtDDMZo0H64xOSnePdp2rEj_zAXg@mail.gmail.com>
Message-ID: <CAPTjJmqiD0Agm-29GbJjDU_T9SF2yzZ476dud20AKU+hF1oTjw@mail.gmail.com>

On Sat, Nov 22, 2014 at 6:33 AM, Chris Barker <chris.barker at noaa.gov> wrote:
> And I think from the perspective of the author of a given generator
> function, they should look as much the same as possible-- i.e. if a
> StopIteration is raised internally, it will behave the same way as it would
> in a iterator class.

Ah, but why? When you write a function like this:

def gen():
    yield 1
    yield 2

why should StopIteration mean any more to it than, say, KeyError does?

>> It creates a
>> separation between "iterator protocol" (which is implemented by
>> __next__) and "generator protocol" (which is written in the body of a
>> function with 'yield' in it).
>
>
> terminology messing me up here -- I understood the "iterator protocol" top
> mean the __iter__ and __next__ -- and a generator satisfies that protocol.
> Is there a "generator protocol" that means somethign different?

A generator *object* does. A generator *function* doesn't - it follows
a protocol that consists of the "yield" and "return" statements.

ChrisA

From chris.barker at noaa.gov  Fri Nov 21 17:53:19 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 21 Nov 2014 08:53:19 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <20141121163032.GN2748@ando.pearwood.info>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info>
Message-ID: <CALGmxEJHxSb-z4YQdci_HUFx42MxE3ooB8JHe+W_uwCmpPPsBQ@mail.gmail.com>

As someone who has written maybe one generator expression in production
code, I have little opinion on the PEP.

But as someone that teaches Python, I have a comment on:

On Fri, Nov 21, 2014 at 10:50:52PM +1100, Chris Angelico wrote:
> > Yes, this would be affected. This proposal causes a separation of
> > generators and iterators, so it's no longer possible to pretend that
> > they're the same thing.
>

 As pointed out by Steven, the _are_ the same thing. When I teach
interators and generators, I get a bit tangled up explaining what the
difference is, and why Python has both. This is what I say:

Conceptually ( outside of language constructs):

An "generator" is something that, well, generates value on the fly, as
requested, until there are no more to generate, and then terminates.

A "iterator" on the other had is something that produces the values in a
pre-existing sequence of values, until there are no more.

IN practice, python uses the exact same protocol (the iterator protocol --
__iter__, __next__) for both, so that you can write, e.g. a for loop, and
not have to know whether the underlying object you are looping through is
iterating or generating...

As you can write a "generator" in the sense above in a class that supports
the iterator protocol (and, can, in fact, write an "iterator" with a
generator function), then I say that generator functions really are only
syntactic sugar -- they are short and sweet and do much of the book keeping
for you.

But given all that keeping the protocols as similar as possible is a *good*
thing, not a bad one -- they should behave as much as possible teh same.

If StopIteration bubbles up from inside an iterator, wouldn't that silently
terminate as well?

Honestly, I'm a bit lost -- but my point is this -- generators and
iterators should behave as much the same as possible.

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141121/608bfe8c/attachment-0001.html>

From rosuav at gmail.com  Fri Nov 21 23:09:07 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 22 Nov 2014 09:09:07 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CALGmxEJHxSb-z4YQdci_HUFx42MxE3ooB8JHe+W_uwCmpPPsBQ@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info>
 <CALGmxEJHxSb-z4YQdci_HUFx42MxE3ooB8JHe+W_uwCmpPPsBQ@mail.gmail.com>
Message-ID: <CAPTjJmrDfS=XFOX0wW6Kk7kWb-VdkN2AoiWYyOE2S6YqeguSjA@mail.gmail.com>

On Sat, Nov 22, 2014 at 3:53 AM, Chris Barker <chris.barker at noaa.gov> wrote:
> As you can write a "generator" in the sense above in a class that supports
> the iterator protocol (and, can, in fact, write an "iterator" with a
> generator function), then I say that generator functions really are only
> syntactic sugar -- they are short and sweet and do much of the book keeping
> for you.

If you want to consider them that way, then sure - but part of the
bookkeeping they do for you is the management of the StopIteration
exception. That becomes purely an implementation detail. You don't
actually use it when you write a generator function.

ChrisA

From guido at python.org  Fri Nov 21 23:29:27 2014
From: guido at python.org (Guido van Rossum)
Date: Fri, 21 Nov 2014 14:29:27 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CALGmxEJHxSb-z4YQdci_HUFx42MxE3ooB8JHe+W_uwCmpPPsBQ@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info>
 <CALGmxEJHxSb-z4YQdci_HUFx42MxE3ooB8JHe+W_uwCmpPPsBQ@mail.gmail.com>
Message-ID: <CAP7+vJ+fLsNm9sVdtRRxwLejoCu61V5TTFAJ1FiA8p+p2rLzrw@mail.gmail.com>

On Fri, Nov 21, 2014 at 8:53 AM, Chris Barker <chris.barker at noaa.gov> wrote:

> As someone who has written maybe one generator expression in production
> code, I have little opinion on the PEP.
>
> But as someone that teaches Python, I have a comment on:
>
> On Fri, Nov 21, 2014 at 10:50:52PM +1100, Chris Angelico wrote:
>> > Yes, this would be affected. This proposal causes a separation of
>> > generators and iterators, so it's no longer possible to pretend that
>> > they're the same thing.
>>
>
>  As pointed out by Steven, the _are_ the same thing. When I teach
> interators and generators, I get a bit tangled up explaining what the
> difference is, and why Python has both. This is what I say:
>
> Conceptually ( outside of language constructs):
>
> An "generator" is something that, well, generates value on the fly, as
> requested, until there are no more to generate, and then terminates.
>
> A "iterator" on the other had is something that produces the values in a
> pre-existing sequence of values, until there are no more.
>
> IN practice, python uses the exact same protocol (the iterator protocol --
> __iter__, __next__) for both, so that you can write, e.g. a for loop, and
> not have to know whether the underlying object you are looping through is
> iterating or generating...
>
> As you can write a "generator" in the sense above in a class that supports
> the iterator protocol (and, can, in fact, write an "iterator" with a
> generator function), then I say that generator functions really are only
> syntactic sugar -- they are short and sweet and do much of the book keeping
> for you.
>
> But given all that keeping the protocols as similar as possible is a
> *good* thing, not a bad one -- they should behave as much as possible teh
> same.
>
> If StopIteration bubbles up from inside an iterator, wouldn't that
> silently terminate as well?
>
> Honestly, I'm a bit lost -- but my point is this -- generators and
> iterators should behave as much the same as possible.
>

I'm sorry you see it that way; we must have done a terrible job explaining
this in the past. :-(

The behavior for the *consumer* of the iteration is unchanged (call next()
until it raises StopIteration -- or let a for-loop take care of the details
for you). The interface for the *producer* has never been all that similar:
In a generator you *yield* subsequent values until you are done; but if you
are not using a generator, you must define a __next__() method (next() in
Python 2) that *returns* a single value each time, until it's done, and
then it has to raise StopIteration. There is no need to raise StopIteration
from a generator, you just return when you are done. Insisting that raising
StopIteration in a generator makes it more similar to a __next__() method
ignores the fact that producing values is done in a completely different
ways.

So, again, the PEP does not change anything about iterators, and generators
will continue to follow the iterator protocol. The change is only for
generator authors (and, most importantly, for people using a certain hack
in generator expressions).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141121/a9291ff4/attachment.html>

From chris.barker at noaa.gov  Sat Nov 22 00:06:10 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 21 Nov 2014 15:06:10 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAP7+vJ+fLsNm9sVdtRRxwLejoCu61V5TTFAJ1FiA8p+p2rLzrw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info>
 <CALGmxEJHxSb-z4YQdci_HUFx42MxE3ooB8JHe+W_uwCmpPPsBQ@mail.gmail.com>
 <CAP7+vJ+fLsNm9sVdtRRxwLejoCu61V5TTFAJ1FiA8p+p2rLzrw@mail.gmail.com>
Message-ID: <CALGmxELgihRDMD=5_xeW7CWwg_7HL9BXwUW+JH-A5VLyg=TJTA@mail.gmail.com>

On Fri, Nov 21, 2014 at 2:29 PM, Guido van Rossum <guido at python.org> wrote:

> Honestly, I'm a bit lost -- but my point is this -- generators and
>> iterators should behave as much the same as possible.
>>
>
> I'm sorry you see it that way; we must have done a terrible job explaining
> this in the past. :-(
>

well, others have found examples in old docs that mingle StopIteration and
generators...so I guess so, but I'm not sure I'm that misinformed. It still
seems to me that there are two ways to write the same thing.

The behavior for the *consumer* of the iteration is unchanged
>

got it -- the issue at hand is what happens to a StopIteration that is
raised by something the generator calls.

I think the point of this PEP is that the author og a generator function is
thinking about using "yield" to provide the next value, and return
(explicit or implicit) to stop the generation of objects. That return is
raise a StopIteration, but the author isn't thinking about it.

So why would they want to think about having to trap StopIteration when
calling other functions.

While the author of a iterator class is thinking about the __next__ method
and raising a StopIteration to terminate. So s/he would naturally think
about trapping StopIteration when calling functions?

I suppose that makes some sense, but to me it seems like a generator
function is a different syntax for creating what is essentially the same
thing -- why shouldn't it have the same behavior?

and of you are writing a generator, presumably you know how it's going to
get use -- i.e. by somethign that expects a StopIteration -- it's not like
you're ignorant of the whole idea.

Consider this far fetched situation:

Either a iterator class or a generator function could take a function
object to call to do part of its work. If that function happened to raise a
StopIteration -- now the user would have to know which type of object they
were workign with, so they would know how to handle the termination of the
iter/gener-artion

OK -- far more far fetched than the proceeding example of confusion, but
the point is this:

AFAIU, the current distinction between generators and iterators is how they
are written -- i.e. syntax, essentially. But this PEP would change the
behavior of generators in some small way, creating a distinction that
doesn't currently exist.

So, again, the PEP does not change anything about iterators, and generators
> will continue to follow the iterator protocol. The change is only for
> generator authors
>

I guess this is where I'm not sure -- it seems to me that the behavior of
generators is being change, not the syntax -- so while mostly of concern to
generator authors, it is, in fact, a chance in behavior that can be seen by
the consumer of (maybe only an oddly designed) generator. In practice, that
difference may only matter to folks using that particular hack in generator
expression, but it is indeed a change.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141121/604eb301/attachment-0001.html>

From rosuav at gmail.com  Sat Nov 22 00:40:37 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 22 Nov 2014 10:40:37 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CALGmxELgihRDMD=5_xeW7CWwg_7HL9BXwUW+JH-A5VLyg=TJTA@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info>
 <CALGmxEJHxSb-z4YQdci_HUFx42MxE3ooB8JHe+W_uwCmpPPsBQ@mail.gmail.com>
 <CAP7+vJ+fLsNm9sVdtRRxwLejoCu61V5TTFAJ1FiA8p+p2rLzrw@mail.gmail.com>
 <CALGmxELgihRDMD=5_xeW7CWwg_7HL9BXwUW+JH-A5VLyg=TJTA@mail.gmail.com>
Message-ID: <CAPTjJmo-YEtZ0OOpNLf3X1N3y5tH5rp+T4_ybHwbaWBYDQjAUQ@mail.gmail.com>

On Sat, Nov 22, 2014 at 10:06 AM, Chris Barker <chris.barker at noaa.gov> wrote:
> I think the point of this PEP is that the author og a generator function is
> thinking about using "yield" to provide the next value, and return (explicit
> or implicit) to stop the generation of objects. That return is raise a
> StopIteration, but the author isn't thinking about it.
>
> So why would they want to think about having to trap StopIteration when
> calling other functions.
>
> While the author of a iterator class is thinking about the __next__ method
> and raising a StopIteration to terminate. So s/he would naturally think
> about trapping StopIteration when calling functions?
>
> I suppose that makes some sense, but to me it seems like a generator
> function is a different syntax for creating what is essentially the same
> thing -- why shouldn't it have the same behavior?

Let's suppose you use a Python-like macro language to generate Python
code. In the macro language, you can write stuff like this:

class X:
    iterator:
        return 5
        return 6
        return 7
        iterator_done

And it will get compiled into something like this:

class X:
    def __init__(self): self._iter_state = 0
    def __iter__(self): return self
    def __next__(self):
        self._iter_state += 1
        if self._iter_state == 1: return 5
        if self._iter_state == 2: return 6
        if self._iter_state == 3: return 7
        raise StopIteration

This is a reasonably plausible macro language, right? It's basically
still a class definition, but it lets you leave out a whole bunch of
boilerplate. Now, the question is: As you write the simplified
version, should you ever need to concern yourself with StopIteration?
I posit no, you should not; it's not a part of the macro language at
all. Of course, if this *were* how things were done, it would probably
be implemented as a very thin wrapper, exposing all its details to
your code; but there's no reason that it *needs* to be so thin. The
language you're writing in doesn't need to have any concept of a
StopIteration exception, because it doesn't need to use an exception
to signal "no more results".

> and of you are writing a generator, presumably you know how it's going to
> get use -- i.e. by somethign that expects a StopIteration -- it's not like
> you're ignorant of the whole idea.

Not necessarily. Can we get someone here who knows asyncio and
coroutines, and can comment on the use of such generators?

> Consider this far fetched situation:
>
> Either a iterator class or a generator function could take a function object
> to call to do part of its work. If that function happened to raise a
> StopIteration -- now the user would have to know which type of object they
> were workign with, so they would know how to handle the termination of the
> iter/gener-artion

Either a __getattr__ or a __getitem__ could use a helper function to
do part of its work, too, but either the helper needs to know which,
or it needs some other way of signalling. They're different protocols,
so they're handled differently. If Python wanted to conflate all of
these, there could be a single "NoReturnValue" exception, used by
every function which needs to be able to return absolutely any object
and also to be able to signal "I don't have anything to return". But
no, Python has separate exceptions for signalling "I don't have any
such key", "I don't have any such attribute", and "I don't have any
more things to iterate over". Generators don't need any of them,
because - like my macro language above - they have two different
keywords and two different byte-codes (yield vs return).

In many cases, the helper function doesn't actually need the
capability to return *absolutely any object*. In that case, the
obvious solution would be to have it return None to say "nothing to
return", and then the calling function can either translate that into
the appropriate exception, or return rather than yielding, as
appropriate. That would also make the helper more useful to other
stand-alone functions. But even if your helper has to be able to
return absolutely anything, you still have a few options:

1) Design the helper as part of __next__, and explicitly catch the exception.

def nexthelper():
    if condition: return value
    raise StopIteration

def __next__(self): return nexthelper()

def gen():
    try: yield nexthelper()
    except StopIteration: pass

2) Write the helper as a generator, and explicitly next() it if you need that:

def genhelper():
    if condition: yield value

def __next__(self): return next(genhelper())

def gen():
    yield from genhelper()

3) Return status and value. I don't like this, but it does work.

def tuplehelper():
    if condition: return True, value
    return False, None

def __next__(self):
    ok, val = tuplehelper()
    if ok: return val
    raise StopIteration

def gen():
    ok, val = tuplehelper()
    if ok: yield val

All these methods work perfectly, because they have a clear boundary
between protocols. If you want to write a __getitem__ that calls on
the same helper, you can do that, and have __getitem__ itself raise
appropriately if there's nothing to return.

> AFAIU, the current distinction between generators and iterators is how they
> are written -- i.e. syntax, essentially. But this PEP would change the
> behavior of generators in some small way, creating a distinction that
> doesn't currently exist.

Generators are currently a leaky abstraction for iterator classes.
This PEP plugs a leak that's capable of masking bugs.

> I guess this is where I'm not sure -- it seems to me that the behavior of
> generators is being change, not the syntax -- so while mostly of concern to
> generator authors, it is, in fact, a chance in behavior that can be seen by
> the consumer of (maybe only an oddly designed) generator. In practice, that
> difference may only matter to folks using that particular hack in generator
> expression, but it is indeed a change.

The only way a consumer will see a change of behaviour is if the
generator author used this specific hack (in which case, instead of
the generator quietly terminating, a RuntimeError will bubble up -
hopefully all the way up until a human sees it). In terms of this PEP,
that's a bug in the generator. Bug-free generators will not appear any
different to the consumer.

ChrisA

From antony.lee at berkeley.edu  Sat Nov 22 00:55:30 2014
From: antony.lee at berkeley.edu (Antony Lee)
Date: Fri, 21 Nov 2014 15:55:30 -0800
Subject: [Python-ideas] Counter comparison
Message-ID: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>

As Counter objects effectively behave like multi-sets, it seems reasonable
to overload <, <=, >, >= as for set objects to check whether a Counter is a
sub/super-set of another Counter:

c < d  <===>  all(c[k] < d[k] for k in c)

Thoughts?

Antony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141121/b917cf54/attachment.html>

From python at mrabarnett.plus.com  Sat Nov 22 01:27:59 2014
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 22 Nov 2014 00:27:59 +0000
Subject: [Python-ideas] Counter comparison
In-Reply-To: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>
References: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>
Message-ID: <546FD88F.6000709@mrabarnett.plus.com>

On 2014-11-21 23:55, Antony Lee wrote:
> As Counter objects effectively behave like multi-sets, it seems
> reasonable to overload <, <=, >, >= as for set objects to check whether
> a Counter is a sub/super-set of another Counter:
>
> c < d  <===>  all(c[k] < d[k] for k in c)
>
> Thoughts?
>
More correctly, it would be:

     all(c[k] < d[k] for k in set(c) | set(d))


From python at 2sn.net  Sat Nov 22 01:56:50 2014
From: python at 2sn.net (Alexander Heger)
Date: Sat, 22 Nov 2014 11:56:50 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CALGmxELgihRDMD=5_xeW7CWwg_7HL9BXwUW+JH-A5VLyg=TJTA@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info>
 <CALGmxEJHxSb-z4YQdci_HUFx42MxE3ooB8JHe+W_uwCmpPPsBQ@mail.gmail.com>
 <CAP7+vJ+fLsNm9sVdtRRxwLejoCu61V5TTFAJ1FiA8p+p2rLzrw@mail.gmail.com>
 <CALGmxELgihRDMD=5_xeW7CWwg_7HL9BXwUW+JH-A5VLyg=TJTA@mail.gmail.com>
Message-ID: <CAN3CYHw79pqQNJMJ40fOWVX8A9ZUvVRKSRwmg4cHsxg2-_o8iQ@mail.gmail.com>

> got it -- the issue at hand is what happens to a StopIteration that is
> raised by something the generator calls.
>
> I think the point of this PEP is that the author og a generator function is
> thinking about using "yield" to provide the next value, and return (explicit
> or implicit) to stop the generation of objects. That return is raise a
> StopIteration, but the author isn't thinking about it.
>
> So why would they want to think about having to trap StopIteration when
> calling other functions.
>
> While the author of a iterator class is thinking about the __next__ method
> and raising a StopIteration to terminate. So s/he would naturally think
> about trapping StopIteration when calling functions?
>
> I suppose that makes some sense, but to me it seems like a generator
> function is a different syntax for creating what is essentially the same
> thing -- why shouldn't it have the same behavior?
>
> and of you are writing a generator, presumably you know how it's going to
> get use -- i.e. by somethign that expects a StopIteration -- it's not like
> you're ignorant of the whole idea.
>
> Consider this far fetched situation:
>
> Either a iterator class or a generator function could take a function object
> to call to do part of its work. If that function happened to raise a
> StopIteration -- now the user would have to know which type of object they
> were workign with, so they would know how to handle the termination of the
> iter/gener-artion

I think this is a good point.

Maybe a way to obtain equivalency to the generator functions in this
case is to "break" this example for the iterator object as well, in
that StopIteration has to be raised in the frame of the generator
object; if it raised in a different context, e.g., a function called
by __next__, that StopIteration should also be converted to a
RuntimeError similar to what is proposed in the PEP for the generator
functions.  Maybe this is not what Chris intends to happen, but it
would make things consistent.

-Alexander

From python at mrabarnett.plus.com  Sat Nov 22 04:26:50 2014
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 22 Nov 2014 03:26:50 +0000
Subject: [Python-ideas] Counter comparison
In-Reply-To: <CAGRr6BHL=K25b8M8ZXzGzkBJVx3gXEA704Ur3jmoHFShx6YbeA@mail.gmail.com>
References: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>	<546FD88F.6000709@mrabarnett.plus.com>
 <CAGRr6BHL=K25b8M8ZXzGzkBJVx3gXEA704Ur3jmoHFShx6YbeA@mail.gmail.com>
Message-ID: <5470027A.7010504@mrabarnett.plus.com>


On 2014-11-22 01:36, Antony Lee wrote:
> I guess it depends on whether you think {"a": 1} < {"a": 2, "b": -1} 
> or not.  I actually thought about it and initially decided that the 
> inequality should hold, but given that most_common works fine with 
> negative counts, perhaps it shouldn't, indeed.
>
The other point is that if the 'b' entry was in the first, then it would 
be checked, and the negative count would matter, but as it wasn't, it 
won't, and it won't.

> 2014-11-21 16:27 GMT-08:00 MRAB <python at mrabarnett.plus.com 
> <mailto:python at mrabarnett.plus.com>>:
>
>     On 2014-11-21 23:55, Antony Lee wrote:
>
>         As Counter objects effectively behave like multi-sets, it seems
>         reasonable to overload <, <=, >, >= as for set objects to
>         check whether
>         a Counter is a sub/super-set of another Counter:
>
>         c < d  <===>  all(c[k] < d[k] for k in c)
>
>         Thoughts?
>
>     More correctly, it would be:
>
>         all(c[k] < d[k] for k in set(c) | set(d))
>


From steve at pearwood.info  Sat Nov 22 11:48:27 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 22 Nov 2014 21:48:27 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmo-YEtZ0OOpNLf3X1N3y5tH5rp+T4_ybHwbaWBYDQjAUQ@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info>
 <CALGmxEJHxSb-z4YQdci_HUFx42MxE3ooB8JHe+W_uwCmpPPsBQ@mail.gmail.com>
 <CAP7+vJ+fLsNm9sVdtRRxwLejoCu61V5TTFAJ1FiA8p+p2rLzrw@mail.gmail.com>
 <CALGmxELgihRDMD=5_xeW7CWwg_7HL9BXwUW+JH-A5VLyg=TJTA@mail.gmail.com>
 <CAPTjJmo-YEtZ0OOpNLf3X1N3y5tH5rp+T4_ybHwbaWBYDQjAUQ@mail.gmail.com>
Message-ID: <20141122104827.GQ2748@ando.pearwood.info>

On Sat, Nov 22, 2014 at 10:40:37AM +1100, Chris Angelico wrote:

> Let's suppose you use a Python-like macro language to generate Python
> code. In the macro language, you can write stuff like this:

Is there really any point in hypothesing imaginary macro languages when 
we have a concrete and existing language (Python itself) to look at? 

[snip made-up example]

> This is a reasonably plausible macro language, right? It's basically
> still a class definition, but it lets you leave out a whole bunch of
> boilerplate. Now, the question is: As you write the simplified
> version, should you ever need to concern yourself with StopIteration?

Sure, why not? It is part of the concrete protocol: iterators raise 
StopIteration to halt. That's not a secret, and it is not an 
implementation detail, it is a concrete, public part of the API.


> I posit no, you should not; it's not a part of the macro language at
> all.

This is why talking about imaginary macro languages is pointless. You 
say it is not part of the macro language. I say it is. Since the 
language doesn't actually exist, who is to say which is right?

In real Python code, "raise StopIteration" does exist, and does work in 
generators. Sometimes the fact that it works is a nuisance, when you 
have an unexpected StopIteration. Sometimes the fact that it works is 
exactly what you want, when you have an expected StopIteration. You seem 
to think that allowing a generator function to delegate the decision to 
halt to a helper function is a Bad Thing. I say it is a Good Thing, even 
if it occasionally makes buggy code a bit harder to debug.


[...]
> > and of you are writing a generator, presumably you know how it's going to
> > get use -- i.e. by somethign that expects a StopIteration -- it's not like
> > you're ignorant of the whole idea.
> 
> Not necessarily. Can we get someone here who knows asyncio and
> coroutines, and can comment on the use of such generators?

What about them? I don't understand your question.



-- 
Steven

From rosuav at gmail.com  Sat Nov 22 11:57:26 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 22 Nov 2014 21:57:26 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <20141122104827.GQ2748@ando.pearwood.info>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info>
 <CALGmxEJHxSb-z4YQdci_HUFx42MxE3ooB8JHe+W_uwCmpPPsBQ@mail.gmail.com>
 <CAP7+vJ+fLsNm9sVdtRRxwLejoCu61V5TTFAJ1FiA8p+p2rLzrw@mail.gmail.com>
 <CALGmxELgihRDMD=5_xeW7CWwg_7HL9BXwUW+JH-A5VLyg=TJTA@mail.gmail.com>
 <CAPTjJmo-YEtZ0OOpNLf3X1N3y5tH5rp+T4_ybHwbaWBYDQjAUQ@mail.gmail.com>
 <20141122104827.GQ2748@ando.pearwood.info>
Message-ID: <CAPTjJmqvvsbdPScMRcOB6Zx7SS+4xx04N=WAKYiySxiRh5+AfA@mail.gmail.com>

On Sat, Nov 22, 2014 at 9:48 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Sat, Nov 22, 2014 at 10:40:37AM +1100, Chris Angelico wrote:
>
>> Let's suppose you use a Python-like macro language to generate Python
>> code. In the macro language, you can write stuff like this:
>
> Is there really any point in hypothesing imaginary macro languages when
> we have a concrete and existing language (Python itself) to look at?
>
> [snip made-up example]
>
>> This is a reasonably plausible macro language, right? It's basically
>> still a class definition, but it lets you leave out a whole bunch of
>> boilerplate. Now, the question is: As you write the simplified
>> version, should you ever need to concern yourself with StopIteration?
>
> Sure, why not? It is part of the concrete protocol: iterators raise
> StopIteration to halt. That's not a secret, and it is not an
> implementation detail, it is a concrete, public part of the API.
>
>
>> I posit no, you should not; it's not a part of the macro language at
>> all.
>
> This is why talking about imaginary macro languages is pointless. You
> say it is not part of the macro language. I say it is. Since the
> language doesn't actually exist, who is to say which is right?

A generator function is exactly the same thing: it's a way to create
an iterator, but it's not a class with a __next__ function. I could
write an iterator-creation function in many ways, none of which
involve StopIteration:

def gen():
    if condition: raise StopIteration # Wrong
    return iter([1,2,3])

>> > and of you are writing a generator, presumably you know how it's going to
>> > get use -- i.e. by somethign that expects a StopIteration -- it's not like
>> > you're ignorant of the whole idea.
>>
>> Not necessarily. Can we get someone here who knows asyncio and
>> coroutines, and can comment on the use of such generators?
>
> What about them? I don't understand your question.

If you have a lengthy nested chain of coroutines, and one of them
unexpectedly raises StopIteration, is it right for something to
quietly terminate, or should the exception bubble up and be printed to
console?

ChrisA

From raymond.hettinger at gmail.com  Sat Nov 22 16:47:51 2014
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sat, 22 Nov 2014 07:47:51 -0800
Subject: [Python-ideas] Counter comparison
In-Reply-To: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>
References: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>
Message-ID: <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com>


> On Nov 21, 2014, at 3:55 PM, Antony Lee <antony.lee at berkeley.edu> wrote:
> 
> As Counter objects effectively behave like multi-sets, it seems reasonable to overload <, <=, >, >= as for set objects to check whether a Counter is a sub/super-set of another Counter:
> 
> c < d  <===>  all(c[k] < d[k] for k in c)
> 
> Thoughts

This is something that could be done, but I think the demonstrated need is nearly zero.  It doesn't seem to arise in practice.

Conceptually, a counter is just a dictionary that returns zero instead of raising a KeyError.  So, while they can be used as multisets, they have other uses as well (for example, allowing negative counts or fractional counts).

Another thought, is that overloading comparison operators risks isn't always a good thing.  Even for regular sets, people sometimes get surprised that the sort order isn't deterministic (because sets have a partial ordering instead of a total ordering).

The idea of adding counter comparisons was considered from the outset and was intentionally not included.  If compelling use cases were arising in practice, we could add comparison support, but until then it is more prudent to be conservative with the API.
  

Raymond



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141122/73c54ad7/attachment.html>

From solipsis at pitrou.net  Sat Nov 22 17:06:19 2014
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 22 Nov 2014 17:06:19 +0100
Subject: [Python-ideas] Counter comparison
References: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>
 <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com>
Message-ID: <20141122170619.07722252@fsol>

On Sat, 22 Nov 2014 07:47:51 -0800
Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
> 
> > On Nov 21, 2014, at 3:55 PM, Antony Lee <antony.lee at berkeley.edu> wrote:
> > 
> > As Counter objects effectively behave like multi-sets, it seems reasonable to overload <, <=, >, >= as for set objects to check whether a Counter is a sub/super-set of another Counter:
> > 
> > c < d  <===>  all(c[k] < d[k] for k in c)
> > 
> > Thoughts
> 
> This is something that could be done, but I think the demonstrated need is nearly zero.  It doesn't seem to arise in practice.

You say this while you closed an issue where the poster asked for
this very feature, and clearly stated what his needs were:
http://bugs.python.org/issue22515

> Conceptually, a counter is just a dictionary that returns zero instead of raising a KeyError.

Come on. If that were the case, it wouldn't expose other operators such
as addition, subtraction, intersection and union.
https://docs.python.org/3.3/library/collections.html#collections.Counter

> So, while they can be used as multisets, they have other uses as well
> (for example, allowing negative counts or fractional counts).

The demonstrated need for those "other uses" is nearly zero. It doesn't
seem to arise in practice.

Your insistance on defending irrational API decisions is staggering.

Regards

Antoine.



From jeanpierreda at gmail.com  Sat Nov 22 17:43:42 2014
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Sat, 22 Nov 2014 08:43:42 -0800
Subject: [Python-ideas] Counter comparison
In-Reply-To: <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com>
References: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>
 <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com>
Message-ID: <CABicbJKOBJ6+RmdN53jBVMssbyicuC-wxRNPHi0bRS9Scv6D3w@mail.gmail.com>

On Sat, Nov 22, 2014 at 7:47 AM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
> Conceptually, a counter is just a dictionary that returns zero instead of
> raising a KeyError.  So, while they can be used as multisets, they have
> other uses as well (for example, allowing negative counts or fractional
> counts).

This does little to nothing to explain most of the behavior of
counters, like that of &. They are most naturally thought of as
multisets.

> Another thought, is that overloading comparison operators risks isn't always
> a good thing.  Even for regular sets, people sometimes get surprised that
> the sort order isn't deterministic (because sets have a partial ordering
> instead of a total ordering).

Sure. OTOH it is possible to still be conservative while adding this:
these risks do not exist for the set comparison operators: isdisjoint,
issubset, issuperset.

-- Devin

From storchaka at gmail.com  Sat Nov 22 19:03:02 2014
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 22 Nov 2014 20:03:02 +0200
Subject: [Python-ideas] Counter comparison
In-Reply-To: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>
References: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>
Message-ID: <m4qj6q$hdu$1@ger.gmane.org>

On 22.11.14 01:55, Antony Lee wrote:
> As Counter objects effectively behave like multi-sets, it seems
> reasonable to overload <, <=, >, >= as for set objects to check whether
> a Counter is a sub/super-set of another Counter:
>
> c < d  <===>  all(c[k] < d[k] for k in c)
>
> Thoughts?

http://bugs.python.org/issue22515



From raymond.hettinger at gmail.com  Sun Nov 23 03:51:30 2014
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sat, 22 Nov 2014 18:51:30 -0800
Subject: [Python-ideas] Counter comparison
In-Reply-To: <20141122170619.07722252@fsol>
References: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>
 <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com>
 <20141122170619.07722252@fsol>
Message-ID: <B1ACF73D-6BD1-4C12-A9CB-8FE8269DBC22@gmail.com>


> On Nov 22, 2014, at 8:06 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> 
> Your insistance on defending irrational API decisions is staggering.

Gee, that's a little harsh.

I'm trying to exercise good judgment over API design decisions.  You may not agree, but I should at least share my thought process:

* Even for regular sets, overriding the comparison operators was a bit dodgy leading to some oddities -- one because it is partial ordering that confuses sort() and another because of people's propensity to write "set_a < set_b" when they really meant "set_a <= set_b".

* Also, I have reservations about having added any multi-set operations to counters in the first place.  That seems to be the only the part of counter API that has agitated or confused anyone.  The problems arise the counter is an extension of the dict API rather than being a fully encapsulated bag or multiset that can prevent counts from being zero, negative, or non-integral.  In other words, the multi-set operations may have added some value but they weren't a perfect fit.  

* With those two thoughts in mind, I had some reluctance to add anything "doubly dodgy" by adding multi-set comparisons.

* Next, there is some question of what the comparison operations should do in the presence of zeros or negative counts.  Inspired by the design principles in the decimal module (all numbers are exact and rounding is only applied after operations rather than before), the current multi-set operations eliminate non-negative results after the operation rather than before.   That said, I don't know whether it makes sense multi-set comparison to follow that model (for consistency) or to clean-up the inputs prior to the operation.  For example, should Counter(a=-1, b=0, c=1) be considered a proper subset of Counter(c=2, d=3, e=0, f=-1)?    There are reasons to say yes and reasons you could say no.  To inform the best possible decision, it would be great to have real use cases to guide the design process.

* Which gets us to use cases.  When Ram suggested multiset operations in a feature request, all he said was "I suggest implementing `Counter.__lt__` which will be a partial order, similarly to `set.__lt__`.".  There was no mention of use case until later where all he said was "I needed it for an internal calculation in a combinatorics package I'm writing.
" and he did not elaborate further.  As the discussion evolved, no further use case detail emerged.  However, later in the discussion it because clear that several of the posters either thought the idea didn't make sense or were confused about what the it should do.   Near the end of the discussion, the OP said he had given up on the idea because of the issues regarding the "hybrid mapping/multiset" design.  After the -1s, examples of people being confused, and the OP's withdrawal, I stepped in and closed the feature request.

* This current thread did not begin with a use case either.  It was more along the lines of "in for a penny, in for a pound".  Since some multiset operations were implemented, maybe you should do them all.

For the reasons listed above, I prefer to be conservative and not expand the API further.  For the most part, people seem to use counters for counting.  They are good at that task and popular.  I don't feel a strong need to expand the API into dodgy territory unless really good use cases arise to guide the design and make it feel like the added benefits were worth moving further on to shaky ground.

Also, I've taken some comfort in knowing that since a counter is a dictionary subclass, it is trivially easy to extend for users.  If the user can already meet any perceived need by writing "all(c[k] < d[k] for k in c)", then I'm not sure we can make much of a value add.

You may not agree with the outcome of my thought process, but it is unfair to call it irrational.

not-everything-that-can-be-done-should-be-done-ly yours,


Raymond



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141122/b52be67a/attachment-0001.html>

From ncoghlan at gmail.com  Sun Nov 23 06:33:05 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 23 Nov 2014 15:33:05 +1000
Subject: [Python-ideas] Counter comparison
In-Reply-To: <B1ACF73D-6BD1-4C12-A9CB-8FE8269DBC22@gmail.com>
References: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>
 <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com>
 <20141122170619.07722252@fsol>
 <B1ACF73D-6BD1-4C12-A9CB-8FE8269DBC22@gmail.com>
Message-ID: <CADiSq7eHZtb2vGAAcararuWgrcAnoab6AE=G+saikmikQvn=RQ@mail.gmail.com>

On 23 November 2014 at 12:51, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
[snip]

> For the reasons listed above, I prefer to be conservative and not expand the
> API further.  For the most part, people seem to use counters for counting.
> They are good at that task and popular.  I don't feel a strong need to
> expand the API into dodgy territory unless really good use cases arise to
> guide the design and make it feel like the added benefits were worth moving
> further on to shaky ground.
>
> Also, I've taken some comfort in knowing that since a counter is a
> dictionary subclass, it is trivially easy to extend for users.  If the user
> can already meet any perceived need by writing "all(c[k] < d[k] for k in
> c)", then I'm not sure we can make much of a value add.
>
> You may not agree with the outcome of my thought process, but it is unfair
> to call it irrational.

Thanks for the detailed explanation.

> not-everything-that-can-be-done-should-be-done-ly yours,

This is a key point. Providing APIs that have odd edge case behaviour
can be worse than not providing a particular feature at all. When the
feature doesn't exist, folks are more inclined to just write their own
custom class or function that does exactly what they need. It's
generally feasible to keep those quite concise, since they don't need
to handle all possible variations that may arise.

With the existing Counter-as-multiset features already offering some
potential for confusion, and the potential for further confusing
interactions between additional multiset features and Counter's
support for non-integer values, zero values (distinct from keys being
missing entirely) and negative values, there may be scope for a
separate true multiset API that draws more heavily on the set API
design than the dict API design.

A dedicated multiset API would potentially be able to avoid the
confusing aspects of Counters-as-multisets by only allowing
non-negative integer values. Is there sufficient value in such an API
to justify adding it? Or would it just create confusion as folks tried
to decide between using Counter or using the new multiset/bag
container for their algorithm?

That's an open question, but at the very least, it's worth considering
as an alternative to further elaborating on an already confusing
aspect of the collections.Counter design. There's at least one such
example of a bag API already available on PyPI:
https://pypi.python.org/pypi/data-structures/0.1.4#bag (you need
"-Counter" in a Google search to find that, as most current hits
describe the use of Counter as a multiset, rather than using a
dedicated container type)

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ethan at stoneleaf.us  Sun Nov 23 16:40:46 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 23 Nov 2014 07:40:46 -0800
Subject: [Python-ideas] Counter comparison
In-Reply-To: <CADiSq7eHZtb2vGAAcararuWgrcAnoab6AE=G+saikmikQvn=RQ@mail.gmail.com>
References: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>
 <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com>
 <20141122170619.07722252@fsol>
 <B1ACF73D-6BD1-4C12-A9CB-8FE8269DBC22@gmail.com>
 <CADiSq7eHZtb2vGAAcararuWgrcAnoab6AE=G+saikmikQvn=RQ@mail.gmail.com>
Message-ID: <5471FFFE.4080309@stoneleaf.us>

On 11/22/2014 09:33 PM, Nick Coghlan wrote:
> 
> With the existing Counter-as-multiset features already offering some
> potential for confusion, and the potential for further confusing
> interactions between additional multiset features and Counter's
> support for non-integer values, zero values (distinct from keys being
> missing entirely) and negative values, there may be scope for a
> separate true multiset API that draws more heavily on the set API
> design than the dict API design.
> 
> A dedicated multiset API would potentially be able to avoid the
> confusing aspects of Counters-as-multisets by only allowing
> non-negative integer values. Is there sufficient value in such an API
> to justify adding it? Or would it just create confusion as folks tried
> to decide between using Counter or using the new multiset/bag
> container for their algorithm?
> 
> That's an open question, but at the very least, it's worth considering
> as an alternative to further elaborating on an already confusing
> aspect of the collections.Counter design. There's at least one such
> example of a bag API already available on PyPI:
> https://pypi.python.org/pypi/data-structures/0.1.4#bag (you need
> "-Counter" in a Google search to find that, as most current hits
> describe the use of Counter as a multiset, rather than using a
> dedicated container type)

I have also wondered about the feasibility of separating out the multiset features into a distinct type.  Seems like
that would avoid a bunch of confusion.

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141123/5f0baa68/attachment.sig>

From antony.lee at berkeley.edu  Sun Nov 23 20:30:25 2014
From: antony.lee at berkeley.edu (Antony Lee)
Date: Sun, 23 Nov 2014 11:30:25 -0800
Subject: [Python-ideas] Counter comparison
In-Reply-To: <5471FFFE.4080309@stoneleaf.us>
References: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>
 <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com>
 <20141122170619.07722252@fsol>
 <B1ACF73D-6BD1-4C12-A9CB-8FE8269DBC22@gmail.com>
 <CADiSq7eHZtb2vGAAcararuWgrcAnoab6AE=G+saikmikQvn=RQ@mail.gmail.com>
 <5471FFFE.4080309@stoneleaf.us>
Message-ID: <CAGRr6BG68h17SGo5wtDO6bnhwYTf2ZfabAOye83ki4Q9JtLv3A@mail.gmail.com>

My goal was very simply to check whether it was possible to remove a
multi-set of elements from another, without any counts going below 0 (as
should be the case for "natural" counters).
Antony

2014-11-23 7:40 GMT-08:00 Ethan Furman <ethan at stoneleaf.us>:

> On 11/22/2014 09:33 PM, Nick Coghlan wrote:
> >
> > With the existing Counter-as-multiset features already offering some
> > potential for confusion, and the potential for further confusing
> > interactions between additional multiset features and Counter's
> > support for non-integer values, zero values (distinct from keys being
> > missing entirely) and negative values, there may be scope for a
> > separate true multiset API that draws more heavily on the set API
> > design than the dict API design.
> >
> > A dedicated multiset API would potentially be able to avoid the
> > confusing aspects of Counters-as-multisets by only allowing
> > non-negative integer values. Is there sufficient value in such an API
> > to justify adding it? Or would it just create confusion as folks tried
> > to decide between using Counter or using the new multiset/bag
> > container for their algorithm?
> >
> > That's an open question, but at the very least, it's worth considering
> > as an alternative to further elaborating on an already confusing
> > aspect of the collections.Counter design. There's at least one such
> > example of a bag API already available on PyPI:
> > https://pypi.python.org/pypi/data-structures/0.1.4#bag (you need
> > "-Counter" in a Google search to find that, as most current hits
> > describe the use of Counter as a multiset, rather than using a
> > dedicated container type)
>
> I have also wondered about the feasibility of separating out the multiset
> features into a distinct type.  Seems like
> that would avoid a bunch of confusion.
>
> --
> ~Ethan~
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141123/4abffb66/attachment.html>

From steve at pearwood.info  Mon Nov 24 01:04:38 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 24 Nov 2014 11:04:38 +1100
Subject: [Python-ideas] Counter comparison
In-Reply-To: <CAGRr6BG68h17SGo5wtDO6bnhwYTf2ZfabAOye83ki4Q9JtLv3A@mail.gmail.com>
References: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>
 <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com>
 <20141122170619.07722252@fsol>
 <B1ACF73D-6BD1-4C12-A9CB-8FE8269DBC22@gmail.com>
 <CADiSq7eHZtb2vGAAcararuWgrcAnoab6AE=G+saikmikQvn=RQ@mail.gmail.com>
 <5471FFFE.4080309@stoneleaf.us>
 <CAGRr6BG68h17SGo5wtDO6bnhwYTf2ZfabAOye83ki4Q9JtLv3A@mail.gmail.com>
Message-ID: <20141124000438.GW2748@ando.pearwood.info>

On Sun, Nov 23, 2014 at 11:30:25AM -0800, Antony Lee wrote:
> My goal was very simply to check whether it was possible to remove a
> multi-set of elements from another, without any counts going below 0 (as
> should be the case for "natural" counters).

Do you mean this?

py> from collections import Counter
py> c1 = Counter({'a': 5, 'b': 2})
py> c2 = Counter({'a': 1, 'b': 4})
py> c1 - c2
Counter({'a': 4})



-- 
Steven

From ethan at stoneleaf.us  Mon Nov 24 02:03:27 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 23 Nov 2014 17:03:27 -0800
Subject: [Python-ideas] Counter comparison
In-Reply-To: <20141124000438.GW2748@ando.pearwood.info>
References: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>
 <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com>
 <20141122170619.07722252@fsol>
 <B1ACF73D-6BD1-4C12-A9CB-8FE8269DBC22@gmail.com>
 <CADiSq7eHZtb2vGAAcararuWgrcAnoab6AE=G+saikmikQvn=RQ@mail.gmail.com>
 <5471FFFE.4080309@stoneleaf.us>
 <CAGRr6BG68h17SGo5wtDO6bnhwYTf2ZfabAOye83ki4Q9JtLv3A@mail.gmail.com>
 <20141124000438.GW2748@ando.pearwood.info>
Message-ID: <547283DF.8030406@stoneleaf.us>

On 11/23/2014 04:04 PM, Steven D'Aprano wrote:
> On Sun, Nov 23, 2014 at 11:30:25AM -0800, Antony Lee wrote:
>>
>> My goal was very simply to check whether it was possible to remove a
>> multi-set of elements from another, without any counts going below 0 (as
>> should be the case for "natural" counters).
> 
> Do you mean this?
> 
> py> from collections import Counter
> py> c1 = Counter({'a': 5, 'b': 2})
> py> c2 = Counter({'a': 1, 'b': 4})
> py> c1 - c2
> Counter({'a': 4})

Um, how does that show that 'b' went below zero?

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141123/28092660/attachment.sig>

From steve at pearwood.info  Mon Nov 24 02:20:09 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 24 Nov 2014 12:20:09 +1100
Subject: [Python-ideas] Counter comparison
In-Reply-To: <547283DF.8030406@stoneleaf.us>
References: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>
 <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com>
 <20141122170619.07722252@fsol>
 <B1ACF73D-6BD1-4C12-A9CB-8FE8269DBC22@gmail.com>
 <CADiSq7eHZtb2vGAAcararuWgrcAnoab6AE=G+saikmikQvn=RQ@mail.gmail.com>
 <5471FFFE.4080309@stoneleaf.us>
 <CAGRr6BG68h17SGo5wtDO6bnhwYTf2ZfabAOye83ki4Q9JtLv3A@mail.gmail.com>
 <20141124000438.GW2748@ando.pearwood.info> <547283DF.8030406@stoneleaf.us>
Message-ID: <20141124012009.GZ2748@ando.pearwood.info>

On Sun, Nov 23, 2014 at 05:03:27PM -0800, Ethan Furman wrote:
> On 11/23/2014 04:04 PM, Steven D'Aprano wrote:
> > On Sun, Nov 23, 2014 at 11:30:25AM -0800, Antony Lee wrote:
> >>
> >> My goal was very simply to check whether it was possible to remove a
> >> multi-set of elements from another, without any counts going below 0 (as
> >> should be the case for "natural" counters).
> > 
> > Do you mean this?
> > 
> > py> from collections import Counter
> > py> c1 = Counter({'a': 5, 'b': 2})
> > py> c2 = Counter({'a': 1, 'b': 4})
> > py> c1 - c2
> > Counter({'a': 4})
> 
> Um, how does that show that 'b' went below zero?

It doesn't, which is the point. Antony asked for removal WITHOUT counts 
going below zero.



-- 
Steve

From ethan at stoneleaf.us  Mon Nov 24 02:30:45 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 23 Nov 2014 17:30:45 -0800
Subject: [Python-ideas] Counter comparison
In-Reply-To: <20141124012009.GZ2748@ando.pearwood.info>
References: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>
 <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com>
 <20141122170619.07722252@fsol>
 <B1ACF73D-6BD1-4C12-A9CB-8FE8269DBC22@gmail.com>
 <CADiSq7eHZtb2vGAAcararuWgrcAnoab6AE=G+saikmikQvn=RQ@mail.gmail.com>
 <5471FFFE.4080309@stoneleaf.us>
 <CAGRr6BG68h17SGo5wtDO6bnhwYTf2ZfabAOye83ki4Q9JtLv3A@mail.gmail.com>
 <20141124000438.GW2748@ando.pearwood.info> <547283DF.8030406@stoneleaf.us>
 <20141124012009.GZ2748@ando.pearwood.info>
Message-ID: <54728A45.6030801@stoneleaf.us>

On 11/23/2014 05:20 PM, Steven D'Aprano wrote:
> On Sun, Nov 23, 2014 at 05:03:27PM -0800, Ethan Furman wrote:
>> On 11/23/2014 04:04 PM, Steven D'Aprano wrote:
>>> On Sun, Nov 23, 2014 at 11:30:25AM -0800, Antony Lee wrote:
>>>>
>>>> My goal was very simply to check whether it was possible to remove a
>>>> multi-set of elements from another, without any counts going below 0 (as
>>>> should be the case for "natural" counters).
>>>
>>> Do you mean this?
>>>
>>> py> from collections import Counter
>>> py> c1 = Counter({'a': 5, 'b': 2})
>>> py> c2 = Counter({'a': 1, 'b': 4})
>>> py> c1 - c2
>>> Counter({'a': 4})
>>
>> Um, how does that show that 'b' went below zero?
> 
> It doesn't, which is the point. Antony asked for removal WITHOUT counts 
> going below zero.

Ah.  I read that as "being able to tell if any count would go below zero" -- as in, counter1 is my inventory, counter2
is an order, can I fill the order?

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141123/c4767ade/attachment-0001.sig>

From antony.lee at berkeley.edu  Mon Nov 24 06:37:20 2014
From: antony.lee at berkeley.edu (Antony Lee)
Date: Sun, 23 Nov 2014 21:37:20 -0800
Subject: [Python-ideas] Counter comparison
In-Reply-To: <54728A45.6030801@stoneleaf.us>
References: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>
 <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com>
 <20141122170619.07722252@fsol>
 <B1ACF73D-6BD1-4C12-A9CB-8FE8269DBC22@gmail.com>
 <CADiSq7eHZtb2vGAAcararuWgrcAnoab6AE=G+saikmikQvn=RQ@mail.gmail.com>
 <5471FFFE.4080309@stoneleaf.us>
 <CAGRr6BG68h17SGo5wtDO6bnhwYTf2ZfabAOye83ki4Q9JtLv3A@mail.gmail.com>
 <20141124000438.GW2748@ando.pearwood.info>
 <547283DF.8030406@stoneleaf.us>
 <20141124012009.GZ2748@ando.pearwood.info>
 <54728A45.6030801@stoneleaf.us>
Message-ID: <CAGRr6BE7Xt_Kvujk-CNxP3r64WLgtu7krDf_R+t3hfCoT=FxJQ@mail.gmail.com>

I guess my post was unclear but my intent was indeed Ethan's (can I satisfy
an order from my inventory?), not Steven's.

2014-11-23 17:30 GMT-08:00 Ethan Furman <ethan at stoneleaf.us>:

> On 11/23/2014 05:20 PM, Steven D'Aprano wrote:
> > On Sun, Nov 23, 2014 at 05:03:27PM -0800, Ethan Furman wrote:
> >> On 11/23/2014 04:04 PM, Steven D'Aprano wrote:
> >>> On Sun, Nov 23, 2014 at 11:30:25AM -0800, Antony Lee wrote:
> >>>>
> >>>> My goal was very simply to check whether it was possible to remove a
> >>>> multi-set of elements from another, without any counts going below 0
> (as
> >>>> should be the case for "natural" counters).
> >>>
> >>> Do you mean this?
> >>>
> >>> py> from collections import Counter
> >>> py> c1 = Counter({'a': 5, 'b': 2})
> >>> py> c2 = Counter({'a': 1, 'b': 4})
> >>> py> c1 - c2
> >>> Counter({'a': 4})
> >>
> >> Um, how does that show that 'b' went below zero?
> >
> > It doesn't, which is the point. Antony asked for removal WITHOUT counts
> > going below zero.
>
> Ah.  I read that as "being able to tell if any count would go below zero"
> -- as in, counter1 is my inventory, counter2
> is an order, can I fill the order?
>
> --
> ~Ethan~
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141123/78ce7ef8/attachment.html>

From random832 at fastmail.us  Mon Nov 24 07:09:51 2014
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Mon, 24 Nov 2014 01:09:51 -0500
Subject: [Python-ideas] Counter comparison
In-Reply-To: <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com>
References: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>
 <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com>
Message-ID: <1416809391.2601292.194579353.687F5B82@webmail.messagingengine.com>

On Sat, Nov 22, 2014, at 10:47, Raymond Hettinger wrote:
> Another thought, is that overloading comparison operators risks isn't
> always a good thing.  Even for regular sets, people sometimes get
> surprised that the sort order isn't deterministic (because sets have a
> partial ordering instead of a total ordering).

Thank you, incidentally, for an example better than floats (in presence
of nan) of a type that will break sorting algorithms.

From guido at python.org  Mon Nov 24 16:43:41 2014
From: guido at python.org (Guido van Rossum)
Date: Mon, 24 Nov 2014 07:43:41 -0800
Subject: [Python-ideas] Counter comparison
In-Reply-To: <1416809391.2601292.194579353.687F5B82@webmail.messagingengine.com>
References: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>
 <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com>
 <1416809391.2601292.194579353.687F5B82@webmail.messagingengine.com>
Message-ID: <CAP7+vJ+f=vRR_BuE=AR+54FkyWJfDnWaFZ7dXxkvqqBT6h_MJw@mail.gmail.com>

On Sun, Nov 23, 2014 at 10:09 PM, <random832 at fastmail.us> wrote:

> On Sat, Nov 22, 2014, at 10:47, Raymond Hettinger wrote:
> > Another thought, is that overloading comparison operators risks isn't
> > always a good thing.  Even for regular sets, people sometimes get
> > surprised that the sort order isn't deterministic (because sets have a
> > partial ordering instead of a total ordering).
>
> Thank you, incidentally, for an example better than floats (in presence
> of nan) of a type that will break sorting algorithms.
>

But does it matter? If you've got a list of sets, and you want it sorted,
you have to specify how you want it sorted anyways. You can complain that <
is only a partial order, but that is the mathematical root of the
situation, not a problem caused by an poorly chosen language default.
(Leaving < undefined and sorting by object address, now *that* would be an
example of the latter. :-)

Math is often surprising. This how you learn. (You may also learn more
about sorting algorithms this way.)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141124/9dc3bc13/attachment.html>

From chris.barker at noaa.gov  Mon Nov 24 17:38:49 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 24 Nov 2014 08:38:49 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAN3CYHw79pqQNJMJ40fOWVX8A9ZUvVRKSRwmg4cHsxg2-_o8iQ@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info>
 <CALGmxEJHxSb-z4YQdci_HUFx42MxE3ooB8JHe+W_uwCmpPPsBQ@mail.gmail.com>
 <CAP7+vJ+fLsNm9sVdtRRxwLejoCu61V5TTFAJ1FiA8p+p2rLzrw@mail.gmail.com>
 <CALGmxELgihRDMD=5_xeW7CWwg_7HL9BXwUW+JH-A5VLyg=TJTA@mail.gmail.com>
 <CAN3CYHw79pqQNJMJ40fOWVX8A9ZUvVRKSRwmg4cHsxg2-_o8iQ@mail.gmail.com>
Message-ID: <CALGmxEKX3yeeA0P===h7AmLwOGTkAPCiCJPht6xX-6oiG+aPAg@mail.gmail.com>

On Fri, Nov 21, 2014 at 4:56 PM, Alexander Heger <python at 2sn.net> wrote:

> Maybe a way to obtain equivalency to the generator functions in this
> case is to "break" this example for the iterator object as well, in
> that StopIteration has to be raised in the frame of the generator
> object; if it raised in a different context, e.g., a function called
> by __next__, that StopIteration should also be converted to a
> RuntimeError similar to what is proposed in the PEP for the generator
> functions.  Maybe this is not what Chris intends to happen, but it
> would make things consistent.
>

I"mn not sure which Chris you are refering to, but for my part, yes and no:

Yes, that would keep iterator classes and generator functions consistent,
which would be a good thing.

No: I don't think we should do that -- StopIteration is part of the
iterator protocol -- generators are another way to write something that
complies with the iterator protocol -- generators should handle
StopIteration the same way that iterator classes do.

Yes, there are some cases that can be confusing and hard to debug -- but
them's the breaks.

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141124/86510d0e/attachment.html>

From chris.barker at noaa.gov  Mon Nov 24 17:41:46 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 24 Nov 2014 08:41:46 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmqvvsbdPScMRcOB6Zx7SS+4xx04N=WAKYiySxiRh5+AfA@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info>
 <CALGmxEJHxSb-z4YQdci_HUFx42MxE3ooB8JHe+W_uwCmpPPsBQ@mail.gmail.com>
 <CAP7+vJ+fLsNm9sVdtRRxwLejoCu61V5TTFAJ1FiA8p+p2rLzrw@mail.gmail.com>
 <CALGmxELgihRDMD=5_xeW7CWwg_7HL9BXwUW+JH-A5VLyg=TJTA@mail.gmail.com>
 <CAPTjJmo-YEtZ0OOpNLf3X1N3y5tH5rp+T4_ybHwbaWBYDQjAUQ@mail.gmail.com>
 <20141122104827.GQ2748@ando.pearwood.info>
 <CAPTjJmqvvsbdPScMRcOB6Zx7SS+4xx04N=WAKYiySxiRh5+AfA@mail.gmail.com>
Message-ID: <CALGmxELZjO5GPu9ZTNyxc9PatO59qOr7urDmj8Wo-O8zP2MgRQ@mail.gmail.com>

On Sat, Nov 22, 2014 at 2:57 AM, Chris Angelico <rosuav at gmail.com> wrote:

> If you have a lengthy nested chain of coroutines, and one of them
> unexpectedly raises StopIteration, is it right for something to
> quietly terminate, or should the exception bubble up and be printed to
> console?


Couldn't you have a nested pile of iterator classes as well that would
exhibit the exact same behavior?

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141124/886ff683/attachment.html>

From rosuav at gmail.com  Mon Nov 24 18:06:28 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 25 Nov 2014 04:06:28 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CALGmxEKX3yeeA0P===h7AmLwOGTkAPCiCJPht6xX-6oiG+aPAg@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info>
 <CALGmxEJHxSb-z4YQdci_HUFx42MxE3ooB8JHe+W_uwCmpPPsBQ@mail.gmail.com>
 <CAP7+vJ+fLsNm9sVdtRRxwLejoCu61V5TTFAJ1FiA8p+p2rLzrw@mail.gmail.com>
 <CALGmxELgihRDMD=5_xeW7CWwg_7HL9BXwUW+JH-A5VLyg=TJTA@mail.gmail.com>
 <CAN3CYHw79pqQNJMJ40fOWVX8A9ZUvVRKSRwmg4cHsxg2-_o8iQ@mail.gmail.com>
 <CALGmxEKX3yeeA0P===h7AmLwOGTkAPCiCJPht6xX-6oiG+aPAg@mail.gmail.com>
Message-ID: <CAPTjJmrQWOkrukEgm6k-QkueaZu_zA1t5GpMHXNcNd-XzRcb7Q@mail.gmail.com>

On Tue, Nov 25, 2014 at 3:38 AM, Chris Barker <chris.barker at noaa.gov> wrote:
> On Fri, Nov 21, 2014 at 4:56 PM, Alexander Heger <python at 2sn.net> wrote:
>>
>> Maybe a way to obtain equivalency to the generator functions in this
>> case is to "break" this example for the iterator object as well, in
>> that StopIteration has to be raised in the frame of the generator
>> object; if it raised in a different context, e.g., a function called
>> by __next__, that StopIteration should also be converted to a
>> RuntimeError similar to what is proposed in the PEP for the generator
>> functions.  Maybe this is not what Chris intends to happen, but it
>> would make things consistent.
>
>
> I"mn not sure which Chris you are refering to, but for my part, yes and no:

That's one of the perils of geeky communities - there'll often be
multiple people named Chris. I have a brother named Michael who's a
rail enthusiast, and that world has the same issue with his name - he
was once in a car with three other people named Michael.

> Yes, that would keep iterator classes and generator functions consistent,
> which would be a good thing.
>
> No: I don't think we should do that -- StopIteration is part of the iterator
> protocol -- generators are another way to write something that complies with
> the iterator protocol -- generators should handle StopIteration the same way
> that iterator classes do.

I've done my "explain it twice, then shut up" on this subject, so I'll
just point you to the list archive, where it's been stated clearly
that generators are like __iter__, not like __next__. Please, could
you respond to previously-given explanations, rather than simply
restating that generators should be like __next__? I'd like to move
forward with that discussion, rather than reiterating the same points.

ChrisA

From rosuav at gmail.com  Mon Nov 24 18:07:57 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 25 Nov 2014 04:07:57 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CALGmxELZjO5GPu9ZTNyxc9PatO59qOr7urDmj8Wo-O8zP2MgRQ@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info>
 <CALGmxEJHxSb-z4YQdci_HUFx42MxE3ooB8JHe+W_uwCmpPPsBQ@mail.gmail.com>
 <CAP7+vJ+fLsNm9sVdtRRxwLejoCu61V5TTFAJ1FiA8p+p2rLzrw@mail.gmail.com>
 <CALGmxELgihRDMD=5_xeW7CWwg_7HL9BXwUW+JH-A5VLyg=TJTA@mail.gmail.com>
 <CAPTjJmo-YEtZ0OOpNLf3X1N3y5tH5rp+T4_ybHwbaWBYDQjAUQ@mail.gmail.com>
 <20141122104827.GQ2748@ando.pearwood.info>
 <CAPTjJmqvvsbdPScMRcOB6Zx7SS+4xx04N=WAKYiySxiRh5+AfA@mail.gmail.com>
 <CALGmxELZjO5GPu9ZTNyxc9PatO59qOr7urDmj8Wo-O8zP2MgRQ@mail.gmail.com>
Message-ID: <CAPTjJmoYZeGo2VK5_Vz8NkuuJ-kgvXnvzen_PXGns2a55_BjZA@mail.gmail.com>

On Tue, Nov 25, 2014 at 3:41 AM, Chris Barker <chris.barker at noaa.gov> wrote:
> On Sat, Nov 22, 2014 at 2:57 AM, Chris Angelico <rosuav at gmail.com> wrote:
>>
>> If you have a lengthy nested chain of coroutines, and one of them
>> unexpectedly raises StopIteration, is it right for something to
>> quietly terminate, or should the exception bubble up and be printed to
>> console?
>
>
> Couldn't you have a nested pile of iterator classes as well that would
> exhibit the exact same behavior?

Potentially, but that would be a different thing. Also, I don't know
of cases where a __next__ function chains to a next() call through
arbitrary numbers of levels, but :"yield from" gets a solid work-out
in asyncio and related, so it's more likely to come up. But I don't
personally use asyncio, so I'd like to hear from someone who does.

ChrisA

From chris.barker at noaa.gov  Mon Nov 24 18:18:30 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 24 Nov 2014 09:18:30 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmrQWOkrukEgm6k-QkueaZu_zA1t5GpMHXNcNd-XzRcb7Q@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info>
 <CALGmxEJHxSb-z4YQdci_HUFx42MxE3ooB8JHe+W_uwCmpPPsBQ@mail.gmail.com>
 <CAP7+vJ+fLsNm9sVdtRRxwLejoCu61V5TTFAJ1FiA8p+p2rLzrw@mail.gmail.com>
 <CALGmxELgihRDMD=5_xeW7CWwg_7HL9BXwUW+JH-A5VLyg=TJTA@mail.gmail.com>
 <CAN3CYHw79pqQNJMJ40fOWVX8A9ZUvVRKSRwmg4cHsxg2-_o8iQ@mail.gmail.com>
 <CALGmxEKX3yeeA0P===h7AmLwOGTkAPCiCJPht6xX-6oiG+aPAg@mail.gmail.com>
 <CAPTjJmrQWOkrukEgm6k-QkueaZu_zA1t5GpMHXNcNd-XzRcb7Q@mail.gmail.com>
Message-ID: <CALGmxELizVetA5txCCkUB1uRwwjf5F3o93XvYaF7vZX2OH5VVg@mail.gmail.com>

On Mon, Nov 24, 2014 at 9:06 AM, Chris Angelico <rosuav at gmail.com> wrote:

> I've done my "explain it twice, then shut up" on this subject, so I'll
> just point you to the list archive, where it's been stated clearly
> that generators are like __iter__, not like __next__. Please, could
> you respond to previously-given explanations, rather than simply
> restating that generators should be like __next__?


I'm not sure if I've responded or not to previously given explanations --
but  you're right, it's time for me to shut up having made my point, too.

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141124/a8af2146/attachment.html>

From rosuav at gmail.com  Mon Nov 24 18:23:31 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 25 Nov 2014 04:23:31 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CALGmxELizVetA5txCCkUB1uRwwjf5F3o93XvYaF7vZX2OH5VVg@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info>
 <CALGmxEJHxSb-z4YQdci_HUFx42MxE3ooB8JHe+W_uwCmpPPsBQ@mail.gmail.com>
 <CAP7+vJ+fLsNm9sVdtRRxwLejoCu61V5TTFAJ1FiA8p+p2rLzrw@mail.gmail.com>
 <CALGmxELgihRDMD=5_xeW7CWwg_7HL9BXwUW+JH-A5VLyg=TJTA@mail.gmail.com>
 <CAN3CYHw79pqQNJMJ40fOWVX8A9ZUvVRKSRwmg4cHsxg2-_o8iQ@mail.gmail.com>
 <CALGmxEKX3yeeA0P===h7AmLwOGTkAPCiCJPht6xX-6oiG+aPAg@mail.gmail.com>
 <CAPTjJmrQWOkrukEgm6k-QkueaZu_zA1t5GpMHXNcNd-XzRcb7Q@mail.gmail.com>
 <CALGmxELizVetA5txCCkUB1uRwwjf5F3o93XvYaF7vZX2OH5VVg@mail.gmail.com>
Message-ID: <CAPTjJmrqo0VyaZ52e5idhwOjnP9gpxKGRqWP5WfSZFEPxkWVsA@mail.gmail.com>

On Tue, Nov 25, 2014 at 4:18 AM, Chris Barker <chris.barker at noaa.gov> wrote:
> On Mon, Nov 24, 2014 at 9:06 AM, Chris Angelico <rosuav at gmail.com> wrote:
>>
>> I've done my "explain it twice, then shut up" on this subject, so I'll
>> just point you to the list archive, where it's been stated clearly
>> that generators are like __iter__, not like __next__. Please, could
>> you respond to previously-given explanations, rather than simply
>> restating that generators should be like __next__?
>
>
> I'm not sure if I've responded or not to previously given explanations --
> but  you're right, it's time for me to shut up having made my point, too.

Well, there is probably more to be said about this - along the lines
of *why* generators ought to be more like iterators. (They're
iterables, not iterators.) It's just that we seem to be rehashing the
same arguments - or maybe that's just my impression, as there's been
discussion on three different channels (-ideas, -dev, and the issue
tracker - mercifully very little on the latter).

ChrisA

From random832 at fastmail.us  Mon Nov 24 18:37:39 2014
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Mon, 24 Nov 2014 12:37:39 -0500
Subject: [Python-ideas] Counter comparison
In-Reply-To: <CAP7+vJ+f=vRR_BuE=AR+54FkyWJfDnWaFZ7dXxkvqqBT6h_MJw@mail.gmail.com>
References: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>
 <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com>
 <1416809391.2601292.194579353.687F5B82@webmail.messagingengine.com>
 <CAP7+vJ+f=vRR_BuE=AR+54FkyWJfDnWaFZ7dXxkvqqBT6h_MJw@mail.gmail.com>
Message-ID: <1416850659.2181529.194813961.7C685009@webmail.messagingengine.com>

On Mon, Nov 24, 2014, at 10:43, Guido van Rossum wrote:
> But does it matter? If you've got a list of sets, and you want it sorted,
> you have to specify how you want it sorted anyways. You can complain that
> <
> is only a partial order, but that is the mathematical root of the
> situation, not a problem caused by an poorly chosen language default.
> (Leaving < undefined and sorting by object address, now *that* would be
> an
> example of the latter. :-)

You seem to be suggesting that sorting a list of partially ordered items
is not a meaningful thing to ask for, rather than merely not being
implemented by the list.sort method.

From solipsis at pitrou.net  Mon Nov 24 18:53:00 2014
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 24 Nov 2014 18:53:00 +0100
Subject: [Python-ideas] Counter comparison
References: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>
 <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com>
 <1416809391.2601292.194579353.687F5B82@webmail.messagingengine.com>
 <CAP7+vJ+f=vRR_BuE=AR+54FkyWJfDnWaFZ7dXxkvqqBT6h_MJw@mail.gmail.com>
 <1416850659.2181529.194813961.7C685009@webmail.messagingengine.com>
Message-ID: <20141124185300.69e8f871@fsol>

On Mon, 24 Nov 2014 12:37:39 -0500
random832 at fastmail.us wrote:
> On Mon, Nov 24, 2014, at 10:43, Guido van Rossum wrote:
> > But does it matter? If you've got a list of sets, and you want it sorted,
> > you have to specify how you want it sorted anyways. You can complain that
> > <
> > is only a partial order, but that is the mathematical root of the
> > situation, not a problem caused by an poorly chosen language default.
> > (Leaving < undefined and sorting by object address, now *that* would be
> > an
> > example of the latter. :-)
> 
> You seem to be suggesting that sorting a list of partially ordered items
> is not a meaningful thing to ask for, rather than merely not being
> implemented by the list.sort method.

Since list.sort is guaranteed to be stable, the sort order should
ideally be deterministic even in the face of partial ordering.

Regards

Antoine.



From alexander.belopolsky at gmail.com  Mon Nov 24 19:02:02 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Mon, 24 Nov 2014 13:02:02 -0500
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmrqo0VyaZ52e5idhwOjnP9gpxKGRqWP5WfSZFEPxkWVsA@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info>
 <CALGmxEJHxSb-z4YQdci_HUFx42MxE3ooB8JHe+W_uwCmpPPsBQ@mail.gmail.com>
 <CAP7+vJ+fLsNm9sVdtRRxwLejoCu61V5TTFAJ1FiA8p+p2rLzrw@mail.gmail.com>
 <CALGmxELgihRDMD=5_xeW7CWwg_7HL9BXwUW+JH-A5VLyg=TJTA@mail.gmail.com>
 <CAN3CYHw79pqQNJMJ40fOWVX8A9ZUvVRKSRwmg4cHsxg2-_o8iQ@mail.gmail.com>
 <CALGmxEKX3yeeA0P===h7AmLwOGTkAPCiCJPht6xX-6oiG+aPAg@mail.gmail.com>
 <CAPTjJmrQWOkrukEgm6k-QkueaZu_zA1t5GpMHXNcNd-XzRcb7Q@mail.gmail.com>
 <CALGmxELizVetA5txCCkUB1uRwwjf5F3o93XvYaF7vZX2OH5VVg@mail.gmail.com>
 <CAPTjJmrqo0VyaZ52e5idhwOjnP9gpxKGRqWP5WfSZFEPxkWVsA@mail.gmail.com>
Message-ID: <CAP7h-xYWa5s9pHDc5BpTbAigzEumdpxOu_axudtC-8eDzfVufw@mail.gmail.com>

On Mon, Nov 24, 2014 at 12:23 PM, Chris Angelico <rosuav at gmail.com> wrote:

> Well, there is probably more to be said about this - along the lines
> of *why* generators ought to be more like iterators. (They're
> iterables, not iterators.)
>

If generators are not iterators, how do you explain this?

>>> import collections
>>> def g():
...     yield 42
...
>>> isinstance(g(), collections.Iterator)
True
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141124/d63ea8c8/attachment.html>

From guido at python.org  Mon Nov 24 19:11:10 2014
From: guido at python.org (Guido van Rossum)
Date: Mon, 24 Nov 2014 10:11:10 -0800
Subject: [Python-ideas] Counter comparison
In-Reply-To: <1416850659.2181529.194813961.7C685009@webmail.messagingengine.com>
References: <CAGRr6BHXOykRA3dhMpGrE26ZON+jNemNxKKYoFP28=MJn1_AUQ@mail.gmail.com>
 <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com>
 <1416809391.2601292.194579353.687F5B82@webmail.messagingengine.com>
 <CAP7+vJ+f=vRR_BuE=AR+54FkyWJfDnWaFZ7dXxkvqqBT6h_MJw@mail.gmail.com>
 <1416850659.2181529.194813961.7C685009@webmail.messagingengine.com>
Message-ID: <CAP7+vJJCm4NUf+Yhi--on-t1+CYALi67SFgZV-jyGvYgQz=VFQ@mail.gmail.com>

On Mon, Nov 24, 2014 at 9:37 AM, <random832 at fastmail.us> wrote:

> On Mon, Nov 24, 2014, at 10:43, Guido van Rossum wrote:
> > But does it matter? If you've got a list of sets, and you want it sorted,
> > you have to specify how you want it sorted anyways. You can complain that
> > <
> > is only a partial order, but that is the mathematical root of the
> > situation, not a problem caused by an poorly chosen language default.
> > (Leaving < undefined and sorting by object address, now *that* would be
> > an
> > example of the latter. :-)
>
> You seem to be suggesting that sorting a list of partially ordered items
> is not a meaningful thing to ask for, rather than merely not being
> implemented by the list.sort method.
>

I assume you want a topological sort. But isn't that topic by itself a
useful math lesson?

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141124/b3130b04/attachment-0001.html>

From guido at python.org  Mon Nov 24 19:25:04 2014
From: guido at python.org (Guido van Rossum)
Date: Mon, 24 Nov 2014 10:25:04 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAP7h-xYWa5s9pHDc5BpTbAigzEumdpxOu_axudtC-8eDzfVufw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info>
 <CALGmxEJHxSb-z4YQdci_HUFx42MxE3ooB8JHe+W_uwCmpPPsBQ@mail.gmail.com>
 <CAP7+vJ+fLsNm9sVdtRRxwLejoCu61V5TTFAJ1FiA8p+p2rLzrw@mail.gmail.com>
 <CALGmxELgihRDMD=5_xeW7CWwg_7HL9BXwUW+JH-A5VLyg=TJTA@mail.gmail.com>
 <CAN3CYHw79pqQNJMJ40fOWVX8A9ZUvVRKSRwmg4cHsxg2-_o8iQ@mail.gmail.com>
 <CALGmxEKX3yeeA0P===h7AmLwOGTkAPCiCJPht6xX-6oiG+aPAg@mail.gmail.com>
 <CAPTjJmrQWOkrukEgm6k-QkueaZu_zA1t5GpMHXNcNd-XzRcb7Q@mail.gmail.com>
 <CALGmxELizVetA5txCCkUB1uRwwjf5F3o93XvYaF7vZX2OH5VVg@mail.gmail.com>
 <CAPTjJmrqo0VyaZ52e5idhwOjnP9gpxKGRqWP5WfSZFEPxkWVsA@mail.gmail.com>
 <CAP7h-xYWa5s9pHDc5BpTbAigzEumdpxOu_axudtC-8eDzfVufw@mail.gmail.com>
Message-ID: <CAP7+vJLXZc7HY81-27rhtvuPxHGZDQKBTmkmYquf6YMquL3cCQ@mail.gmail.com>

On Mon, Nov 24, 2014 at 10:02 AM, Alexander Belopolsky <
alexander.belopolsky at gmail.com> wrote:

>
> On Mon, Nov 24, 2014 at 12:23 PM, Chris Angelico <rosuav at gmail.com> wrote:
>
>> Well, there is probably more to be said about this - along the lines
>> of *why* generators ought to be more like iterators. (They're
>> iterables, not iterators.)
>>
>
> If generators are not iterators, how do you explain this?
>
> >>> import collections
> >>> def g():
> ...     yield 42
> ...
> >>> isinstance(g(), collections.Iterator)
> True
>

I think Chris A was overzealous here. The word "generator" is ambiguous; it
can refer to either a generator function (a function definition containing
at least one "yield") or to the object you obtain by calling a generator
function. The latter is definitely an iterator (it has a __next__ method).
You can't really call a generator function an iterable (since calling
iter() on it raises TypeError) but it's not an iterator either. For the
rest see my explanation in response to Mark Shannon in python-dev:
http://code.activestate.com/lists/python-dev/133428/

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141124/aca5631e/attachment.html>

From rosuav at gmail.com  Mon Nov 24 23:48:15 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 25 Nov 2014 09:48:15 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAP7h-xYWa5s9pHDc5BpTbAigzEumdpxOu_axudtC-8eDzfVufw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info>
 <CALGmxEJHxSb-z4YQdci_HUFx42MxE3ooB8JHe+W_uwCmpPPsBQ@mail.gmail.com>
 <CAP7+vJ+fLsNm9sVdtRRxwLejoCu61V5TTFAJ1FiA8p+p2rLzrw@mail.gmail.com>
 <CALGmxELgihRDMD=5_xeW7CWwg_7HL9BXwUW+JH-A5VLyg=TJTA@mail.gmail.com>
 <CAN3CYHw79pqQNJMJ40fOWVX8A9ZUvVRKSRwmg4cHsxg2-_o8iQ@mail.gmail.com>
 <CALGmxEKX3yeeA0P===h7AmLwOGTkAPCiCJPht6xX-6oiG+aPAg@mail.gmail.com>
 <CAPTjJmrQWOkrukEgm6k-QkueaZu_zA1t5GpMHXNcNd-XzRcb7Q@mail.gmail.com>
 <CALGmxELizVetA5txCCkUB1uRwwjf5F3o93XvYaF7vZX2OH5VVg@mail.gmail.com>
 <CAPTjJmrqo0VyaZ52e5idhwOjnP9gpxKGRqWP5WfSZFEPxkWVsA@mail.gmail.com>
 <CAP7h-xYWa5s9pHDc5BpTbAigzEumdpxOu_axudtC-8eDzfVufw@mail.gmail.com>
Message-ID: <CAPTjJmp7DwV4HXppZjOaeMvoAL4-mLmW28zne=aevA71Em040w@mail.gmail.com>

On Tue, Nov 25, 2014 at 5:02 AM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
> On Mon, Nov 24, 2014 at 12:23 PM, Chris Angelico <rosuav at gmail.com> wrote:
>>
>> Well, there is probably more to be said about this - along the lines
>> of *why* generators ought to be more like iterators. (They're
>> iterables, not iterators.)
>
>
> If generators are not iterators, how do you explain this?
>
>>>> import collections
>>>> def g():
> ...     yield 42
> ...
>>>> isinstance(g(), collections.Iterator)
> True

My apologies. As Guido said, "generator" is ambiguous; though I was
inaccurate as well. A generator *object* is, as you show above, an
iterator; a generator *function* is not actually iterable, but it is
an iterator factory. An iterator class is also an iterator factory.

ChrisA

From wolfgang.maier at biologie.uni-freiburg.de  Mon Nov 24 23:53:11 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Mon, 24 Nov 2014 23:53:11 +0100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
Message-ID: <m50cso$8h0$1@ger.gmane.org>

On 15.11.2014 10:29, Chris Angelico wrote:
> PEP: 479
> Title: Change StopIteration handling inside generators
> Version: $Revision$
> Last-Modified: $Date$
> Author: Chris Angelico <rosuav at gmail.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 15-Nov-2014
> Python-Version: 3.5
> Post-History: 15-Nov-2014
>
>
> Abstract
> ========
>
> This PEP proposes a semantic change to ``StopIteration`` when raised
> inside a generator, unifying the behaviour of list comprehensions and
> generator expressions somewhat.
>
>
> Rationale
> =========
>
> The interaction of generators and ``StopIteration`` is currently
> somewhat surprising, and can conceal obscure bugs.  An unexpected
> exception should not result in subtly altered behaviour, but should
> cause a noisy and easily-debugged traceback.  Currently,
> ``StopIteration`` can be absorbed by the generator construct.
>
>
> Proposal
> ========
>
> If a ``StopIteration`` is about to bubble out of a generator frame, it
> is replaced with some other exception (maybe ``RuntimeError``, maybe a
> new custom ``Exception`` subclass, but *not* deriving from
> ``StopIteration``) which causes the ``next()`` call (which invoked the
> generator) to fail, passing that exception out.  From then on it's
> just like any old exception. [3]_
>

Now that this PEP is going to be accepted, I'm not sure how much sense 
it still makes to suggest an amendment to it, but anyway:

As stated in the abstract one of the goals of the PEP is to unify 
further the behaviour of comprehensions and generator expressions.

With the PEP in place the following example (taken from Steven 
d'Aprano's post on python-list):

iterable = [iter([])]
list(next(x) for x in iterable)

would raise an error just like

[next(x) for x in iterable]

already does today.

However the comprehension currently raises StopIteration, while the 
proposed error for the generator expression would be of a different 
class (supposedly RuntimeError) - so comprehensions and generator 
expressions would still behave a bit (though much less) differently.

In addition, the PEP leaves an iterator's __next__() method as the only 
reasonable place where user-code should raise StopIteration.
So I would like to argue that instead of just turning StopIteration into 
some other error when it's about to bubble out of a generator frame, it 
should be converted whenever it bubbles out of *anything except an 
iterator's __next__()*. This would include comprehensions, but also any 
other code.

(On the side, I guess the current form of the PEP does address 
hard-to-debug bugs caused by nested generators, but what about nested 
__next__ in iterators ? Shouldn't it using the same logic also be an 
error if a next call inside a __next__ method raises an uncaught 
StopIteration ?)

I think such general behavior would make it much clearer that 
StopIteration is considered special and reserved for the iterator 
protocol. Of course, it might mean more broken code if people use 
StopIteration or a subclass for error signaling outside 
generator/iterators, but this PEP will mean backwards incompatibility 
anyway so why not go all the way and do it consistently.

I'm not sure I'd like the pretty general RuntimeError for this (even 
though Guido favors it for simplicity), instead one could call it 
UnhandledStopIteration ?
I imagine that a dedicated class would help in porting, for example, 
python2 code to python3 (which this PEP does not really simplify 
otherwise) since people/scripts could watch out for something specific ?

Thoughts?
Wolfgang




From rosuav at gmail.com  Tue Nov 25 00:03:46 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 25 Nov 2014 10:03:46 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <m50cso$8h0$1@ger.gmane.org>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <m50cso$8h0$1@ger.gmane.org>
Message-ID: <CAPTjJmpa+esr1ERKY2uEvYi3ouns_hJmfMd2WhwXQ-1zMh6g1g@mail.gmail.com>

On Tue, Nov 25, 2014 at 9:53 AM, Wolfgang Maier
<wolfgang.maier at biologie.uni-freiburg.de> wrote:
> In addition, the PEP leaves an iterator's __next__() method as the only
> reasonable place where user-code should raise StopIteration.
> So I would like to argue that instead of just turning StopIteration into
> some other error when it's about to bubble out of a generator frame, it
> should be converted whenever it bubbles out of *anything except an
> iterator's __next__()*. This would include comprehensions, but also any
> other code.

There'd have to be a special case for next(), where StopIteration is
part of the definition of the function. The question then becomes,
what's the boundary where StopIteration is converted?

The current proposal is quite simple. All the conversion happens in
the one function that (re)starts a generator frame, gen_send_ex() in
Objects/genobject.c. To do this for other functions, there'd need to
be some way of saying which ones are allowed to raise StopIteration
and which aren't.

Additionally, the scope for problems is smaller. A StopIteration
raised anywhere outside of a loop header won't cause silent behavioral
change; with generators, anywhere in the body of the function will
have that effect.

So, while I do agree in principle that it'd be nice, I don't know that
it's practical; however, it might be worth raising a dependent
proposal to extend this exception conversion.

ChrisA

From alexander.belopolsky at gmail.com  Tue Nov 25 00:20:11 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Mon, 24 Nov 2014 18:20:11 -0500
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmp7DwV4HXppZjOaeMvoAL4-mLmW28zne=aevA71Em040w@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info>
 <CALGmxEJHxSb-z4YQdci_HUFx42MxE3ooB8JHe+W_uwCmpPPsBQ@mail.gmail.com>
 <CAP7+vJ+fLsNm9sVdtRRxwLejoCu61V5TTFAJ1FiA8p+p2rLzrw@mail.gmail.com>
 <CALGmxELgihRDMD=5_xeW7CWwg_7HL9BXwUW+JH-A5VLyg=TJTA@mail.gmail.com>
 <CAN3CYHw79pqQNJMJ40fOWVX8A9ZUvVRKSRwmg4cHsxg2-_o8iQ@mail.gmail.com>
 <CALGmxEKX3yeeA0P===h7AmLwOGTkAPCiCJPht6xX-6oiG+aPAg@mail.gmail.com>
 <CAPTjJmrQWOkrukEgm6k-QkueaZu_zA1t5GpMHXNcNd-XzRcb7Q@mail.gmail.com>
 <CALGmxELizVetA5txCCkUB1uRwwjf5F3o93XvYaF7vZX2OH5VVg@mail.gmail.com>
 <CAPTjJmrqo0VyaZ52e5idhwOjnP9gpxKGRqWP5WfSZFEPxkWVsA@mail.gmail.com>
 <CAP7h-xYWa5s9pHDc5BpTbAigzEumdpxOu_axudtC-8eDzfVufw@mail.gmail.com>
 <CAPTjJmp7DwV4HXppZjOaeMvoAL4-mLmW28zne=aevA71Em040w@mail.gmail.com>
Message-ID: <CAP7h-xYsDG3sukPOdTBAZF1wXwv4U28nM_Jqr-aM0ZBoSruvvw@mail.gmail.com>

On Mon, Nov 24, 2014 at 5:48 PM, Chris Angelico <rosuav at gmail.com> wrote:
>
> As Guido said, "generator" is ambiguous; though I was
> inaccurate as well. A generator *object* is, as you show above, an
> iterator; a generator *function* is not actually iterable, but it is
> an iterator factory. An iterator class is also an iterator factory.


This is correct, and I don't think there is any ambiguity:

>>> def g(): yield 42
...
>>> type(g)
<class 'function'>
>>> type(g())
<class 'generator'>

As explained in PEP 255, "a Python generator is a kind of Python
iterator[1], but of an especially powerful kind."  The other term
introduced by PEP 255 is "generator function": "A function that contains a
yield statement is called a generator function."

In my view, PEP 479 naturally follows from careful reading of PEP 225.  All
one needs to understand is the difference between a function that returns
an iterator and its value.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141124/aabc9e7f/attachment-0001.html>

From python at 2sn.net  Tue Nov 25 00:23:22 2014
From: python at 2sn.net (Alexander Heger)
Date: Tue, 25 Nov 2014 10:23:22 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CALGmxEKX3yeeA0P===h7AmLwOGTkAPCiCJPht6xX-6oiG+aPAg@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com>
 <CAPTjJmqaBy6KyYRDcooM7xJ0biOMrVW40KW8D9NqOc4PF=rWYw@mail.gmail.com>
 <20141121163032.GN2748@ando.pearwood.info>
 <CALGmxEJHxSb-z4YQdci_HUFx42MxE3ooB8JHe+W_uwCmpPPsBQ@mail.gmail.com>
 <CAP7+vJ+fLsNm9sVdtRRxwLejoCu61V5TTFAJ1FiA8p+p2rLzrw@mail.gmail.com>
 <CALGmxELgihRDMD=5_xeW7CWwg_7HL9BXwUW+JH-A5VLyg=TJTA@mail.gmail.com>
 <CAN3CYHw79pqQNJMJ40fOWVX8A9ZUvVRKSRwmg4cHsxg2-_o8iQ@mail.gmail.com>
 <CALGmxEKX3yeeA0P===h7AmLwOGTkAPCiCJPht6xX-6oiG+aPAg@mail.gmail.com>
Message-ID: <CAN3CYHyEpkKoc6Aczd9v-+sfns2hvYn5G7wZ0vYu-dC4W64idA@mail.gmail.com>

>> Maybe a way to obtain equivalency to the generator functions in this
>> case is to "break" this example for the iterator object as well, in
>> that StopIteration has to be raised in the frame of the generator
>> object; if it raised in a different context, e.g., a function called
>> by __next__, that StopIteration should also be converted to a
>> RuntimeError similar to what is proposed in the PEP for the generator
>> functions.  Maybe this is not what Chris intends to happen, but it
>> would make things consistent.
>
> I"mn not sure which Chris you are refering to, but for my part, yes and no:

yes, this was a reply to your post

> Yes, that would keep iterator classes and generator functions consistent,
> which would be a good thing.

I think the main goal was to have a consistent interface that is easy
to debug and deals with StopIteration bubbling up - hence such
Exception from other scopes should convert to RuntimeError when
crossing the iterator interface boundary originating from other
scopes.

> No: I don't think we should do that -- StopIteration is part of the iterator
> protocol -- generators are another way to write something that complies with
> the iterator protocol -- generators should handle StopIteration the same way
> that iterator classes do.

You'd Keep StopIteration in the protocol, but only allow it in the local scope.

-Alexander

From wolfgang.maier at biologie.uni-freiburg.de  Tue Nov 25 18:30:07 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Tue, 25 Nov 2014 18:30:07 +0100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmpa+esr1ERKY2uEvYi3ouns_hJmfMd2WhwXQ-1zMh6g1g@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <m50cso$8h0$1@ger.gmane.org>
 <CAPTjJmpa+esr1ERKY2uEvYi3ouns_hJmfMd2WhwXQ-1zMh6g1g@mail.gmail.com>
Message-ID: <m52eab$6ln$1@ger.gmane.org>

On 11/25/2014 12:03 AM, Chris Angelico wrote:
> On Tue, Nov 25, 2014 at 9:53 AM, Wolfgang Maier
> <wolfgang.maier at biologie.uni-freiburg.de> wrote:
>> In addition, the PEP leaves an iterator's __next__() method as the only
>> reasonable place where user-code should raise StopIteration.
>> So I would like to argue that instead of just turning StopIteration into
>> some other error when it's about to bubble out of a generator frame, it
>> should be converted whenever it bubbles out of *anything except an
>> iterator's __next__()*. This would include comprehensions, but also any
>> other code.
>
> There'd have to be a special case for next(), where StopIteration is
> part of the definition of the function. The question then becomes,
> what's the boundary where StopIteration is converted?
>
> The current proposal is quite simple. All the conversion happens in
> the one function that (re)starts a generator frame, gen_send_ex() in
> Objects/genobject.c. To do this for other functions, there'd need to
> be some way of saying which ones are allowed to raise StopIteration
> and which aren't.
>

Well, I'm not familiar with every implementation detail of the 
interpreter so I can't judge how difficult to implement certain things 
would be, but one solution that I could think of is:

allow StopIteration to be raised anywhere, but let it bubble up only 
*one* frame.
So if the next outer frame does not deal with it, the exception would be 
converted to UnhandledStopIteration (or something else) when it's about 
to bubble out of that outer frame.
The builtin next() would simply reset the frame count by catching and 
reraising StopIteration raised inside its argument (whether that's an 
iterator's __next__ or a generator; note that in this scenario using 
raise StopIteration instead of return inside a generator would remain 
possible).

Examples of what would happen:

using next on a generator that raises StopIteration explicitly:
=> next catches the error and reraises StopIteration

using next on a generator that returns:
=> next behaves like currently, raising StopIteration

using next on the __next__ method of an iterator:
=> next catches the error and reraises StopIteration

every direct call of an iterator's __next__ method:
=> has to be guarded by a try/except StopIteration

Likewise in the first three cases, the calling frame, which resumes when 
next returns, (and only this frame) is given a chance to handle the 
error. If that doesn't happen (i.e. the error would bubble out) it gets 
converted.

So different from the current PEP where a StopIteration must be dealt 
with explicitly using try/except only inside generators, but bubbles up 
everywhere else, here StopIteration will be special everywhere, i.e., it 
must be passed upwards explicitly through all frames or will get converted.

Back to Steven's generator expression vs comprehension example:

iterable = [iter([])]
list(next(x) for x in iterable)

would raise UnhandledStopIteration since there is no way, inside the 
generator expression to catch the StopIteration raised by next(x).

... and if that's all complete nonsense because of some technical detail 
I'm not aware of, then please excuse my ignorance.

Wolfgang


From rosuav at gmail.com  Tue Nov 25 18:48:16 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 26 Nov 2014 04:48:16 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <m52eab$6ln$1@ger.gmane.org>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <m50cso$8h0$1@ger.gmane.org>
 <CAPTjJmpa+esr1ERKY2uEvYi3ouns_hJmfMd2WhwXQ-1zMh6g1g@mail.gmail.com>
 <m52eab$6ln$1@ger.gmane.org>
Message-ID: <CAPTjJmrM3r9hrZ1TeqWN7eforM_jV0e0XrpgZCo2QygVU1bjXA@mail.gmail.com>

On Wed, Nov 26, 2014 at 4:30 AM, Wolfgang Maier
<wolfgang.maier at biologie.uni-freiburg.de> wrote:
> Well, I'm not familiar with every implementation detail of the interpreter
> so I can't judge how difficult to implement certain things would be, but one
> solution that I could think of is:

I don't know much about the internal details of CPython either, but
let's just ignore that for the moment and consider specs for the
Python language. AFAIK, not one of the concerns raised (by PEP 479 or
your proposal here) is CPython-specific.

> allow StopIteration to be raised anywhere, but let it bubble up only *one*
> frame.
> So if the next outer frame does not deal with it, the exception would be
> converted to UnhandledStopIteration (or something else) when it's about to
> bubble out of that outer frame.
> The builtin next() would simply reset the frame count by catching and
> reraising StopIteration raised inside its argument (whether that's an
> iterator's __next__ or a generator; note that in this scenario using raise
> StopIteration instead of return inside a generator would remain possible).

Interesting. Makes a measure of sense, and doesn't have much magic to it.

> So different from the current PEP where a StopIteration must be dealt with
> explicitly using try/except only inside generators, but bubbles up
> everywhere else, here StopIteration will be special everywhere, i.e., it
> must be passed upwards explicitly through all frames or will get converted.
>
> Back to Steven's generator expression vs comprehension example:
>
> iterable = [iter([])]
> list(next(x) for x in iterable)
>
> would raise UnhandledStopIteration since there is no way, inside the
> generator expression to catch the StopIteration raised by next(x).

Downside of this is that it's harder to consciously chain iterators,
but maybe that's a cost that has to be paid.

Suggestion for this: Have a new way of "raise-and-return". It's mostly
like raise, except that (a) it can't be caught by a try/except block
in the current function (because there's no point), and (b) it
bypasses the "this exception must not pass unnoticed". It could then
also be used for anything else that needs the "return any object, or
signal lack of return value" option, covering AttributeError and so
on.

So it'd be something like this:

class X:
    def __iter__(self): return self
    def __next__(self):
        if condition: return value
        signal StopIteration

The 'signal' statement would promptly terminate the function (not sure
exactly how it'd interact with context managers and try/finally, but
something would be worked out), and then raise StopIteration in the
calling function. Any other StopIteration which passes out of a
function would become a RuntimeError.

Magic required: Some way of knowing which exceptions should be covered
by this ban on bubbling; also, preferably, some way to raise
StopIteration in the calling function, without losing the end of the
backtrace.

This could be a viable proposal. It'd be rather more complicated than
PEP 479, though, and would require a minimum of five hundred
bikeshedding posts before it comes to any kind of conclusion, but if
you feel this issue is worth it, I'd certainly be an interested
participant in the discussion.

ChrisA

From chris.barker at noaa.gov  Tue Nov 25 18:47:52 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Tue, 25 Nov 2014 09:47:52 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <m52eab$6ln$1@ger.gmane.org>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <m50cso$8h0$1@ger.gmane.org>
 <CAPTjJmpa+esr1ERKY2uEvYi3ouns_hJmfMd2WhwXQ-1zMh6g1g@mail.gmail.com>
 <m52eab$6ln$1@ger.gmane.org>
Message-ID: <CALGmxE+1uaETawZNVTvWYib7X2U3=cNRuu_z-v_oSo==+_jyrA@mail.gmail.com>

On Tue, Nov 25, 2014 at 9:30 AM, Wolfgang Maier <
wolfgang.maier at biologie.uni-freiburg.de> wrote:

> allow StopIteration to be raised anywhere, but let it bubble up only *one*
> frame.
> So if the next outer frame does not deal with it, the exception would be
> converted to UnhandledStopIteration (or something else) when it's about to
> bubble out of that outer frame.
> The builtin next() would simply reset the frame count by catching and
> reraising StopIteration raised inside its argument (whether that's an
> iterator's __next__ or a generator; note that in this scenario using raise
> StopIteration instead of return inside a generator would remain possible).
>

I also have no idea if this is practical from an implementation
perspective, but I like how it support my goal of keeping the behavior of
iterator classes and generators consistent.

-Chris




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141125/a4260a7b/attachment.html>

From guido at python.org  Tue Nov 25 18:53:23 2014
From: guido at python.org (Guido van Rossum)
Date: Tue, 25 Nov 2014 09:53:23 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmrM3r9hrZ1TeqWN7eforM_jV0e0XrpgZCo2QygVU1bjXA@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <m50cso$8h0$1@ger.gmane.org>
 <CAPTjJmpa+esr1ERKY2uEvYi3ouns_hJmfMd2WhwXQ-1zMh6g1g@mail.gmail.com>
 <m52eab$6ln$1@ger.gmane.org>
 <CAPTjJmrM3r9hrZ1TeqWN7eforM_jV0e0XrpgZCo2QygVU1bjXA@mail.gmail.com>
Message-ID: <CAP7+vJL5ASko-Sbe30+qPAorsuDg06=ghL=axXHT5s0bSuX5Rg@mail.gmail.com>

On Tue, Nov 25, 2014 at 9:48 AM, Chris Angelico <rosuav at gmail.com> wrote:

> On Wed, Nov 26, 2014 at 4:30 AM, Wolfgang Maier
> <wolfgang.maier at biologie.uni-freiburg.de> wrote:
> > Well, I'm not familiar with every implementation detail of the
> interpreter
> > so I can't judge how difficult to implement certain things would be, but
> one
> > solution that I could think of is:
>
> I don't know much about the internal details of CPython either, but
> let's just ignore that for the moment and consider specs for the
> Python language. AFAIK, not one of the concerns raised (by PEP 479 or
> your proposal here) is CPython-specific.
>
> > allow StopIteration to be raised anywhere, but let it bubble up only
> *one*
> > frame.
> > So if the next outer frame does not deal with it, the exception would be
> > converted to UnhandledStopIteration (or something else) when it's about
> to
> > bubble out of that outer frame.
> > The builtin next() would simply reset the frame count by catching and
> > reraising StopIteration raised inside its argument (whether that's an
> > iterator's __next__ or a generator; note that in this scenario using
> raise
> > StopIteration instead of return inside a generator would remain
> possible).
>
> Interesting. Makes a measure of sense, and doesn't have much magic to it.
>
> > So different from the current PEP where a StopIteration must be dealt
> with
> > explicitly using try/except only inside generators, but bubbles up
> > everywhere else, here StopIteration will be special everywhere, i.e., it
> > must be passed upwards explicitly through all frames or will get
> converted.
> >
> > Back to Steven's generator expression vs comprehension example:
> >
> > iterable = [iter([])]
> > list(next(x) for x in iterable)
> >
> > would raise UnhandledStopIteration since there is no way, inside the
> > generator expression to catch the StopIteration raised by next(x).
>
> Downside of this is that it's harder to consciously chain iterators,
> but maybe that's a cost that has to be paid.
>
> Suggestion for this: Have a new way of "raise-and-return". It's mostly
> like raise, except that (a) it can't be caught by a try/except block
> in the current function (because there's no point), and (b) it
> bypasses the "this exception must not pass unnoticed". It could then
> also be used for anything else that needs the "return any object, or
> signal lack of return value" option, covering AttributeError and so
> on.
>
> So it'd be something like this:
>
> class X:
>     def __iter__(self): return self
>     def __next__(self):
>         if condition: return value
>         signal StopIteration
>
> The 'signal' statement would promptly terminate the function (not sure
> exactly how it'd interact with context managers and try/finally, but
> something would be worked out), and then raise StopIteration in the
> calling function. Any other StopIteration which passes out of a
> function would become a RuntimeError.
>
> Magic required: Some way of knowing which exceptions should be covered
> by this ban on bubbling; also, preferably, some way to raise
> StopIteration in the calling function, without losing the end of the
> backtrace.
>
> This could be a viable proposal. It'd be rather more complicated than
> PEP 479, though, and would require a minimum of five hundred
> bikeshedding posts before it comes to any kind of conclusion, but if
> you feel this issue is worth it, I'd certainly be an interested
> participant in the discussion.
>

It's not viable. It will break more code than PEP 479, and it will incur a
larger interpreter overhead (every time any exception bubbles out of any
frame we'd have to check whether it is (derived from) StopIteration and
replace it, rather than only when exiting a generator frame. (The check for
StopIteration is relatively expensive -- it's easy to determine that an
exception *is* StopIteration, but in order that it doesn't derive from
StopIteration you have to walk the inheritance tree.)

Please stop panicking.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141125/5fffcaa0/attachment-0001.html>

From rosuav at gmail.com  Tue Nov 25 18:58:16 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 26 Nov 2014 04:58:16 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAP7+vJL5ASko-Sbe30+qPAorsuDg06=ghL=axXHT5s0bSuX5Rg@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <m50cso$8h0$1@ger.gmane.org>
 <CAPTjJmpa+esr1ERKY2uEvYi3ouns_hJmfMd2WhwXQ-1zMh6g1g@mail.gmail.com>
 <m52eab$6ln$1@ger.gmane.org>
 <CAPTjJmrM3r9hrZ1TeqWN7eforM_jV0e0XrpgZCo2QygVU1bjXA@mail.gmail.com>
 <CAP7+vJL5ASko-Sbe30+qPAorsuDg06=ghL=axXHT5s0bSuX5Rg@mail.gmail.com>
Message-ID: <CAPTjJmrXqh2yqrVUFevMRndyjFgNfj4ApmTVvD-KZZkwMGo+Qg@mail.gmail.com>

On Wed, Nov 26, 2014 at 4:53 AM, Guido van Rossum <guido at python.org> wrote:
> It's not viable. It will break more code than PEP 479, and it will incur a
> larger interpreter overhead (every time any exception bubbles out of any
> frame we'd have to check whether it is (derived from) StopIteration and
> replace it, rather than only when exiting a generator frame. (The check for
> StopIteration is relatively expensive -- it's easy to determine that an
> exception *is* StopIteration, but in order that it doesn't derive from
> StopIteration you have to walk the inheritance tree.)
>
> Please stop panicking.

Heh. If that last comment was aimed at me, I wasn't panicking... I
just thought it an intellectually curious concept, having a measure of
interpreter support for the "not-a-return-value" concept.

ChrisA

From guido at python.org  Tue Nov 25 18:59:14 2014
From: guido at python.org (Guido van Rossum)
Date: Tue, 25 Nov 2014 09:59:14 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CALGmxE+1uaETawZNVTvWYib7X2U3=cNRuu_z-v_oSo==+_jyrA@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <m50cso$8h0$1@ger.gmane.org>
 <CAPTjJmpa+esr1ERKY2uEvYi3ouns_hJmfMd2WhwXQ-1zMh6g1g@mail.gmail.com>
 <m52eab$6ln$1@ger.gmane.org>
 <CALGmxE+1uaETawZNVTvWYib7X2U3=cNRuu_z-v_oSo==+_jyrA@mail.gmail.com>
Message-ID: <CAP7+vJ+M6in67F8K13w_7hj7c9oanqX73kwk4+khG9D6rgd+bg@mail.gmail.com>

On Tue, Nov 25, 2014 at 9:47 AM, Chris Barker <chris.barker at noaa.gov> wrote:

>
> [...] I like how it support my goal of keeping the behavior of iterator
> classes and generators consistent.
>

This is a great summary of the general confusion I am trying to clear up.
The behavior of all types of iterators (including generators) from the
*caller's* perspective is not in question and is not changing. It is very
simple: you call next(it) (or it.__next__(), and it returns either the next
value or raises StopIteration (and any other exception is, indeed, an
exception).

>From the POV of *implementor*, iterator classes and generators have never
been at all alike -- one uses *return* to produce a new value, the other
uses *yield*. Returning without a value from a __next__() method is seen as
producing a None value by the caller; returning from a generator is
translated into a StopIteration which will be interpreted by the caller as
the end of series.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141125/766a94ad/attachment.html>

From guido at python.org  Tue Nov 25 19:00:26 2014
From: guido at python.org (Guido van Rossum)
Date: Tue, 25 Nov 2014 10:00:26 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAPTjJmrXqh2yqrVUFevMRndyjFgNfj4ApmTVvD-KZZkwMGo+Qg@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <m50cso$8h0$1@ger.gmane.org>
 <CAPTjJmpa+esr1ERKY2uEvYi3ouns_hJmfMd2WhwXQ-1zMh6g1g@mail.gmail.com>
 <m52eab$6ln$1@ger.gmane.org>
 <CAPTjJmrM3r9hrZ1TeqWN7eforM_jV0e0XrpgZCo2QygVU1bjXA@mail.gmail.com>
 <CAP7+vJL5ASko-Sbe30+qPAorsuDg06=ghL=axXHT5s0bSuX5Rg@mail.gmail.com>
 <CAPTjJmrXqh2yqrVUFevMRndyjFgNfj4ApmTVvD-KZZkwMGo+Qg@mail.gmail.com>
Message-ID: <CAP7+vJKBPfLkc22HSuqzM9UJLVwEJX_z7fvp6x9ckax=NMComQ@mail.gmail.com>

On Tue, Nov 25, 2014 at 9:58 AM, Chris Angelico <rosuav at gmail.com> wrote:

> On Wed, Nov 26, 2014 at 4:53 AM, Guido van Rossum <guido at python.org>
> wrote:
> > It's not viable. It will break more code than PEP 479, and it will incur
> a
> > larger interpreter overhead (every time any exception bubbles out of any
> > frame we'd have to check whether it is (derived from) StopIteration and
> > replace it, rather than only when exiting a generator frame. (The check
> for
> > StopIteration is relatively expensive -- it's easy to determine that an
> > exception *is* StopIteration, but in order that it doesn't derive from
> > StopIteration you have to walk the inheritance tree.)
> >
> > Please stop panicking.
>
> Heh. If that last comment was aimed at me, I wasn't panicking... I
> just thought it an intellectually curious concept, having a measure of
> interpreter support for the "not-a-return-value" concept.
>

OK, then I will just ignore this thread.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141125/1d76db7d/attachment.html>

From chris.barker at noaa.gov  Tue Nov 25 21:31:11 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Tue, 25 Nov 2014 12:31:11 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAP7+vJ+M6in67F8K13w_7hj7c9oanqX73kwk4+khG9D6rgd+bg@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <m50cso$8h0$1@ger.gmane.org>
 <CAPTjJmpa+esr1ERKY2uEvYi3ouns_hJmfMd2WhwXQ-1zMh6g1g@mail.gmail.com>
 <m52eab$6ln$1@ger.gmane.org>
 <CALGmxE+1uaETawZNVTvWYib7X2U3=cNRuu_z-v_oSo==+_jyrA@mail.gmail.com>
 <CAP7+vJ+M6in67F8K13w_7hj7c9oanqX73kwk4+khG9D6rgd+bg@mail.gmail.com>
Message-ID: <CALGmxELA=0aESiE=3L5J_wVv1i0kpnn_atzpP-w2pZKhNVyyHw@mail.gmail.com>

On Tue, Nov 25, 2014 at 9:59 AM, Guido van Rossum <guido at python.org> wrote:

> On Tue, Nov 25, 2014 at 9:47 AM, Chris Barker <chris.barker at noaa.gov>
> wrote:
>
>>
>> [...] I like how it support my goal of keeping the behavior of iterator
>> classes and generators consistent.
>>
>
> This is a great summary of the general confusion I am trying to clear up.
> The behavior of all types of iterators (including generators) from the
> *caller's* perspective is not in question and is not changing. It is very
> simple: you call next(it) (or it.__next__(), and it returns either the next
> value or raises StopIteration (and any other exception is, indeed, an
> exception).
>
> From the POV of *implementor*, iterator classes and generators have never
> been at all alike -- one uses *return* to produce a new value, the other
> uses *yield*. Returning without a value from a __next__() method is seen as
> producing a None value by the caller; returning from a generator is
> translated into a StopIteration which will be interpreted by the caller as
> the end of series.
>

Once you start nesting these things, the distinction between "implementor"
and "caller" gets mingled. And I think this is all about how nested
generators behave, yes?

If I am implementing a iterator of some sort (generator function or
iterator class), and I call next() inside my code, then I am both a
implementor and caller. And if I'm also writing helper functions, then I
need to know about how StopIteration will be handled, and it will be
handled a bit differently by generators and iterator classes.

But not a big deal, agreed, probably a much smaller deal that all the other
stuff you'd better understand to write this kind of code anyway.


-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141125/577800e7/attachment-0001.html>

From guido at python.org  Tue Nov 25 22:05:00 2014
From: guido at python.org (Guido van Rossum)
Date: Tue, 25 Nov 2014 13:05:00 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CALGmxELA=0aESiE=3L5J_wVv1i0kpnn_atzpP-w2pZKhNVyyHw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <m50cso$8h0$1@ger.gmane.org>
 <CAPTjJmpa+esr1ERKY2uEvYi3ouns_hJmfMd2WhwXQ-1zMh6g1g@mail.gmail.com>
 <m52eab$6ln$1@ger.gmane.org>
 <CALGmxE+1uaETawZNVTvWYib7X2U3=cNRuu_z-v_oSo==+_jyrA@mail.gmail.com>
 <CAP7+vJ+M6in67F8K13w_7hj7c9oanqX73kwk4+khG9D6rgd+bg@mail.gmail.com>
 <CALGmxELA=0aESiE=3L5J_wVv1i0kpnn_atzpP-w2pZKhNVyyHw@mail.gmail.com>
Message-ID: <CAP7+vJJnhGoXnVuhPzz92juRPzZjVUfDGvNBVS+2iQPKj38YXQ@mail.gmail.com>

On Tue, Nov 25, 2014 at 12:31 PM, Chris Barker <chris.barker at noaa.gov>
wrote:

> On Tue, Nov 25, 2014 at 9:59 AM, Guido van Rossum <guido at python.org>
> wrote:
>
>> On Tue, Nov 25, 2014 at 9:47 AM, Chris Barker <chris.barker at noaa.gov>
>> wrote:
>>
>>>
>>> [...] I like how it support my goal of keeping the behavior of iterator
>>> classes and generators consistent.
>>>
>>
>> This is a great summary of the general confusion I am trying to clear up.
>> The behavior of all types of iterators (including generators) from the
>> *caller's* perspective is not in question and is not changing. It is very
>> simple: you call next(it) (or it.__next__(), and it returns either the next
>> value or raises StopIteration (and any other exception is, indeed, an
>> exception).
>>
>> From the POV of *implementor*, iterator classes and generators have never
>> been at all alike -- one uses *return* to produce a new value, the other
>> uses *yield*. Returning without a value from a __next__() method is seen as
>> producing a None value by the caller; returning from a generator is
>> translated into a StopIteration which will be interpreted by the caller as
>> the end of series.
>>
>
> Once you start nesting these things, the distinction between "implementor"
> and "caller" gets mingled.
>

Hm. An implementer of one protocol is likely the caller of many other
protocols. It's not clear that calling something that implements the *same*
protocol should deserve special status. For example I could be implementing
an iterator processing the lines of a CSV file. Inside this iterator I may
be using another iterator that loops over the fields of the current line.
(This doesn't sound too far-fetched.) But if I run out of fields in a line,
why should that translate into terminating the outer iterator? And the
outer iterator may itself be called by another, more outer iterator that
iterators over a list of files.


> And I think this is all about how nested generators behave, yes?
>

The inner iterator doesn't have to be a generator (apart from send() and
throw(), they have the same interface).

And the point of the PEP is that an exhausted inner iterator shouldn't be
taken to automatically terminate the outer one.

(You could point out that I don't do anything about the similar problem
when the outer iterator is implemented as a class with a __next__() method.
If I could, I would -- but that case is different because there you *must*
raise StopIteration to terminate the iteration, so it becomes more similar
to an accidental KeyError being masked when it occurs inside a
__getitem__() method.)


> If I am implementing a iterator of some sort (generator function or
> iterator class), and I call next() inside my code, then I am both a
> implementor and caller. And if I'm also writing helper functions, then I
> need to know about how StopIteration will be handled, and it will be
> handled a bit differently by generators and iterator classes.
>

A helper function also defines an interface. If you are writing a helper
function for a generator (and the helper function is participating in the
same iteration as the outer generator, i.e. not in the CSV fies / lines /
fields example), the best way to do it is probably to write it as a helper
generator, and use "yield from" in the outer generator.


>
> But not a big deal, agreed, probably a much smaller deal that all the
> other stuff you'd better understand to write this kind of code anyway.
>

Which I'm sorry to see is much less widely understood than I had assumed.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141125/f71945d1/attachment.html>

From chris.barker at noaa.gov  Tue Nov 25 22:18:07 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Tue, 25 Nov 2014 13:18:07 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <CAP7+vJJnhGoXnVuhPzz92juRPzZjVUfDGvNBVS+2iQPKj38YXQ@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <m50cso$8h0$1@ger.gmane.org>
 <CAPTjJmpa+esr1ERKY2uEvYi3ouns_hJmfMd2WhwXQ-1zMh6g1g@mail.gmail.com>
 <m52eab$6ln$1@ger.gmane.org>
 <CALGmxE+1uaETawZNVTvWYib7X2U3=cNRuu_z-v_oSo==+_jyrA@mail.gmail.com>
 <CAP7+vJ+M6in67F8K13w_7hj7c9oanqX73kwk4+khG9D6rgd+bg@mail.gmail.com>
 <CALGmxELA=0aESiE=3L5J_wVv1i0kpnn_atzpP-w2pZKhNVyyHw@mail.gmail.com>
 <CAP7+vJJnhGoXnVuhPzz92juRPzZjVUfDGvNBVS+2iQPKj38YXQ@mail.gmail.com>
Message-ID: <CALGmxE+-avKQoQ+2AiFrbFQhSh5VvunZt2iOrEcGBL2VYzJGxw@mail.gmail.com>

On Tue, Nov 25, 2014 at 1:05 PM, Guido van Rossum <guido at python.org> wrote:

> (You could point out that I don't do anything about the similar problem
> when the outer iterator is implemented as a class with a __next__() method.
>

Indeed -- that is the key point here -- but you were pretty clear about how
special casing StopIteration is a non-starter.


> If I could, I would -- but that case is different because there you *must*
> raise StopIteration to terminate the iteration, so it becomes more similar
> to an accidental KeyError being masked when it occurs inside a
> __getitem__() method.
>

Well, I guess it a good thing to make things easier/clearer where you can
--  even if you can't do it everywhere.

I suppose if you think of generator functions as an easier way to write an
iterator (where it makes sense) then this is one more thing that makes it
even easier easier / safer. It does even more of the book keeping for you.
So a consistent win-win.

Thanks for the time taken clarifying your point.

But not a big deal, agreed, probably a much smaller deal that all the other
>> stuff you'd better understand to write this kind of code anyway.
>>
>
> Which I'm sorry to see is much less widely understood than I had assumed.
>

Well, this PEP does make for one less detail you need to understand (or
more to the point, keep in mind) when writing generator functions -- so
that's a good thing.

-Chris




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141125/4ba049ef/attachment.html>

From random832 at fastmail.us  Tue Nov 25 22:56:48 2014
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Tue, 25 Nov 2014 16:56:48 -0500
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
 generators
In-Reply-To: <CALGmxELA=0aESiE=3L5J_wVv1i0kpnn_atzpP-w2pZKhNVyyHw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <m50cso$8h0$1@ger.gmane.org>
 <CAPTjJmpa+esr1ERKY2uEvYi3ouns_hJmfMd2WhwXQ-1zMh6g1g@mail.gmail.com>
 <m52eab$6ln$1@ger.gmane.org>
 <CALGmxE+1uaETawZNVTvWYib7X2U3=cNRuu_z-v_oSo==+_jyrA@mail.gmail.com>
 <CAP7+vJ+M6in67F8K13w_7hj7c9oanqX73kwk4+khG9D6rgd+bg@mail.gmail.com>
 <CALGmxELA=0aESiE=3L5J_wVv1i0kpnn_atzpP-w2pZKhNVyyHw@mail.gmail.com>
Message-ID: <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com>

On Tue, Nov 25, 2014, at 15:31, Chris Barker wrote:
> Once you start nesting these things, the distinction between
> "implementor"
> and "caller" gets mingled. And I think this is all about how nested
> generators behave, yes?

For something more concrete, we can consider a naive implementation of
iteration over adjacent pairs:

def pairs(x):
    i = iter(x)
    while True:
        yield next(i), next(i)

>>> list(pairs(range(1)))
[]
>>> list(pairs(range(2)))
[(0, 1)]
>>> list(pairs(range(3)))
[(0, 1)]
>>> list(pairs(range(4)))
[(0, 1), (2, 3)]
>>> list(pairs(range(5)))
[(0, 1), (2, 3)]
>>> list(pairs(range(6)))
[(0, 1), (2, 3), (4, 5)]

To work under the new paradigm, you need to catch StopIteration
explicitly:

def pairs(x):
    i = iter(x)
    while True:
        try:
            a = next(i)
            b = next(i)
        except StopIteration:
            return
        yield a, b

From ian at feete.org  Wed Nov 26 00:19:01 2014
From: ian at feete.org (Ian Foote)
Date: Tue, 25 Nov 2014 23:19:01 +0000
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
 generators
In-Reply-To: <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <m50cso$8h0$1@ger.gmane.org>
 <CAPTjJmpa+esr1ERKY2uEvYi3ouns_hJmfMd2WhwXQ-1zMh6g1g@mail.gmail.com>
 <m52eab$6ln$1@ger.gmane.org>
 <CALGmxE+1uaETawZNVTvWYib7X2U3=cNRuu_z-v_oSo==+_jyrA@mail.gmail.com>
 <CAP7+vJ+M6in67F8K13w_7hj7c9oanqX73kwk4+khG9D6rgd+bg@mail.gmail.com>
 <CALGmxELA=0aESiE=3L5J_wVv1i0kpnn_atzpP-w2pZKhNVyyHw@mail.gmail.com>
 <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com>
Message-ID: <54750E65.7010602@feete.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 25/11/14 21:56, random832 at fastmail.us wrote:
> On Tue, Nov 25, 2014, at 15:31, Chris Barker wrote:
>> Once you start nesting these things, the distinction between 
>> "implementor" and "caller" gets mingled. And I think this is all
>> about how nested generators behave, yes?
> 
> For something more concrete, we can consider a naive implementation
> of iteration over adjacent pairs:
> 
> def pairs(x): i = iter(x) while True: yield next(i), next(i)
> 

<snip>

> 
> To work under the new paradigm, you need to catch StopIteration 
> explicitly:
> 
> def pairs(x): i = iter(x) while True: try: a = next(i) b = next(i) 
> except StopIteration: return yield a, b
> 

You're right that you need to catch the StopIteration, but it seems to
me the natural way to write your second example is:

def pairs(x):
    i = iter(x)
    while True:
        try:
            yield next(i), next(i)
        except StopIteration:
            return

Adding the extra variables a and b is unnecessary and distracts from
the change actually required.

Regards,
Ian F
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJUdQ5lAAoJEODsV4MF7PWz/zsIALLI67/W+HAG3l0Xe+kd2/Xw
QEI5NrOyT/izRHbV69K3zvOVKKCfiUXjkK5rPGxFiBmF96hOmQyro7Z4UiCSYzsT
N+8dy8M6/gAWolEbD1EZoXZorNHa9nsZ8q3hBltl824CAl4Kx7FFKssUVIFWjyrD
IgPjI4PIJBl12uX9F9VLMaBjfEy+QiCUa3a8s7ZdqS1asm1M4udei/qvt1t/NaIL
uoYGuBO/mzxP9sdWtP4z53sX07gOMPUWdBTXFX91+G1pCaUuGCHpDcHwexatxVgk
JxgfajyadFj+44QeHpsY10pu0HhaRH+Cbg7vADa2KQ6N3kQic1qHR9FkcHCvaA0=
=SdtN
-----END PGP SIGNATURE-----

From rosuav at gmail.com  Wed Nov 26 00:31:38 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 26 Nov 2014 10:31:38 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <m50cso$8h0$1@ger.gmane.org>
 <CAPTjJmpa+esr1ERKY2uEvYi3ouns_hJmfMd2WhwXQ-1zMh6g1g@mail.gmail.com>
 <m52eab$6ln$1@ger.gmane.org>
 <CALGmxE+1uaETawZNVTvWYib7X2U3=cNRuu_z-v_oSo==+_jyrA@mail.gmail.com>
 <CAP7+vJ+M6in67F8K13w_7hj7c9oanqX73kwk4+khG9D6rgd+bg@mail.gmail.com>
 <CALGmxELA=0aESiE=3L5J_wVv1i0kpnn_atzpP-w2pZKhNVyyHw@mail.gmail.com>
 <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com>
Message-ID: <CAPTjJmo7BmJOLBS0=anoPuqemT4UubvVUj8nMWXR3bs1up82Nw@mail.gmail.com>

On Wed, Nov 26, 2014 at 8:56 AM,  <random832 at fastmail.us> wrote:
> For something more concrete, we can consider a naive implementation of
> iteration over adjacent pairs:
>
> def pairs(x):
>     i = iter(x)
>     while True:
>         yield next(i), next(i)

Okay, it's simple and naive. How about this version:

def pairs(x):
    i = iter(x)
    for val in i:
        yield val, next(i)

Also simple, but subtly different from your version. What's the
difference? Will it be obvious to everyone who reads it?

ChrisA

From ethan at stoneleaf.us  Wed Nov 26 00:46:00 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 25 Nov 2014 15:46:00 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
 generators
In-Reply-To: <CAPTjJmo7BmJOLBS0=anoPuqemT4UubvVUj8nMWXR3bs1up82Nw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <m50cso$8h0$1@ger.gmane.org>
 <CAPTjJmpa+esr1ERKY2uEvYi3ouns_hJmfMd2WhwXQ-1zMh6g1g@mail.gmail.com>
 <m52eab$6ln$1@ger.gmane.org>
 <CALGmxE+1uaETawZNVTvWYib7X2U3=cNRuu_z-v_oSo==+_jyrA@mail.gmail.com>
 <CAP7+vJ+M6in67F8K13w_7hj7c9oanqX73kwk4+khG9D6rgd+bg@mail.gmail.com>
 <CALGmxELA=0aESiE=3L5J_wVv1i0kpnn_atzpP-w2pZKhNVyyHw@mail.gmail.com>
 <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com>
 <CAPTjJmo7BmJOLBS0=anoPuqemT4UubvVUj8nMWXR3bs1up82Nw@mail.gmail.com>
Message-ID: <547514B8.4090703@stoneleaf.us>

On 11/25/2014 03:31 PM, Chris Angelico wrote:
> On Wed, Nov 26, 2014 at 8:56 AM,  <random832 at fastmail.us> wrote:
>> For something more concrete, we can consider a naive implementation of
>> iteration over adjacent pairs:
>>
>> def pairs(x):
>>     i = iter(x)
>>     while True:
>>         yield next(i), next(i)
> 
> Okay, it's simple and naive. How about this version:
> 
> def pairs(x):
>     i = iter(x)
>     for val in i:
>         yield val, next(i)
> 
> Also simple, but subtly different from your version. What's the
> difference? Will it be obvious to everyone who reads it?

I don't see the difference being subtle enough -- if an odd number of items is tossed in, that `next(i)` is still going
to raise a StopIteration, which under PEP 479 will become a RunTimeError.

Or did you mean that even numbered iterators will work fine, but odd-numbered ones will still raise?  Nice.  :)

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141125/d24d03f5/attachment.sig>

From rosuav at gmail.com  Wed Nov 26 01:37:17 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 26 Nov 2014 11:37:17 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <547514B8.4090703@stoneleaf.us>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <m50cso$8h0$1@ger.gmane.org>
 <CAPTjJmpa+esr1ERKY2uEvYi3ouns_hJmfMd2WhwXQ-1zMh6g1g@mail.gmail.com>
 <m52eab$6ln$1@ger.gmane.org>
 <CALGmxE+1uaETawZNVTvWYib7X2U3=cNRuu_z-v_oSo==+_jyrA@mail.gmail.com>
 <CAP7+vJ+M6in67F8K13w_7hj7c9oanqX73kwk4+khG9D6rgd+bg@mail.gmail.com>
 <CALGmxELA=0aESiE=3L5J_wVv1i0kpnn_atzpP-w2pZKhNVyyHw@mail.gmail.com>
 <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com>
 <CAPTjJmo7BmJOLBS0=anoPuqemT4UubvVUj8nMWXR3bs1up82Nw@mail.gmail.com>
 <547514B8.4090703@stoneleaf.us>
Message-ID: <CAPTjJmoWueAmL5147o-QR51vb-KSK9jtbqg7MZ_fzEzeuBMehw@mail.gmail.com>

On Wed, Nov 26, 2014 at 10:46 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> I don't see the difference being subtle enough -- if an odd number of items is tossed in, that `next(i)` is still going
> to raise a StopIteration, which under PEP 479 will become a RunTimeError.
>
> Or did you mean that even numbered iterators will work fine, but odd-numbered ones will still raise?  Nice.  :)

Yes, I meant the silent termination of an iterable with an odd number
of items. That's pretty subtle.

ChrisA

From ethan at stoneleaf.us  Wed Nov 26 01:46:31 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 25 Nov 2014 16:46:31 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
 generators
In-Reply-To: <CAPTjJmoWueAmL5147o-QR51vb-KSK9jtbqg7MZ_fzEzeuBMehw@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <m50cso$8h0$1@ger.gmane.org>
 <CAPTjJmpa+esr1ERKY2uEvYi3ouns_hJmfMd2WhwXQ-1zMh6g1g@mail.gmail.com>
 <m52eab$6ln$1@ger.gmane.org>
 <CALGmxE+1uaETawZNVTvWYib7X2U3=cNRuu_z-v_oSo==+_jyrA@mail.gmail.com>
 <CAP7+vJ+M6in67F8K13w_7hj7c9oanqX73kwk4+khG9D6rgd+bg@mail.gmail.com>
 <CALGmxELA=0aESiE=3L5J_wVv1i0kpnn_atzpP-w2pZKhNVyyHw@mail.gmail.com>
 <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com>
 <CAPTjJmo7BmJOLBS0=anoPuqemT4UubvVUj8nMWXR3bs1up82Nw@mail.gmail.com>
 <547514B8.4090703@stoneleaf.us>
 <CAPTjJmoWueAmL5147o-QR51vb-KSK9jtbqg7MZ_fzEzeuBMehw@mail.gmail.com>
Message-ID: <547522E7.6050301@stoneleaf.us>

On 11/25/2014 04:37 PM, Chris Angelico wrote:
> On Wed, Nov 26, 2014 at 10:46 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
>> I don't see the difference being subtle enough -- if an odd number of items is tossed in, that `next(i)` is still going
>> to raise a StopIteration, which under PEP 479 will become a RunTimeError.
>>
>> Or did you mean that even numbered iterators will work fine, but odd-numbered ones will still raise?  Nice.  :)
> 
> Yes, I meant the silent termination of an iterable with an odd number
> of items. That's pretty subtle.

You mean an even number of items, yes?

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141125/39430182/attachment.sig>

From rosuav at gmail.com  Wed Nov 26 01:56:46 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 26 Nov 2014 11:56:46 +1100
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <547522E7.6050301@stoneleaf.us>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <m50cso$8h0$1@ger.gmane.org>
 <CAPTjJmpa+esr1ERKY2uEvYi3ouns_hJmfMd2WhwXQ-1zMh6g1g@mail.gmail.com>
 <m52eab$6ln$1@ger.gmane.org>
 <CALGmxE+1uaETawZNVTvWYib7X2U3=cNRuu_z-v_oSo==+_jyrA@mail.gmail.com>
 <CAP7+vJ+M6in67F8K13w_7hj7c9oanqX73kwk4+khG9D6rgd+bg@mail.gmail.com>
 <CALGmxELA=0aESiE=3L5J_wVv1i0kpnn_atzpP-w2pZKhNVyyHw@mail.gmail.com>
 <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com>
 <CAPTjJmo7BmJOLBS0=anoPuqemT4UubvVUj8nMWXR3bs1up82Nw@mail.gmail.com>
 <547514B8.4090703@stoneleaf.us>
 <CAPTjJmoWueAmL5147o-QR51vb-KSK9jtbqg7MZ_fzEzeuBMehw@mail.gmail.com>
 <547522E7.6050301@stoneleaf.us>
Message-ID: <CAPTjJmpU5Cx1WjkRhkuxcbAX0zrL99uon0S=A3kSxQ=eqViz=Q@mail.gmail.com>

On Wed, Nov 26, 2014 at 11:46 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 11/25/2014 04:37 PM, Chris Angelico wrote:
>> On Wed, Nov 26, 2014 at 10:46 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
>>> I don't see the difference being subtle enough -- if an odd number of items is tossed in, that `next(i)` is still going
>>> to raise a StopIteration, which under PEP 479 will become a RunTimeError.
>>>
>>> Or did you mean that even numbered iterators will work fine, but odd-numbered ones will still raise?  Nice.  :)
>>
>> Yes, I meant the silent termination of an iterable with an odd number
>> of items. That's pretty subtle.
>
> You mean an even number of items, yes?

Presumably the even case is the correct one. It's intended to work
that way. If you give it an odd number of items, pre-479 they'll both
silently terminate. (Post-479, the next(),next() one will always raise
RuntimeError, which indicates clearly that it's not written
appropriately, but that's not subtle.)

ChrisA

From apalala at gmail.com  Wed Nov 26 04:51:25 2014
From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=)
Date: Tue, 25 Nov 2014 23:21:25 -0430
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <m50cso$8h0$1@ger.gmane.org>
 <CAPTjJmpa+esr1ERKY2uEvYi3ouns_hJmfMd2WhwXQ-1zMh6g1g@mail.gmail.com>
 <m52eab$6ln$1@ger.gmane.org>
 <CALGmxE+1uaETawZNVTvWYib7X2U3=cNRuu_z-v_oSo==+_jyrA@mail.gmail.com>
 <CAP7+vJ+M6in67F8K13w_7hj7c9oanqX73kwk4+khG9D6rgd+bg@mail.gmail.com>
 <CALGmxELA=0aESiE=3L5J_wVv1i0kpnn_atzpP-w2pZKhNVyyHw@mail.gmail.com>
 <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com>
Message-ID: <CAN1YFWsz0HUaaBWcrEby2_+LqK-O5UyzzzUfPPz7K4BqdA3atA@mail.gmail.com>

On Tue, Nov 25, 2014 at 5:26 PM, <random832 at fastmail.us> wrote:

> def pairs(x):
>     i = iter(x)
>     while True:
>         yield next(i), next(i)
>

That's not too pythonic, and trying to support non-pythonic code while
evolving the language is a dead-end street.

The web documents pythonic ways to obtain pairs from an iterable:

def pairs(x):
    i = iter(x)
    return zip(i, i)

Or even:

def pairs(x):
     return zip(*[iter(x)]*2)

The usual way of dealing with an odd number of elements is to use
zip_longest.

I don't remember seeing documented that raising StopIteration will cancel
more than one iterator. If it's undocumented, then code that relies on the
non-feature is broken.

Cheers,

-- 
Juancarlo *A?ez*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141125/675ee6a3/attachment.html>

From random832 at fastmail.us  Wed Nov 26 14:59:34 2014
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Wed, 26 Nov 2014 08:59:34 -0500
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
 generators
In-Reply-To: <CAN1YFWsz0HUaaBWcrEby2_+LqK-O5UyzzzUfPPz7K4BqdA3atA@mail.gmail.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <m50cso$8h0$1@ger.gmane.org>
 <CAPTjJmpa+esr1ERKY2uEvYi3ouns_hJmfMd2WhwXQ-1zMh6g1g@mail.gmail.com>
 <m52eab$6ln$1@ger.gmane.org>
 <CALGmxE+1uaETawZNVTvWYib7X2U3=cNRuu_z-v_oSo==+_jyrA@mail.gmail.com>
 <CAP7+vJ+M6in67F8K13w_7hj7c9oanqX73kwk4+khG9D6rgd+bg@mail.gmail.com>
 <CALGmxELA=0aESiE=3L5J_wVv1i0kpnn_atzpP-w2pZKhNVyyHw@mail.gmail.com>
 <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com>
 <CAN1YFWsz0HUaaBWcrEby2_+LqK-O5UyzzzUfPPz7K4BqdA3atA@mail.gmail.com>
Message-ID: <1417010374.2937928.195655729.58930BAB@webmail.messagingengine.com>

On Tue, Nov 25, 2014, at 22:51, Juancarlo A?ez wrote:
> On Tue, Nov 25, 2014 at 5:26 PM, <random832 at fastmail.us> wrote:
> 
> > def pairs(x):
> >     i = iter(x)
> >     while True:
> >         yield next(i), next(i)
> >
> 
> That's not too pythonic, and trying to support non-pythonic code while
> evolving the language is a dead-end street.

Out of curiosity, what explicit uses of next are pythonic?

> The web documents pythonic ways to obtain pairs from an iterable:
> 
> def pairs(x):
>     i = iter(x)
>     return zip(i, i)
> 
> Or even:
> 
> def pairs(x):
>      return zip(*[iter(x)]*2)

See, both of those seem way too implicit to me.

From guido at python.org  Wed Nov 26 18:28:59 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 26 Nov 2014 09:28:59 -0800
Subject: [Python-ideas] PEP 479: Change StopIteration handling inside
	generators
In-Reply-To: <1417010374.2937928.195655729.58930BAB@webmail.messagingengine.com>
References: <CAPTjJmqNSvvxMpWOSpDmbtCcp0xkiOdir0QctD+PofCR411miw@mail.gmail.com>
 <m50cso$8h0$1@ger.gmane.org>
 <CAPTjJmpa+esr1ERKY2uEvYi3ouns_hJmfMd2WhwXQ-1zMh6g1g@mail.gmail.com>
 <m52eab$6ln$1@ger.gmane.org>
 <CALGmxE+1uaETawZNVTvWYib7X2U3=cNRuu_z-v_oSo==+_jyrA@mail.gmail.com>
 <CAP7+vJ+M6in67F8K13w_7hj7c9oanqX73kwk4+khG9D6rgd+bg@mail.gmail.com>
 <CALGmxELA=0aESiE=3L5J_wVv1i0kpnn_atzpP-w2pZKhNVyyHw@mail.gmail.com>
 <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com>
 <CAN1YFWsz0HUaaBWcrEby2_+LqK-O5UyzzzUfPPz7K4BqdA3atA@mail.gmail.com>
 <1417010374.2937928.195655729.58930BAB@webmail.messagingengine.com>
Message-ID: <CAP7+vJLRLrv3Wh5Zwpz6o7ObFFey_13bpUmw1xGCdkAoXrR4aw@mail.gmail.com>

On Wed, Nov 26, 2014 at 5:59 AM, <random832 at fastmail.us> wrote:

> Out of curiosity, what explicit uses of next are pythonic?


Ones immediately enclosed in try/except StopIteration, e.g.:

  try:
      x = next(it)
      print(x)
  except StopIteration:
      print('nothing')

You could rewrite this particular one as follows:

  for x in it:
      print(x)
      break
  else:
      print('nothing')

But if you have multiple next() calls you might be able to have a single
try/except catching StopIteration from all of them, so the first pattern is
more general.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141126/d519c108/attachment.html>

From paul at colomiets.name  Wed Nov 26 18:35:23 2014
From: paul at colomiets.name (Paul Colomiets)
Date: Wed, 26 Nov 2014 19:35:23 +0200
Subject: [Python-ideas] Asynchronous IO ideas for Python
Message-ID: <CAA0gF6ranMazGaH1XpQU8Hp2S-SsU2n_XT=xZ_j+tFMXczezFg@mail.gmail.com>

Hi,

I've written an article about how I perceive the future of
asynchronous I/O in Python. It's not something that should directly be
incorporated into python now, but I believe it's useful for
python-ideas list.

https://medium.com/@paulcolomiets/the-future-of-asynchronous-io-in-python-ce200536d847

And a place for comments at Hacker News:

https://news.ycombinator.com/item?id=8662782

I hope being helpful with this writeup :)

-- 
Paul

From trent at snakebite.org  Wed Nov 26 21:56:07 2014
From: trent at snakebite.org (Trent Nelson)
Date: Wed, 26 Nov 2014 12:56:07 -0800
Subject: [Python-ideas] Asynchronous IO ideas for Python
References: <CAA0gF6ranMazGaH1XpQU8Hp2S-SsU2n_XT=xZ_j+tFMXczezFg@mail.gmail.com> 
Message-ID: <5E3A89398864414E8EE2CB192E0C965C19A3508EDB@EXMBX10.exchhosting.com>

Relevant part of the video with the normal Python stats on the left and PyParallel on the right:

https://www.youtube.com/watch?v=4L4Ww3ROuro#t=838

Transcribed stats:

Regular Python HTTP server:

Thread Stats	Avg	Stdev	Max
 Latency	4.93ms	714us	10ms
 Req/Seq	552	154	1.1k
10,480 requests in 10s, 69MB
1048 reqs/sec, 6.9MB/s	

PyParallel (4 core Windows VM):

Thread Stats	Avg	Stdev	Max
 Latency	2.41ms	531us	10ms
 Req/Seq	1.74k	183	2.33k
32,831 requests in 10s, 216MB
3263 reqs/sec, 21MB/s

So basically a bit less than linear scaling with more cores, which isn't too bad for a full debug build running on a VM.

-----Original Message-----
From: Trent Nelson 
Sent: Wednesday, November 26, 2014 3:36 PM
To: 'Paul Colomiets'; python-ideas
Subject: RE: [Python-ideas] Asynchronous IO ideas for Python

Have you seen this?:

	https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores

I spend the first 80-ish slides on async I/O.

(That was a year ago.  I've done 2-3 sprints on it since then and have gotten it to a point where I can back up the claims with hard numbers on load testing benchmarks, demonstrated in the most recent video: https://www.youtube.com/watch?v=4L4Ww3ROuro.)


	Trent.

-----Original Message-----
From: Python-ideas [mailto:python-ideas-bounces+trent=snakebite.org at python.org] On Behalf Of Paul Colomiets
Sent: Wednesday, November 26, 2014 12:35 PM
To: python-ideas
Subject: [Python-ideas] Asynchronous IO ideas for Python

Hi,

I've written an article about how I perceive the future of asynchronous I/O in Python. It's not something that should directly be incorporated into python now, but I believe it's useful for python-ideas list.

https://medium.com/@paulcolomiets/the-future-of-asynchronous-io-in-python-ce200536d847

And a place for comments at Hacker News:

https://news.ycombinator.com/item?id=8662782

I hope being helpful with this writeup :)

--
Paul
_______________________________________________
Python-ideas mailing list
Python-ideas at python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

From guido at python.org  Wed Nov 26 22:48:41 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 26 Nov 2014 13:48:41 -0800
Subject: [Python-ideas] Asynchronous IO ideas for Python
In-Reply-To: <5E3A89398864414E8EE2CB192E0C965C19A3508EDB@EXMBX10.exchhosting.com>
References: <CAA0gF6ranMazGaH1XpQU8Hp2S-SsU2n_XT=xZ_j+tFMXczezFg@mail.gmail.com>
 <5E3A89398864414E8EE2CB192E0C965C19A3508EDB@EXMBX10.exchhosting.com>
Message-ID: <CAP7+vJK-rSER6-Xyg7dLraXa5BbBJ=hRW_n65TDX5yuRrJWBxw@mail.gmail.com>

Trent,

Can you post source for the regular and pyparallel HTTP servers you used?

On Wed, Nov 26, 2014 at 12:56 PM, Trent Nelson <trent at snakebite.org> wrote:

> Relevant part of the video with the normal Python stats on the left and
> PyParallel on the right:
>
> https://www.youtube.com/watch?v=4L4Ww3ROuro#t=838
>
> Transcribed stats:
>
> Regular Python HTTP server:
>
> Thread Stats    Avg     Stdev   Max
>  Latency        4.93ms  714us   10ms
>  Req/Seq        552     154     1.1k
> 10,480 requests in 10s, 69MB
> 1048 reqs/sec, 6.9MB/s
>
> PyParallel (4 core Windows VM):
>
> Thread Stats    Avg     Stdev   Max
>  Latency        2.41ms  531us   10ms
>  Req/Seq        1.74k   183     2.33k
> 32,831 requests in 10s, 216MB
> 3263 reqs/sec, 21MB/s
>
> So basically a bit less than linear scaling with more cores, which isn't
> too bad for a full debug build running on a VM.
>
> -----Original Message-----
> From: Trent Nelson
> Sent: Wednesday, November 26, 2014 3:36 PM
> To: 'Paul Colomiets'; python-ideas
> Subject: RE: [Python-ideas] Asynchronous IO ideas for Python
>
> Have you seen this?:
>
>
> https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores
>
> I spend the first 80-ish slides on async I/O.
>
> (That was a year ago.  I've done 2-3 sprints on it since then and have
> gotten it to a point where I can back up the claims with hard numbers on
> load testing benchmarks, demonstrated in the most recent video:
> https://www.youtube.com/watch?v=4L4Ww3ROuro.)
>
>
>         Trent.
>
> -----Original Message-----
> From: Python-ideas [mailto:python-ideas-bounces+trent=
> snakebite.org at python.org] On Behalf Of Paul Colomiets
> Sent: Wednesday, November 26, 2014 12:35 PM
> To: python-ideas
> Subject: [Python-ideas] Asynchronous IO ideas for Python
>
> Hi,
>
> I've written an article about how I perceive the future of asynchronous
> I/O in Python. It's not something that should directly be incorporated into
> python now, but I believe it's useful for python-ideas list.
>
>
> https://medium.com/@paulcolomiets/the-future-of-asynchronous-io-in-python-ce200536d847
>
> And a place for comments at Hacker News:
>
> https://news.ycombinator.com/item?id=8662782
>
> I hope being helpful with this writeup :)
>
> --
> Paul
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141126/53c2e48c/attachment-0001.html>

From robertc at robertcollins.net  Thu Nov 27 00:45:39 2014
From: robertc at robertcollins.net (Robert Collins)
Date: Thu, 27 Nov 2014 12:45:39 +1300
Subject: [Python-ideas] Extending traceback to (optionally) format and show
	locals.
Message-ID: <CAJ3HoZ3it8x2KTyuOS5fz7zdDbjfi0hSat=VNc-NvC8BFa=WwA@mail.gmail.com>

For context please see http://bugs.python.org/issue22937 and
http://bugs.python.org/issue22936.

I have two questions I'm hoping to get answered through this thread:
 - does the change in question need a PEP? Antoine seemed to think it
didn't, as long as it was opt-in (via the unittest CLI etc)
 - Is my implementation approach sound (for traceback, unittest I
think I have covered :))?


Implementation wise, I think its useful to work in the current
traceback module layout - that is to alter extract_stack to
(optionally) include rendered data about locals and then look for that
and format it in format_list.

I'm sure there is code out there that depends on the quadruple nature
of extract_stack though, so I think we need to preserve that. Three
strategies occured to me; one is to have parallel functions, one
quadruple, one quintuple. A second one is to have the return value of
extract_stack be a quintuple when a new keyword parameter
include_locals is included. Lastly, and this is my preferred one, if
we return a tuple subclass with an attribute containing a dict with
the rendered data on the locals; this can be present but None, or even
just absent when extract_stack was not asked to include locals.

The last option is my preferred one because the other two both imply
having a data structure which is likely to break existing code - and
while you'd have to opt into having them it seems likely to require a
systematic set of upgrades vs having an optional attribute that can be
looked up.

So - thoughts?

-Rob


-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud

From paul at colomiets.name  Thu Nov 27 01:16:36 2014
From: paul at colomiets.name (Paul Colomiets)
Date: Thu, 27 Nov 2014 02:16:36 +0200
Subject: [Python-ideas] Asynchronous IO ideas for Python
In-Reply-To: <52564bb8db944de58d5d5df2357a2f05@BLUPR02MB341.namprd02.prod.outlook.com>
References: <CAA0gF6ranMazGaH1XpQU8Hp2S-SsU2n_XT=xZ_j+tFMXczezFg@mail.gmail.com>
 <52564bb8db944de58d5d5df2357a2f05@BLUPR02MB341.namprd02.prod.outlook.com>
Message-ID: <CAA0gF6q0Fhsp+idTnZwtJKirVFY8Z8S-iHZtSq9fk++fR7PTUQ@mail.gmail.com>

Hi Trent,

On Wed, Nov 26, 2014 at 10:36 PM, Trent Nelson <trent at trent.me> wrote:
> Have you seen this?:
>
>         https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores
>
> I spend the first 80-ish slides on async I/O.
>
> (That was a year ago.  I've done 2-3 sprints on it since then and have gotten it to a point where I can back up the claims with hard numbers on load testing benchmarks, demonstrated in the most recent video: https://www.youtube.com/watch?v=4L4Ww3ROuro.)
>
>

Thanks. Have never seen it before. Do you have some detailed
description about how I/O subsystem works? From what I see, pyparallel
relies on offloading I/O operations to the IOCP. And it seems that I/O
kernel which I'm describing can replace IOCP on unix. And allowing the
C code to do more high-level stuff (like reconnecting and message
delimitation), would be of a benefit for pyparallel too, as it would
allow less heap allocations, smaller number of heap snasphot, etc.

Still PyParallel, is much bigger departure from current Python. As it
doesn't allow arbitrary code to run in threads, so it's much longer
perspective, than I/O kernel which will be just a library on PyPI.

-- 
Paul

From abarnert at yahoo.com  Thu Nov 27 02:12:45 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 26 Nov 2014 17:12:45 -0800
Subject: [Python-ideas] Extending traceback to (optionally) format and
	show locals.
In-Reply-To: <CAJ3HoZ3it8x2KTyuOS5fz7zdDbjfi0hSat=VNc-NvC8BFa=WwA@mail.gmail.com>
References: <CAJ3HoZ3it8x2KTyuOS5fz7zdDbjfi0hSat=VNc-NvC8BFa=WwA@mail.gmail.com>
Message-ID: <A02DE549-9800-41B9-938E-132D570BE226@yahoo.com>

On Nov 26, 2014, at 15:45, Robert Collins <robertc at robertcollins.net> wrote:

> For context please see http://bugs.python.org/issue22937 and
> http://bugs.python.org/issue22936.
> 
> I have two questions I'm hoping to get answered through this thread:
> - does the change in question need a PEP? Antoine seemed to think it
> didn't, as long as it was opt-in (via the unittest CLI etc)
> - Is my implementation approach sound (for traceback, unittest I
> think I have covered :))?
> 
> 
> Implementation wise, I think its useful to work in the current
> traceback module layout - that is to alter extract_stack to
> (optionally) include rendered data about locals and then look for that
> and format it in format_list.
> 
> I'm sure there is code out there that depends on the quadruple nature
> of extract_stack though, so I think we need to preserve that. Three
> strategies occured to me; one is to have parallel functions, one
> quadruple, one quintuple. A second one is to have the return value of
> extract_stack be a quintuple when a new keyword parameter
> include_locals is included. Lastly, and this is my preferred one, if
> we return a tuple subclass with an attribute containing a dict with
> the rendered data on the locals; this can be present but None, or even
> just absent when extract_stack was not asked to include locals.

There are lots of other cases in the stdlib where something is usable as a tuple of n fields or as a structseq/namedtuple of >n fields: stat results, struct_tm, etc. So, why not do the same thing here?

> The last option is my preferred one because the other two both imply
> having a data structure which is likely to break existing code - and
> while you'd have to opt into having them it seems likely to require a
> systematic set of upgrades vs having an optional attribute that can be
> looked up.
> 
> So - thoughts?
> 
> -Rob
> 
> 
> -- 
> Robert Collins <rbtcollins at hp.com>
> Distinguished Technologist
> HP Converged Cloud
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From ncoghlan at gmail.com  Thu Nov 27 03:30:51 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 27 Nov 2014 12:30:51 +1000
Subject: [Python-ideas] Asynchronous IO ideas for Python
In-Reply-To: <CAA0gF6ranMazGaH1XpQU8Hp2S-SsU2n_XT=xZ_j+tFMXczezFg@mail.gmail.com>
References: <CAA0gF6ranMazGaH1XpQU8Hp2S-SsU2n_XT=xZ_j+tFMXczezFg@mail.gmail.com>
Message-ID: <CADiSq7eAcE4N=ZuFB3BG3H3R-qDXE_hLyLJLTWmckv7B3OVB2A@mail.gmail.com>

On 27 November 2014 at 03:35, Paul Colomiets <paul at colomiets.name> wrote:
> Hi,
>
> I've written an article about how I perceive the future of
> asynchronous I/O in Python. It's not something that should directly be
> incorporated into python now, but I believe it's useful for
> python-ideas list.
>
> https://medium.com/@paulcolomiets/the-future-of-asynchronous-io-in-python-ce200536d847

Thanks Paul, that's an interesting write-up.

Another couple of potentially relevant projects in the higher level
"service discovery" space (beyond Zookeeper, which you already
mention) are Fedora's fedmsg (http://www.fedmsg.com/en/latest/ - since
also adopted by Debian I believe) and the Zato ESB project
(https://zato.io/). Those are the kinds of things a service discovery
plugin system would want to be able to handle.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From donald at stufft.io  Thu Nov 27 03:34:04 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 26 Nov 2014 21:34:04 -0500
Subject: [Python-ideas] Asynchronous IO ideas for Python
In-Reply-To: <CADiSq7eAcE4N=ZuFB3BG3H3R-qDXE_hLyLJLTWmckv7B3OVB2A@mail.gmail.com>
References: <CAA0gF6ranMazGaH1XpQU8Hp2S-SsU2n_XT=xZ_j+tFMXczezFg@mail.gmail.com>
 <CADiSq7eAcE4N=ZuFB3BG3H3R-qDXE_hLyLJLTWmckv7B3OVB2A@mail.gmail.com>
Message-ID: <23D1E7AE-8C1D-4EB1-B11E-71A5D85B155B@stufft.io>


> On Nov 26, 2014, at 9:30 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 27 November 2014 at 03:35, Paul Colomiets <paul at colomiets.name> wrote:
>> Hi,
>> 
>> I've written an article about how I perceive the future of
>> asynchronous I/O in Python. It's not something that should directly be
>> incorporated into python now, but I believe it's useful for
>> python-ideas list.
>> 
>> https://medium.com/@paulcolomiets/the-future-of-asynchronous-io-in-python-ce200536d847
> 
> Thanks Paul, that's an interesting write-up.
> 
> Another couple of potentially relevant projects in the higher level
> "service discovery" space (beyond Zookeeper, which you already
> mention) are Fedora's fedmsg (http://www.fedmsg.com/en/latest/ - since
> also adopted by Debian I believe) and the Zato ESB project
> (https://zato.io/). Those are the kinds of things a service discovery
> plugin system would want to be able to handle.
> 

We?re using consul for service discovery in psf-salt FWIW. It?s been pretty
good for us so far. It implements service discovery along with health checks
so that if a service starts failing health checks it gets kicked out of the
service discovery rotation for that service. It also implements a DNS resolver
so you can use DNS to discover instead of it?s API if you want.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From ncoghlan at gmail.com  Thu Nov 27 04:08:42 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 27 Nov 2014 13:08:42 +1000
Subject: [Python-ideas] Extending traceback to (optionally) format and
 show locals.
In-Reply-To: <CAJ3HoZ3it8x2KTyuOS5fz7zdDbjfi0hSat=VNc-NvC8BFa=WwA@mail.gmail.com>
References: <CAJ3HoZ3it8x2KTyuOS5fz7zdDbjfi0hSat=VNc-NvC8BFa=WwA@mail.gmail.com>
Message-ID: <CADiSq7eS2zA1zEvKtspXizcppPYPeJJmvarigrTqJaS+-s5sOQ@mail.gmail.com>

On 27 November 2014 at 09:45, Robert Collins <robertc at robertcollins.net> wrote:
> For context please see http://bugs.python.org/issue22937 and
> http://bugs.python.org/issue22936.

Another useful bit of context is the current RFE to have a way to
extract and save a traceback *summary* that omits the local variable
details: http://bugs.python.org/issue17911

Our inclination to resolve that one was to design a new higher level
traceback manipulation API, which seems relevant to this proposal as
well.

> I have two questions I'm hoping to get answered through this thread:
>  - does the change in question need a PEP? Antoine seemed to think it
> didn't, as long as it was opt-in (via the unittest CLI etc)

I agree this isn't a PEP level change, just a normal RFE. That said,
if the API design details get too confusing, a PEP may still end up
being a useful way of clarifying specific details.

>  - Is my implementation approach sound (for traceback, unittest I
> think I have covered :))?
>
> Implementation wise, I think its useful to work in the current
> traceback module layout - that is to alter extract_stack to
> (optionally) include rendered data about locals and then look for that
> and format it in format_list.

For 17911, we're not so sure about that - there's a strong case to be
made for exposing a new object-oriented API, rather than continuing
with the C-style "records + functions that work on them" model that
the traceback module currently uses.

We made similar additions over the last couple of releases for both
inspect (via inspect.Signature & inspect.Parameter) and dis (via
dis.ByteCode & dis.Instruction).

> I'm sure there is code out there that depends on the quadruple nature
> of extract_stack though, so I think we need to preserve that. Three
> strategies occured to me; one is to have parallel functions, one
> quadruple, one quintuple. A second one is to have the return value of
> extract_stack be a quintuple when a new keyword parameter
> include_locals is included. Lastly, and this is my preferred one, if
> we return a tuple subclass with an attribute containing a dict with
> the rendered data on the locals; this can be present but None, or even
> just absent when extract_stack was not asked to include locals.

Fourth, expand the new ExceptionSummary API proposed in 17911 to also
include the ability to *optionally* preserve the local variable data
from the stack frame, rather than always discarding it.

> The last option is my preferred one because the other two both imply
> having a data structure which is likely to break existing code - and
> while you'd have to opt into having them it seems likely to require a
> systematic set of upgrades vs having an optional attribute that can be
> looked up.
>
> So - thoughts?

I don't see a lot of value in adhering too strictly to the current API
design model - I think it would make more sense to design a new higher
level API, and then look at including some elements to make it easy to
adopt the new tools in existing code without having to rewrite the
world (e.g. the inspect module work in Python 3.4 that switched the
legacy APIs to actually call the new inspect.signature API internally,
greatly expanding the scope of the types they could handle).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From trent at snakebite.org  Thu Nov 27 02:18:05 2014
From: trent at snakebite.org (Trent Nelson)
Date: Wed, 26 Nov 2014 17:18:05 -0800
Subject: [Python-ideas] Asynchronous IO ideas for Python
In-Reply-To: <CAP7+vJK-rSER6-Xyg7dLraXa5BbBJ=hRW_n65TDX5yuRrJWBxw@mail.gmail.com>
References: <CAA0gF6ranMazGaH1XpQU8Hp2S-SsU2n_XT=xZ_j+tFMXczezFg@mail.gmail.com>
 <5E3A89398864414E8EE2CB192E0C965C19A3508EDB@EXMBX10.exchhosting.com>
 <CAP7+vJK-rSER6-Xyg7dLraXa5BbBJ=hRW_n65TDX5yuRrJWBxw@mail.gmail.com>
Message-ID: <5E3A89398864414E8EE2CB192E0C965C19A3508EE1@EXMBX10.exchhosting.com>

Sure can!  I dumped the entire contents of my PyParallel source repository (including the full build .exes/dlls, Visual Studio files, etc) to here about an hour before I gave that presentation:



    https://github.com/pyparallel/release



So to replicate that test you'd clone that, then run one of the helpers, python33-http-server.bat or pyparallel-http-server.bat.



(There's also http://trent.snakebite.net/pyparallel-0.1-3.3.5.zip,<http://trent.snakebite.net/pyparallel-0.1-3.3.5.zip> but it's 117MB, so... I'd recommend github.com.)



The Python 3.3 version is literally `python -m http.server`; the only change I made to the stdlib http/server.py is this:



% diff -u cpython.hg/Lib/http/server.py pyparallel-lib-http-server.py
--- cpython.hg/Lib/http/server.py       2014-08-14 16:00:29.000000000 -0400
+++ pyparallel-lib-http-server.py       2014-11-26 18:12:58.000000000 -0500
@@ -328,7 +328,7 @@
         conntype = self.headers.get('Connection', "")
         if conntype.lower() == 'close':
             self.close_connection = 1
-        elif (conntype.lower() == 'keep-alive' and
+        elif (conntype.lower() == 'keep-alive' or
               self.protocol_version >= "HTTP/1.1"):
             self.close_connection = 0
         # Examine the headers and look for an Expect directive
@@ -440,7 +440,7 @@
         version and the current date.

         """
-        self.log_request(code)
+        #self.log_request(code)
         self.send_response_only(code, message)
         self.send_header('Server', self.version_string())
         self.send_header('Date', self.date_time_string())



The PyParallel version is basically `python_d -m async.http.server`, which is this:



https://github.com/pyparallel/release/blob/master/Lib/async/http/server.py#L251



I started with the stdlib version and mostly refactored it a bit for personal style reasons, then made it work for PyParallel.  The only PyParallel-specific piece is actually this line:



https://github.com/pyparallel/release/blob/master/Lib/async/http/server.py#L437




return response.transport.sendfile(before, path, None)



Everything else is just normal Python, nothing special -- it just conforms to the current constraints of PyParallel.  Basically, the HttpServer.data_received() method will be invoked from parallel threads, not the main interpreter thread.



To give you an idea how the protocol/transport stuff is wired up, the standalone launcher stuff is at the bottom of that file:



ipaddr = socket.gethostbyname(socket.gethostname())

server = async.server(ipaddr, 8080)

async.register(transport=server, protocol=HttpServer)

async.run()





As for why I haven't publicized this stuff until now, to quote myself from that video... "It currently crashes, a lot.  I know why it's crashing, I just haven't had the time to fix it yet.  But hey, it's super fast until it does crash ;-)"



By crash, I mean I'm hitting an assert() in my code -- it happens after the benchmark runs and has to do with the asynchronous socket disconnect logic.  I tried fixing it properly before giving that talk, but ran out of time (https://bitbucket.org/tpn/pyparallel/branch/3.3-px-pygotham-2014-sprint).



I'll fix all of that the next sprint... which will be... heh, hopefully around Christmas?



Oh, actually, the big takeaway from the PyGotham sprint was that I spent an evening re-applying all the wild commit and hackery I'd accumulated to a branch created from the 3.3.5 tag: https://bitbucket.org/tpn/pyparallel/commits/branch/3.3-px.  So diff that against the 3.3.5 tag to get an idea of what interpreter changes I needed to make to get to this point.  (I have no idea why I didn't pick a tag to work off when I first started -- I literally just started hacking on whatever my local tip was on, which was some indeterminate state between... 3.2 and 3.3?)



Side note: I'm really happy with how everything has worked out so far, it is exactly how I envisioned it way back in those python-ideas@ discussions that resulted in tulip/asyncio.  I was seeing ridiculously good scaling on my beefier machine at home (8 core, running native) -- to the point where I was maxing out the client machine at about 50,000 requests/sec (~100MB/s) and the PyParallel box was only at about 40% CPU use.



Oh, and it appears to be much faster than node.js's http-server too (`npm install http-server`, cd into the website directory, `http-server -s .` to get an equivalent HTTP server from node.js), which I thought was cute.  Well, I expected it to be, that's the whole point of being able to exploit all cores and not doing single threaded multiplexing -- so it was good to see that being the case.



Node wasn't actually that much faster than Python's normal http.server if I remember correctly.  It definitely used less CPU overall than the Python one -- basically what I'm seeing is that Python will be maxing out one core, which should only be 25% CPU (4 core VM), but actual CPU use is up around 50%, and it's mostly kernel time making up the other half.  Node will also max out a core, but overall CPU use is ~30%.  I attribute this to Python's http.server using select(), whereas I believe node.js ends up using IOCP in a single-threaded event loop.  So, you could expect Python asyncio to get similar performance to node, but they're both crushed by PyParallel (until it crashes, heh) as soon as you've got more than one core, which was the point I've been vehemently making from day one ;-)



And I just realized I'm writing this e-mail on the same laptop that did that demo, so I can actually back all of this up with a quick run now.



Python 3.3

On Windows:

C:\Users\Trent\src\pyparallel-0.1-3.3.5
? python33-http-server.bat
Serving HTTP on 0.0.0.0 port 8000 ...

On Mac:



(trent at raptor:ttys003) (Wed/19:06) .. (~s/wrk)

% ./wrk -c 8 -t 2 -d 10 --latency http://192.168.46.131:8000/index.html

Running 10s test @ http://192.168.46.131:8000/index.html

2 threads and 8 connections

Thread Stats Avg Stdev Max +/- Stdev

Latency 6.33ms 1.74ms 18.42ms 75.65%

Req/Sec 419.77 119.93 846.00 67.43%

Latency Distribution

50% 6.26ms

75% 7.15ms

90% 8.21ms

99% 12.42ms

8100 requests in 10.00s, 53.48MB read

Requests/sec: 809.92

Transfer/sec: 5.35MB



Node.js

On Windows:



C:\Users\Trent\src\pyparallel-0.1-3.3.5\website
? http-server -s .



On Mac:

(trent at raptor:ttys003) (Wed/19:07) .. (~s/wrk)

% ./wrk -c 8 -t 2 -d 10 --latency http://192.168.46.131:8080/index.html

Running 10s test @ http://192.168.46.131:8080/index.html

2 threads and 8 connections

Thread Stats Avg Stdev Max +/- Stdev

Latency 6.44ms 2.40ms 19.70ms 84.77%

Req/Sec 621.94 124.26 0.94k 68.05%

Latency Distribution

50% 5.93ms

75% 7.00ms

90% 8.97ms

99% 16.17ms

12021 requests in 10.00s, 80.84MB read

Requests/sec: 1201.98

Transfer/sec: 8.08MB





PyParallel

On Windows:



C:\Users\Trent\src\pyparallel-0.1-3.3.5
? pyparallel-http-server.bat
Serving HTTP on 192.168.46.131 port 8080 ...
Traceback (most recent call last):
  File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\runpy.py", line 160, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\runpy.py", line 73, in _run_code
    exec(code, run_globals)
  File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 518, in <module>
    cli = run(*args)
  File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 488, in run
    return CLI(*args, **kwds)
  File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 272, in __init__
    self.run()
  File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 278, in run
    self._process_commandline()
  File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 424, in _process_commandline
    cl.run(args)
  File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 217, in run
    self.command.start()
  File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\command.py", line 455, in start
    self.run()
  File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\px\commands.py", line 90, in run
    async.run()
OSError: [WinError 8] Not enough storage is available to process this command
_PyParallel_Finalize(): px->contexts_active: 462
[92105 refs]
_PyParallel_DeletingThreadState(): px->contexts_active: 462



Oh dear :-)  Hadn't seen that before.  The VM has 4GB allocated to it... I checked taskmgr and it was reporting ~90% physical memory use.  Closed a bunch of things and got it down to 54%, then re-ran, that did the trick.  Including this info in case anyone else runs into this.



Re-run:

C:\Users\Trent\src\pyparallel-0.1-3.3.5
? pyparallel-http-server.bat
Serving HTTP on 192.168.46.131 port 8080 ...



On Mac:

(trent at raptor:ttys003) (Wed/19:16) .. (~s/wrk)

% ./wrk -c 8 -t 2 -d 10 --latency http://192.168.46.131:8080/index.html

Running 10s test @ http://192.168.46.131:8080/index.html

2 threads and 8 connections

Thread Stats Avg Stdev Max +/- Stdev

Latency 4.04ms 1.80ms 23.35ms 91.16%

Req/Sec 1.07k 191.81 1.54k 75.00%

Latency Distribution

50% 3.68ms

75% 4.41ms

90% 5.40ms

99% 13.04ms

20317 requests in 10.00s, 134.22MB read

Requests/sec: 2031.33

Transfer/sec: 13.42MB



And then back on Windows after the benchmark completes:



C:\Users\Trent\src\pyparallel-0.1-3.3.5
? pyparallel-http-server.bat
Serving HTTP on 192.168.46.131 port 8080 ...
Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c, line 6311
Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c, line 6311
Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c, line 6311
Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c, line 6311
Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c, line 6311
Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c, line 6311

Heh.  That's the crashing I was referring to.



So basically, it's better in every category (lowest latency, lowest jitter (stddev), highest throughput) for the duration of the benchmark, then crashes :-)



(Basically, my DisconnectEx assumptions regarding overlapped sockets, socket resource reuse, I/O completion ports, and thread pools were... not correct apparently.)



I remain committed to the assertion that the Windows' kernel approach to asynchronous I/O via I/O completion ports is fundamentally superior to the UNIX/POSIX approach in every aspect if you want to optimally use contemporary multicore hardware (exploit all cores as efficiently as possible).  I talk about this in more detail here: https://speakerdeck.com/trent/parallelism-and-concurrency-with-python?slide=26.



But good grief, it is orders of magnitude more complex at every level.  A less stubborn version of me would have given up waaaay earlier.  Glad I stuck with it though, really happy with results so far.



    Trent.




From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf Of Guido van Rossum
Sent: Wednesday, November 26, 2014 4:49 PM
To: Trent Nelson
Cc: Paul Colomiets; python-ideas
Subject: Re: [Python-ideas] Asynchronous IO ideas for Python

Trent,

Can you post source for the regular and pyparallel HTTP servers you used?

On Wed, Nov 26, 2014 at 12:56 PM, Trent Nelson <trent at snakebite.org<mailto:trent at snakebite.org>> wrote:
Relevant part of the video with the normal Python stats on the left and PyParallel on the right:

https://www.youtube.com/watch?v=4L4Ww3ROuro#t=838

Transcribed stats:

Regular Python HTTP server:

Thread Stats    Avg     Stdev   Max
 Latency        4.93ms  714us   10ms
 Req/Seq        552     154     1.1k
10,480 requests in 10s, 69MB
1048 reqs/sec, 6.9MB/s

PyParallel (4 core Windows VM):

Thread Stats    Avg     Stdev   Max
 Latency        2.41ms  531us   10ms
 Req/Seq        1.74k   183     2.33k
32,831 requests in 10s, 216MB
3263 reqs/sec, 21MB/s

So basically a bit less than linear scaling with more cores, which isn't too bad for a full debug build running on a VM.

-----Original Message-----
From: Trent Nelson
Sent: Wednesday, November 26, 2014 3:36 PM
To: 'Paul Colomiets'; python-ideas
Subject: RE: [Python-ideas] Asynchronous IO ideas for Python

Have you seen this?:

        https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores

I spend the first 80-ish slides on async I/O.

(That was a year ago.  I've done 2-3 sprints on it since then and have gotten it to a point where I can back up the claims with hard numbers on load testing benchmarks, demonstrated in the most recent video: https://www.youtube.com/watch?v=4L4Ww3ROuro.)


        Trent.

-----Original Message-----
From: Python-ideas [mailto:python-ideas-bounces+trent<mailto:python-ideas-bounces%2Btrent>=snakebite.org at python.org<mailto:snakebite.org at python.org>] On Behalf Of Paul Colomiets
Sent: Wednesday, November 26, 2014 12:35 PM
To: python-ideas
Subject: [Python-ideas] Asynchronous IO ideas for Python

Hi,

I've written an article about how I perceive the future of asynchronous I/O in Python. It's not something that should directly be incorporated into python now, but I believe it's useful for python-ideas list.

https://medium.com/@paulcolomiets/the-future-of-asynchronous-io-in-python-ce200536d847

And a place for comments at Hacker News:

https://news.ycombinator.com/item?id=8662782

I hope being helpful with this writeup :)

--
Paul
_______________________________________________
Python-ideas mailing list
Python-ideas at python.org<mailto:Python-ideas at python.org>
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________
Python-ideas mailing list
Python-ideas at python.org<mailto:Python-ideas at python.org>
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/



--
--Guido van Rossum (python.org/~guido<http://python.org/~guido>)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141126/30d97844/attachment-0001.html>

From stefano.borini at ferrara.linux.it  Thu Nov 27 13:09:34 2014
From: stefano.borini at ferrara.linux.it (Stefano Borini)
Date: Thu, 27 Nov 2014 13:09:34 +0100
Subject: [Python-ideas] Fast context creation
Message-ID: <20141127120934.GA2433@ferrara.linux.it>

I have a situation where, no matter how the routine ends, I need to return the focus to a widget. It would be
rather clever to do this with a context, but I would expect something like this to be a possible strategy

    with contextlib.context(enter=None, exit=lambda *args: my_widget.setFocus()):
        do what I need to do

as far as I know, at the moment it's not possible. Am I right? I think it would be an easy and practical 
addition to the contextlib module to quickly register two routines for enter and exit.



From rosuav at gmail.com  Thu Nov 27 13:30:40 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 27 Nov 2014 23:30:40 +1100
Subject: [Python-ideas] Fast context creation
In-Reply-To: <20141127120934.GA2433@ferrara.linux.it>
References: <20141127120934.GA2433@ferrara.linux.it>
Message-ID: <CAPTjJmrtBKrgvGWrg5h=SVAArff+6HMs1AxMfxw1wG9Ms9wgUQ@mail.gmail.com>

On Thu, Nov 27, 2014 at 11:09 PM, Stefano Borini
<stefano.borini at ferrara.linux.it> wrote:
> I have a situation where, no matter how the routine ends, I need to return the focus to a widget. It would be
> rather clever to do this with a context, but I would expect something like this to be a possible strategy
>
>     with contextlib.context(enter=None, exit=lambda *args: my_widget.setFocus()):
>         do what I need to do
>
> as far as I know, at the moment it's not possible. Am I right? I think it would be an easy and practical
> addition to the contextlib module to quickly register two routines for enter and exit.

You could abuse try/finally for this purpose.

try:
    whatever you need to do
finally:
    my_widget.setFocus()

ChrisA

From 4kir4.1i at gmail.com  Thu Nov 27 14:04:21 2014
From: 4kir4.1i at gmail.com (Akira Li)
Date: Thu, 27 Nov 2014 16:04:21 +0300
Subject: [Python-ideas] Fast context creation
References: <20141127120934.GA2433@ferrara.linux.it>
Message-ID: <87zjbcztey.fsf@gmail.com>

Stefano Borini <stefano.borini at ferrara.linux.it>
writes:

> I have a situation where, no matter how the routine ends, I need to return the focus to a widget. It would be
> rather clever to do this with a context, but I would expect something like this to be a possible strategy
>
>     with contextlib.context(enter=None, exit=lambda *args: my_widget.setFocus()):
>         do what I need to do
>
> as far as I know, at the moment it's not possible. Am I right? I think it would be an easy and practical 
> addition to the contextlib module to quickly register two routines for enter and exit.


  with ExitStack() as stack:
      stack.callback(my_widget.setFocus)
      cm = stack.enter_context(run_your_enter_routine())

https://docs.python.org/3/library/contextlib.html#contextlib.ExitStack


--
Akira


From apalala at gmail.com  Thu Nov 27 14:16:02 2014
From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=)
Date: Thu, 27 Nov 2014 08:46:02 -0430
Subject: [Python-ideas] Fast context creation
In-Reply-To: <CAPTjJmrtBKrgvGWrg5h=SVAArff+6HMs1AxMfxw1wG9Ms9wgUQ@mail.gmail.com>
References: <20141127120934.GA2433@ferrara.linux.it>
 <CAPTjJmrtBKrgvGWrg5h=SVAArff+6HMs1AxMfxw1wG9Ms9wgUQ@mail.gmail.com>
Message-ID: <CAN1YFWtf5Aj1Y=tUMymtFyxxGRXg8bV2EOUMEg7g++daT6O0ZQ@mail.gmail.com>

On Thu, Nov 27, 2014 at 8:00 AM, Chris Angelico <rosuav at gmail.com> wrote:

> You could abuse try/finally for this purpose.
>
> try:
>     whatever you need to do
> finally:
>     my_widget.setFocus()
>

There's no abuse there. "finally" is there precisely for that sort of
intention.

I think that what the OP wants is to "announce" the final action at the
-- 
Juancarlo *A?ez*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141127/0fe4aca6/attachment.html>

From solipsis at pitrou.net  Thu Nov 27 14:17:34 2014
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 27 Nov 2014 14:17:34 +0100
Subject: [Python-ideas] Fast context creation
References: <20141127120934.GA2433@ferrara.linux.it>
 <CAPTjJmrtBKrgvGWrg5h=SVAArff+6HMs1AxMfxw1wG9Ms9wgUQ@mail.gmail.com>
Message-ID: <20141127141734.4162ae1f@fsol>

On Thu, 27 Nov 2014 23:30:40 +1100
Chris Angelico <rosuav at gmail.com> wrote:
> On Thu, Nov 27, 2014 at 11:09 PM, Stefano Borini
> <stefano.borini at ferrara.linux.it> wrote:
> > I have a situation where, no matter how the routine ends, I need to return the focus to a widget. It would be
> > rather clever to do this with a context, but I would expect something like this to be a possible strategy
> >
> >     with contextlib.context(enter=None, exit=lambda *args: my_widget.setFocus()):
> >         do what I need to do
> >
> > as far as I know, at the moment it's not possible. Am I right? I think it would be an easy and practical
> > addition to the contextlib module to quickly register two routines for enter and exit.
> 
> You could abuse try/finally for this purpose.

Or just use it, since it's a normal use of try/finally ;-)

Regards

Antoine.


> 
> try:
>     whatever you need to do
> finally:
>     my_widget.setFocus()
> 
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> 




From apalala at gmail.com  Thu Nov 27 14:18:39 2014
From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=)
Date: Thu, 27 Nov 2014 08:48:39 -0430
Subject: [Python-ideas] Fast context creation
In-Reply-To: <20141127120934.GA2433@ferrara.linux.it>
References: <20141127120934.GA2433@ferrara.linux.it>
Message-ID: <CAN1YFWsV7TQCfYgRdPe6yz2QThsZ2RnAoGa0Q+E0oeW=4pLA4Q@mail.gmail.com>

On Thu, Nov 27, 2014 at 7:39 AM, Stefano Borini <
stefano.borini at ferrara.linux.it> wrote:

>
>     with contextlib.context(enter=None, exit=lambda *args:
> my_widget.setFocus()):
>         do what I need to do
>

You can easily define your own context manager to do what you want:

    @contextmanager
    def finally_focus(widget):
        try:
            yield
        finally:
            widget.setFocus()

    with finally_focus(my_widget):
        # do what I need to do


-- 
Juancarlo *A?ez*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141127/1799b9f3/attachment.html>

From rosuav at gmail.com  Thu Nov 27 14:50:19 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 28 Nov 2014 00:50:19 +1100
Subject: [Python-ideas] Fast context creation
In-Reply-To: <20141127141734.4162ae1f@fsol>
References: <20141127120934.GA2433@ferrara.linux.it>
 <CAPTjJmrtBKrgvGWrg5h=SVAArff+6HMs1AxMfxw1wG9Ms9wgUQ@mail.gmail.com>
 <20141127141734.4162ae1f@fsol>
Message-ID: <CAPTjJmoioPAkJEuzQP_zDoqfPh31L5WN_d9ZGCEiYX9Y878pLw@mail.gmail.com>

On Fri, Nov 28, 2014 at 12:17 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> You could abuse try/finally for this purpose.
>
> Or just use it, since it's a normal use of try/finally ;-)
>

I guess so. I normally think of try/finally as an implication that an
exception is expected (rather than a function-level  "atexit" type of
thing), but that's what's happening here. So yes, take off the "ab"
and just use try/finally.

ChrisA

From stefano.borini at ferrara.linux.it  Thu Nov 27 15:24:13 2014
From: stefano.borini at ferrara.linux.it (Stefano Borini)
Date: Thu, 27 Nov 2014 15:24:13 +0100
Subject: [Python-ideas] Fast context creation
In-Reply-To: <CAPTjJmrtBKrgvGWrg5h=SVAArff+6HMs1AxMfxw1wG9Ms9wgUQ@mail.gmail.com>
References: <20141127120934.GA2433@ferrara.linux.it>
 <CAPTjJmrtBKrgvGWrg5h=SVAArff+6HMs1AxMfxw1wG9Ms9wgUQ@mail.gmail.com>
Message-ID: <20141127142413.GA5579@ferrara.linux.it>

On Thu, Nov 27, 2014 at 11:30:40PM +1100, Chris Angelico wrote:
> You could abuse try/finally for this purpose.
> 
> try:
>     whatever you need to do
> finally:
>     my_widget.setFocus()

I thought about this solution, but I am concerned about communication of intent. Using the try 
communicates to a code reader that the code inside is expected to throw, which is not the case.

However, I agree there are good alternative strategies.

From ncoghlan at gmail.com  Thu Nov 27 15:29:54 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 28 Nov 2014 00:29:54 +1000
Subject: [Python-ideas] Fast context creation
In-Reply-To: <87zjbcztey.fsf@gmail.com>
References: <20141127120934.GA2433@ferrara.linux.it> <87zjbcztey.fsf@gmail.com>
Message-ID: <CADiSq7fd3iHj1agZhFhpoVGGjaiFZoFS3kgkePidbf2=iSpwHQ@mail.gmail.com>

On 27 November 2014 at 23:04, Akira Li <4kir4.1i at gmail.com> wrote:
> Stefano Borini <stefano.borini at ferrara.linux.it>
> writes:
>
>> I have a situation where, no matter how the routine ends, I need to return the focus to a widget. It would be
>> rather clever to do this with a context, but I would expect something like this to be a possible strategy
>>
>>     with contextlib.context(enter=None, exit=lambda *args: my_widget.setFocus()):
>>         do what I need to do
>>
>> as far as I know, at the moment it's not possible. Am I right? I think it would be an easy and practical
>> addition to the contextlib module to quickly register two routines for enter and exit.
>
>
>   with ExitStack() as stack:
>       stack.callback(my_widget.setFocus)
>       cm = stack.enter_context(run_your_enter_routine())
>
> https://docs.python.org/3/library/contextlib.html#contextlib.ExitStack

Right, while the underlying try/except/finally construct is generally
a better option for simple cases of inline ad hoc exception handling,
ExitStack is useful when you really do want to reproduce the context
management protocols "pass the exception details to a callback"
behaviour. For example, the feature requested by the OP can be
implemented as:

    @contextmanager
    def context(enter=None, exit=None):
        enter_result = enter() if enter is not None else None
        with ExitStack() as stack:
            if exit is not None:
                stack.push(exit)
            yield enter_result

However, I'm hard pressed to think of a case where using such a
construct would be clearer than writing out a suitable
try/except/finally block. A lot of the value of context managers lies
in our ability to give them *names*, such that it's immediately clear
what they're doing (or which docs to look up to find out more).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From clint.hepner at gmail.com  Thu Nov 27 17:01:09 2014
From: clint.hepner at gmail.com (Clint Hepner)
Date: Thu, 27 Nov 2014 11:01:09 -0500
Subject: [Python-ideas] Fast context creation
In-Reply-To: <20141127142413.GA5579@ferrara.linux.it>
References: <20141127120934.GA2433@ferrara.linux.it>
 <CAPTjJmrtBKrgvGWrg5h=SVAArff+6HMs1AxMfxw1wG9Ms9wgUQ@mail.gmail.com>
 <20141127142413.GA5579@ferrara.linux.it>
Message-ID: <CAKjp4B6UP43NRwTLdkiXva1yQ6m1oZBFBfd=Uct8xN1d_9bM6Q@mail.gmail.com>

On Thu, Nov 27, 2014 at 9:24 AM, Stefano Borini <
stefano.borini at ferrara.linux.it> wrote:

> On Thu, Nov 27, 2014 at 11:30:40PM +1100, Chris Angelico wrote:
> > You could abuse try/finally for this purpose.
> >
> > try:
> >     whatever you need to do
> > finally:
> >     my_widget.setFocus()
>
> I thought about this solution, but I am concerned about communication of
> intent. Using the try
> communicates to a code reader that the code inside is expected to throw,
> which is not the case.
>
> If there is absolutely no possibility of an exception, you don't need any
special construct. Just use

    whatever you need to do
    my_widget.setFocus()

Otherwise, you are tacitly agreeing that an exception *might* occur, so
try/finally is not misleading.


> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141127/a67a064e/attachment.html>

From cf.natali at gmail.com  Thu Nov 27 22:59:45 2014
From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=)
Date: Thu, 27 Nov 2014 21:59:45 +0000
Subject: [Python-ideas] Asynchronous IO ideas for Python
In-Reply-To: <5E3A89398864414E8EE2CB192E0C965C19A3508EE1@EXMBX10.exchhosting.com>
References: <CAA0gF6ranMazGaH1XpQU8Hp2S-SsU2n_XT=xZ_j+tFMXczezFg@mail.gmail.com>
 <5E3A89398864414E8EE2CB192E0C965C19A3508EDB@EXMBX10.exchhosting.com>
 <CAP7+vJK-rSER6-Xyg7dLraXa5BbBJ=hRW_n65TDX5yuRrJWBxw@mail.gmail.com>
 <5E3A89398864414E8EE2CB192E0C965C19A3508EE1@EXMBX10.exchhosting.com>
Message-ID: <CAH_1eM1h=Dy4g=2ygs1PsKqwbU5ckScpKd_0aO=ntucL6_rbXA@mail.gmail.com>

2014-11-27 1:18 GMT+00:00 Trent Nelson <trent at snakebite.org>:
>
> Everything else is just normal Python, nothing special -- it just conforms
> to the current constraints of PyParallel.  Basically, the
> HttpServer.data_received() method will be invoked from parallel threads, not
> the main interpreter thread.

So, still no garbage collection from the threads?


> To give you an idea how the protocol/transport stuff is wired up, the
> standalone launcher stuff is at the bottom of that file:
>
>
>
> ipaddr = socket.gethostbyname(socket.gethostname())
>
> server = async.server(ipaddr, 8080)
>
> async.register(transport=server, protocol=HttpServer)
>
> async.run()
>
>
>
>
>
> As for why I haven't publicized this stuff until now, to quote myself from
> that video... "It currently crashes, a lot.  I know why it's crashing, I
> just haven't had the time to fix it yet.  But hey, it's super fast until it
> does crash ;-)"
>
>
>
> By crash, I mean I'm hitting an assert() in my code -- it happens after the
> benchmark runs and has to do with the asynchronous socket disconnect logic.
> I tried fixing it properly before giving that talk, but ran out of time
> (https://bitbucket.org/tpn/pyparallel/branch/3.3-px-pygotham-2014-sprint).
>
>
>
> I'll fix all of that the next sprint... which will be... heh, hopefully
> around Christmas?
>
>
>
> Oh, actually, the big takeaway from the PyGotham sprint was that I spent an
> evening re-applying all the wild commit and hackery I'd accumulated to a
> branch created from the 3.3.5 tag:
> https://bitbucket.org/tpn/pyparallel/commits/branch/3.3-px.  So diff that
> against the 3.3.5 tag to get an idea of what interpreter changes I needed to
> make to get to this point.  (I have no idea why I didn't pick a tag to work
> off when I first started -- I literally just started hacking on whatever my
> local tip was on, which was some indeterminate state between... 3.2 and
> 3.3?)
>
>
>
> Side note: I'm really happy with how everything has worked out so far, it is
> exactly how I envisioned it way back in those python-ideas@ discussions that
> resulted in tulip/asyncio.  I was seeing ridiculously good scaling on my
> beefier machine at home (8 core, running native) -- to the point where I was
> maxing out the client machine at about 50,000 requests/sec (~100MB/s) and
> the PyParallel box was only at about 40% CPU use.
>
>
>
> Oh, and it appears to be much faster than node.js's http-server too (`npm
> install http-server`, cd into the website directory, `http-server -s .` to
> get an equivalent HTTP server from node.js), which I thought was cute.
> Well, I expected it to be, that's the whole point of being able to exploit
> all cores and not doing single threaded multiplexing -- so it was good to
> see that being the case.
>
>
>
> Node wasn't actually that much faster than Python's normal http.server if I
> remember correctly.  It definitely used less CPU overall than the Python one
> -- basically what I'm seeing is that Python will be maxing out one core,
> which should only be 25% CPU (4 core VM), but actual CPU use is up around
> 50%, and it's mostly kernel time making up the other half.  Node will also
> max out a core, but overall CPU use is ~30%.  I attribute this to Python's
> http.server using select(), whereas I believe node.js ends up using IOCP in
> a single-threaded event loop.  So, you could expect Python asyncio to get
> similar performance to node, but they're both crushed by PyParallel (until
> it crashes, heh) as soon as you've got more than one core, which was the
> point I've been vehemently making from day one ;-)
>
>
>
> And I just realized I'm writing this e-mail on the same laptop that did that
> demo, so I can actually back all of this up with a quick run now.
>
>
>
> Python 3.3
>
> On Windows:
>
> C:\Users\Trent\src\pyparallel-0.1-3.3.5
> ? python33-http-server.bat
> Serving HTTP on 0.0.0.0 port 8000 ...
>
> On Mac:
>
>
>
> (trent at raptor:ttys003) (Wed/19:06) .. (~s/wrk)
>
> % ./wrk -c 8 -t 2 -d 10 --latency http://192.168.46.131:8000/index.html
>
> Running 10s test @ http://192.168.46.131:8000/index.html
>
> 2 threads and 8 connections
>
> Thread Stats Avg Stdev Max +/- Stdev
>
> Latency 6.33ms 1.74ms 18.42ms 75.65%
>
> Req/Sec 419.77 119.93 846.00 67.43%
>
> Latency Distribution
>
> 50% 6.26ms
>
> 75% 7.15ms
>
> 90% 8.21ms
>
> 99% 12.42ms
>
> 8100 requests in 10.00s, 53.48MB read
>
> Requests/sec: 809.92
>
> Transfer/sec: 5.35MB
>
>
>
> Node.js
>
> On Windows:
>
>
>
> C:\Users\Trent\src\pyparallel-0.1-3.3.5\website
> ? http-server -s .
>
>
>
> On Mac:
>
> (trent at raptor:ttys003) (Wed/19:07) .. (~s/wrk)
>
> % ./wrk -c 8 -t 2 -d 10 --latency http://192.168.46.131:8080/index.html
>
> Running 10s test @ http://192.168.46.131:8080/index.html
>
> 2 threads and 8 connections
>
> Thread Stats Avg Stdev Max +/- Stdev
>
> Latency 6.44ms 2.40ms 19.70ms 84.77%
>
> Req/Sec 621.94 124.26 0.94k 68.05%
>
> Latency Distribution
>
> 50% 5.93ms
>
> 75% 7.00ms
>
> 90% 8.97ms
>
> 99% 16.17ms
>
> 12021 requests in 10.00s, 80.84MB read
>
> Requests/sec: 1201.98
>
> Transfer/sec: 8.08MB
>
>
>
>
>
> PyParallel
>
> On Windows:
>
>
>
> C:\Users\Trent\src\pyparallel-0.1-3.3.5
> ? pyparallel-http-server.bat
> Serving HTTP on 192.168.46.131 port 8080 ...
> Traceback (most recent call last):
>   File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\runpy.py", line 160, in
> _run_module_as_main
>     "__main__", fname, loader, pkg_name)
>   File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\runpy.py", line 73, in
> _run_code
>     exec(code, run_globals)
>   File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 518,
> in <module>
>     cli = run(*args)
>   File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 488,
> in run
>     return CLI(*args, **kwds)
>   File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 272,
> in __init__
>     self.run()
>   File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 278,
> in run
>     self._process_commandline()
>   File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 424,
> in _process_commandline
>     cl.run(args)
>   File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 217,
> in run
>     self.command.start()
>   File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\command.py", line
> 455, in start
>     self.run()
>   File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\px\commands.py", line
> 90, in run
>     async.run()
> OSError: [WinError 8] Not enough storage is available to process this
> command
> _PyParallel_Finalize(): px->contexts_active: 462
> [92105 refs]
> _PyParallel_DeletingThreadState(): px->contexts_active: 462
>
>
>
> Oh dear :-)  Hadn't seen that before.  The VM has 4GB allocated to it... I
> checked taskmgr and it was reporting ~90% physical memory use.  Closed a
> bunch of things and got it down to 54%, then re-ran, that did the trick.
> Including this info in case anyone else runs into this.
>
>
>
> Re-run:
>
> C:\Users\Trent\src\pyparallel-0.1-3.3.5
> ? pyparallel-http-server.bat
> Serving HTTP on 192.168.46.131 port 8080 ...
>
>
>
> On Mac:
>
> (trent at raptor:ttys003) (Wed/19:16) .. (~s/wrk)
>
> % ./wrk -c 8 -t 2 -d 10 --latency http://192.168.46.131:8080/index.html
>
> Running 10s test @ http://192.168.46.131:8080/index.html
>
> 2 threads and 8 connections
>
> Thread Stats Avg Stdev Max +/- Stdev
>
> Latency 4.04ms 1.80ms 23.35ms 91.16%
>
> Req/Sec 1.07k 191.81 1.54k 75.00%
>
> Latency Distribution
>
> 50% 3.68ms
>
> 75% 4.41ms
>
> 90% 5.40ms
>
> 99% 13.04ms
>
> 20317 requests in 10.00s, 134.22MB read
>
> Requests/sec: 2031.33
>
> Transfer/sec: 13.42MB
>
>
>
> And then back on Windows after the benchmark completes:
>
>
>
> C:\Users\Trent\src\pyparallel-0.1-3.3.5
> ? pyparallel-http-server.bat
> Serving HTTP on 192.168.46.131 port 8080 ...
> Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c,
> line 6311
> Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c,
> line 6311
> Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c,
> line 6311
> Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c,
> line 6311
> Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c,
> line 6311
> Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c,
> line 6311
>
> Heh.  That's the crashing I was referring to.
>
>
>
> So basically, it's better in every category (lowest latency, lowest jitter
> (stddev), highest throughput) for the duration of the benchmark, then
> crashes :-)
>
>
>
> (Basically, my DisconnectEx assumptions regarding overlapped sockets, socket
> resource reuse, I/O completion ports, and thread pools were... not correct
> apparently.)
>
>
>
> I remain committed to the assertion that the Windows' kernel approach to
> asynchronous I/O via I/O completion ports is fundamentally superior to the
> UNIX/POSIX approach in every aspect if you want to optimally use
> contemporary multicore hardware (exploit all cores as efficiently as
> possible).  I talk about this in more detail here:
> https://speakerdeck.com/trent/parallelism-and-concurrency-with-python?slide=26.
>
>
>
> But good grief, it is orders of magnitude more complex at every level.  A
> less stubborn version of me would have given up waaaay earlier.  Glad I
> stuck with it though, really happy with results so far.
>
>
>
>     Trent.
>
>
>
>
>
>
>
> From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf Of Guido
> van Rossum
> Sent: Wednesday, November 26, 2014 4:49 PM
> To: Trent Nelson
> Cc: Paul Colomiets; python-ideas
> Subject: Re: [Python-ideas] Asynchronous IO ideas for Python
>
>
>
> Trent,
>
> Can you post source for the regular and pyparallel HTTP servers you used?
>
>
>
> On Wed, Nov 26, 2014 at 12:56 PM, Trent Nelson <trent at snakebite.org> wrote:
>
> Relevant part of the video with the normal Python stats on the left and
> PyParallel on the right:
>
> https://www.youtube.com/watch?v=4L4Ww3ROuro#t=838
>
> Transcribed stats:
>
> Regular Python HTTP server:
>
> Thread Stats    Avg     Stdev   Max
>  Latency        4.93ms  714us   10ms
>  Req/Seq        552     154     1.1k
> 10,480 requests in 10s, 69MB
> 1048 reqs/sec, 6.9MB/s
>
> PyParallel (4 core Windows VM):
>
> Thread Stats    Avg     Stdev   Max
>  Latency        2.41ms  531us   10ms
>  Req/Seq        1.74k   183     2.33k
> 32,831 requests in 10s, 216MB
> 3263 reqs/sec, 21MB/s
>
> So basically a bit less than linear scaling with more cores, which isn't too
> bad for a full debug build running on a VM.
>
> -----Original Message-----
> From: Trent Nelson
> Sent: Wednesday, November 26, 2014 3:36 PM
> To: 'Paul Colomiets'; python-ideas
> Subject: RE: [Python-ideas] Asynchronous IO ideas for Python
>
> Have you seen this?:
>
>
> https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores
>
> I spend the first 80-ish slides on async I/O.
>
> (That was a year ago.  I've done 2-3 sprints on it since then and have
> gotten it to a point where I can back up the claims with hard numbers on
> load testing benchmarks, demonstrated in the most recent video:
> https://www.youtube.com/watch?v=4L4Ww3ROuro.)
>
>
>         Trent.
>
>
> -----Original Message-----
> From: Python-ideas
> [mailto:python-ideas-bounces+trent=snakebite.org at python.org] On Behalf Of
> Paul Colomiets
> Sent: Wednesday, November 26, 2014 12:35 PM
> To: python-ideas
> Subject: [Python-ideas] Asynchronous IO ideas for Python
>
> Hi,
>
> I've written an article about how I perceive the future of asynchronous I/O
> in Python. It's not something that should directly be incorporated into
> python now, but I believe it's useful for python-ideas list.
>
> https://medium.com/@paulcolomiets/the-future-of-asynchronous-io-in-python-ce200536d847
>
> And a place for comments at Hacker News:
>
> https://news.ycombinator.com/item?id=8662782
>
> I hope being helpful with this writeup :)
>
> --
> Paul
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
>
> --
>
> --Guido van Rossum (python.org/~guido)
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From trent at snakebite.org  Fri Nov 28 00:11:20 2014
From: trent at snakebite.org (Trent Nelson)
Date: Thu, 27 Nov 2014 18:11:20 -0500
Subject: [Python-ideas] Asynchronous IO ideas for Python]
Message-ID: <20141127231119.GA7729@snakebite.org>

On Thu, Nov 27, 2014 at 09:59:45PM +0000, Charles-Fran?ois Natali wrote:
> 2014-11-27 1:18 GMT+00:00 Trent Nelson <trent at snakebite.org>:
> >
> > Everything else is just normal Python, nothing special -- it just conforms
> > to the current constraints of PyParallel.  Basically, the
> > HttpServer.data_received() method will be invoked from parallel threads, not
> > the main interpreter thread.
>
> So, still no garbage collection from the threads?

    Correct.

    Not having garbage collection has surprisingly not gotten in the way
    so far, so it's not even on the radar anymore.  There are other means
    available for persisting objects past the lifetime of the parallel
    context, and you could always do an @async.call_from_main_thread if
    you want to have the main thread's memory allocator (and thus, GC)
    kick in.

    At one point, all these tests passed, just to give you an idea of
    some of the facilities that are available:

        https://bitbucket.org/tpn/pyparallel/src/89576c868a3f41747d138a473b090e0f2c6fef61/Lib/async/test/test_primitives.py?at=3.3-px

    (I haven't removed any of those facilities, I just haven't spent any
     time on them since switching over to the async socket stuff, so I
     can't comment on their current state.)


        Trent.

----- End forwarded message -----

From robertc at robertcollins.net  Fri Nov 28 01:48:15 2014
From: robertc at robertcollins.net (Robert Collins)
Date: Fri, 28 Nov 2014 13:48:15 +1300
Subject: [Python-ideas] Extending traceback to (optionally) format and
 show locals.
In-Reply-To: <A02DE549-9800-41B9-938E-132D570BE226@yahoo.com>
References: <CAJ3HoZ3it8x2KTyuOS5fz7zdDbjfi0hSat=VNc-NvC8BFa=WwA@mail.gmail.com>
 <A02DE549-9800-41B9-938E-132D570BE226@yahoo.com>
Message-ID: <CAJ3HoZ1A2SDN5NMKRrCjOV35eNFZmWs_d8SzqTt8A_XNfuP2dw@mail.gmail.com>

On 27 November 2014 at 14:12, Andrew Barnert <abarnert at yahoo.com> wrote:
> On Nov 26, 2014, at 15:45, Robert Collins <robertc at robertcollins.net> wrote:

>> I'm sure there is code out there that depends on the quadruple nature
>> of extract_stack though, so I think we need to preserve that. Three
>> strategies occured to me; one is to have parallel functions, one
>> quadruple, one quintuple. A second one is to have the return value of
>> extract_stack be a quintuple when a new keyword parameter
>> include_locals is included. Lastly, and this is my preferred one, if
>> we return a tuple subclass with an attribute containing a dict with
>> the rendered data on the locals; this can be present but None, or even
>> just absent when extract_stack was not asked to include locals.
>
> There are lots of other cases in the stdlib where something is usable as a tuple of n fields or as a structseq/namedtuple of >n fields: stat results, struct_tm, etc. So, why not do the same thing here?

Because backwards compatibility. Moving to a namedtuple is fine -
changing the length of the tuple is a problem.

-Rob


-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud

From robertc at robertcollins.net  Fri Nov 28 01:54:35 2014
From: robertc at robertcollins.net (Robert Collins)
Date: Fri, 28 Nov 2014 13:54:35 +1300
Subject: [Python-ideas] Extending traceback to (optionally) format and
 show locals.
In-Reply-To: <CADiSq7eS2zA1zEvKtspXizcppPYPeJJmvarigrTqJaS+-s5sOQ@mail.gmail.com>
References: <CAJ3HoZ3it8x2KTyuOS5fz7zdDbjfi0hSat=VNc-NvC8BFa=WwA@mail.gmail.com>
 <CADiSq7eS2zA1zEvKtspXizcppPYPeJJmvarigrTqJaS+-s5sOQ@mail.gmail.com>
Message-ID: <CAJ3HoZ2TW2PK6sqWSiarkpLu8pGMjsRwqE7fhUaZJA7CHRbmfA@mail.gmail.com>

On 27 November 2014 at 16:08, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 27 November 2014 at 09:45, Robert Collins <robertc at robertcollins.net> wrote:
>> For context please see http://bugs.python.org/issue22937 and
>> http://bugs.python.org/issue22936.
>
> Another useful bit of context is the current RFE to have a way to
> extract and save a traceback *summary* that omits the local variable
> details: http://bugs.python.org/issue17911

AIUI the specific desire is to allow a minimal cost capture of
tracebacks without frames (so as to allow gc), with enough data to
render  a traceback later should the thing turn out to escape the
system. So largely skipping linecache lookup etc.

I think thats a good thing to do :).

> Our inclination to resolve that one was to design a new higher level
> traceback manipulation API, which seems relevant to this proposal as
> well.
....
>>  - Is my implementation approach sound (for traceback, unittest I
>> think I have covered :))?
>>
>> Implementation wise, I think its useful to work in the current
>> traceback module layout - that is to alter extract_stack to
>> (optionally) include rendered data about locals and then look for that
>> and format it in format_list.
>
> For 17911, we're not so sure about that - there's a strong case to be
> made for exposing a new object-oriented API, rather than continuing
> with the C-style "records + functions that work on them" model that
> the traceback module currently uses.

There's a nice functional feel there more than a C thing, IMO - in
that there is no hidden state, and everything is laid out for direct
view.

OTOH I've no objection to a more objects-and-methods feel, though
we'll want a thunk through to the new code which will essentially just
be constructing objects just-in-time. Seems straight forward enough to
write though.

> Fourth, expand the new ExceptionSummary API proposed in 17911 to also
> include the ability to *optionally* preserve the local variable data
> from the stack frame, rather than always discarding it.

A couple of other related things I should be clear about: I want
something I can backport successfully via traceback2, for use in
unittest2. I don't see any issue with the proposal so far, other than
the linecache API change needed to support __loader__, which implies a
linecache2 backport as well.

>> The last option is my preferred one because the other two both imply
>> having a data structure which is likely to break existing code - and
>> while you'd have to opt into having them it seems likely to require a
>> systematic set of upgrades vs having an optional attribute that can be
>> looked up.
>>
>> So - thoughts?
>
> I don't see a lot of value in adhering too strictly to the current API
> design model - I think it would make more sense to design a new higher
> level API, and then look at including some elements to make it easy to
> adopt the new tools in existing code without having to rewrite the
> world (e.g. the inspect module work in Python 3.4 that switched the
> legacy APIs to actually call the new inspect.signature API internally,
> greatly expanding the scope of the types they could handle).

Sure. AIUI noone is actively pushing on the new thing, so I'll put my
hand up for it now and we'll see where I get to in my available
cycles.

-Rob

-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud

From abarnert at yahoo.com  Fri Nov 28 03:43:37 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 27 Nov 2014 18:43:37 -0800
Subject: [Python-ideas] Extending traceback to (optionally) format and
	show locals.
In-Reply-To: <CAJ3HoZ1A2SDN5NMKRrCjOV35eNFZmWs_d8SzqTt8A_XNfuP2dw@mail.gmail.com>
References: <CAJ3HoZ3it8x2KTyuOS5fz7zdDbjfi0hSat=VNc-NvC8BFa=WwA@mail.gmail.com>
 <A02DE549-9800-41B9-938E-132D570BE226@yahoo.com>
 <CAJ3HoZ1A2SDN5NMKRrCjOV35eNFZmWs_d8SzqTt8A_XNfuP2dw@mail.gmail.com>
Message-ID: <7F8C6BEE-BB37-4C09-8A81-5053FC747B39@yahoo.com>

On Nov 27, 2014, at 16:48, Robert Collins <robertc at robertcollins.net> wrote:

> On 27 November 2014 at 14:12, Andrew Barnert <abarnert at yahoo.com> wrote:
>> On Nov 26, 2014, at 15:45, Robert Collins <robertc at robertcollins.net> wrote:
> 
>>> I'm sure there is code out there that depends on the quadruple nature
>>> of extract_stack though, so I think we need to preserve that. Three
>>> strategies occured to me; one is to have parallel functions, one
>>> quadruple, one quintuple. A second one is to have the return value of
>>> extract_stack be a quintuple when a new keyword parameter
>>> include_locals is included. Lastly, and this is my preferred one, if
>>> we return a tuple subclass with an attribute containing a dict with
>>> the rendered data on the locals; this can be present but None, or even
>>> just absent when extract_stack was not asked to include locals.
>> 
>> There are lots of other cases in the stdlib where something is usable as a tuple of n fields or as a structseq/namedtuple of >n fields: stat results, struct_tm, etc. So, why not do the same thing here?
> 
> Because backwards compatibility. Moving to a namedtuple is fine -
> changing the length of the tuple is a problem.

That's the whole point: you're _not_ changing the length. Again, look at the examples that are already in the stdlib: they have a fixed length as a tuple, with extra fields accessible by name only. And it's dead easy to do with structseq.

From liam.marsh.home at gmail.com  Fri Nov 28 17:29:32 2014
From: liam.marsh.home at gmail.com (Liam Marsh)
Date: Fri, 28 Nov 2014 17:29:32 +0100
Subject: [Python-ideas] improve compatibility
Message-ID: <CACPPHzu91DjMwRSCv9TnW7LnUkkn2zj7XsXPmN2022e-Ln_29Q@mail.gmail.com>

hello,
the problem is that even with extreme precaution, it is impossible to keep
ALL modules compatible from a version to another.
what I want to ask is this:
-some "packs" which can, like py-compile, generate .pyc files, but using
"old" versions of the default library, and of __builtins__.
-One will go with every minor version and will be optionnal in the
installation
-any imported py file will be able to choose which version it wants with
the
   "#! py recommended version X.X" or
   "#! py mandatory version X.X" commentaries at the begining of the file.

thank you and have a nice day/evening/night.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141128/cab1e664/attachment.html>

From abarnert at yahoo.com  Sat Nov 29 06:07:24 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 28 Nov 2014 21:07:24 -0800
Subject: [Python-ideas] improve compatibility
In-Reply-To: <CACPPHzu91DjMwRSCv9TnW7LnUkkn2zj7XsXPmN2022e-Ln_29Q@mail.gmail.com>
References: <CACPPHzu91DjMwRSCv9TnW7LnUkkn2zj7XsXPmN2022e-Ln_29Q@mail.gmail.com>
Message-ID: <00DC45BE-A51D-4EB7-BF90-D4BA2F4B3189@yahoo.com>

On Nov 28, 2014, at 8:29, Liam Marsh <liam.marsh.home at gmail.com> wrote:

> hello,
> the problem is that even with extreme precaution, it is impossible to keep ALL modules compatible from a version to another.

Do you have a specific library or app that you've had a problem with? There were a handful of modules that had a problem with the 3.2 to 3.3 conversion, but every one I saw was caused by language and implementation changes, not stdlib changes. I don't think I've seen anything that works with 3.3 but not 3.4. I'm sure it's not impossible for such a thing to happen, but it would be helpful to have at least one real-life example.

> what I want to ask is this:
> -some "packs" which can, like py-compile, generate .pyc files, but using "old" versions of the default library, and of __builtins__.

But how would this work? The same changes that broke a handful of third-party modules between 3.2 and 3.3 probably also mean that the 3.2 stdlib wouldn't work in 3.3 without minor changes. And as for builtins, most of those are exposing internals of the implementation, so trying to make the 3.2 builtins work with 3.3 would take a lot more work than just building the 3.2 code against 3.3.

> -One will go with every minor version and will be optionnal in the installation
> -any imported py file will be able to choose which version it wants with the 
>    "#! py recommended version X.X" or
>    "#! py mandatory version X.X" commentaries at the begining of the file.
> 
> thank you and have a nice day/evening/night.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From liam.marsh.home at gmail.com  Sat Nov 29 16:26:30 2014
From: liam.marsh.home at gmail.com (Liam Marsh)
Date: Sat, 29 Nov 2014 16:26:30 +0100
Subject: [Python-ideas] improve compatibility
In-Reply-To: <D3E4E2C0-159D-4BC4-82D4-E3E90F39E64C@yahoo.com>
References: <CACPPHzu91DjMwRSCv9TnW7LnUkkn2zj7XsXPmN2022e-Ln_29Q@mail.gmail.com>
 <00DC45BE-A51D-4EB7-BF90-D4BA2F4B3189@yahoo.com>
 <CACPPHzt4h9ZGHBGkucq49yifhWAiSCUGzgFwVF_Pes-1w39OPA@mail.gmail.com>
 <D3E4E2C0-159D-4BC4-82D4-E3E90F39E64C@yahoo.com>
Message-ID: <CACPPHzsafOYKogQTHLHfDJj_7=62EHGzsBMtbwTD0HtYbwjwWw@mail.gmail.com>

oops..
I sent an answer to  andrew barnert, who have sent the first answer to my
question, and not to the list...
however , I do have an answer to my question.
thank you!

2014-11-29 11:51 GMT+01:00 Andrew Barnert <abarnert at yahoo.com>:

> On Nov 29, 2014, at 0:46, Liam Marsh <liam.marsh.home at gmail.com> wrote:
>
> hello,
>
> yes, the bug which gave me this idea is a py3.2 to py3.3 bug with Vpython,
> a 3D graphs library.
>
>
> And is the bug a stdlib bug, or a language bug?
>
> I also thought this will be the reconciliation between py2 and py3. (this
> is why I thought the packs will include a version of the stdlibs)
>
>
> The language differences between 2.x and 3.x are huge, and most of the
> stdlib differences between 2.6 and 3.1 or 2.7 and 3.2 are related to those
> language differences. Porting code to 3.x is primarily about fixing Unicode
> stuff, or code that was already deprecated in 2.5 or 2.6 but wasn't broken
> until 3.x. Having the 2.7 stdlib in 3.x would be a huge amount of work for
> almost no benefit.
>
> in fact, how do the .pyc work? were them modified by the "language and
> implementation changes"? how do them import other modules?
>
>
> .pyc files are just compiled bytecode, with a version-specific header.
> If you fixed the 2.7 stdlib to work in 3.x (which, again, would be a huge
> amount of work), you could compile it with 3.4.
>
> But you're missing the fact that large chunks of the stdlib are written in
> C, and compiler against the Python C API. And parts of the stdlib
> (especially builtins and the sys module) are exposing types and functions
> written in C that are part of the core implementation, so the 2.x version
> of the sys module wouldn't be compatible with 2.x code anyway.
>
> Also, did you mean to write just to me instead of to the list?
>
>
> thank you!
>
>
>
> 2014-11-29 6:07 GMT+01:00 Andrew Barnert <abarnert at yahoo.com>:
>
>> On Nov 28, 2014, at 8:29, Liam Marsh <liam.marsh.home at gmail.com> wrote:
>>
>> > hello,
>> > the problem is that even with extreme precaution, it is impossible to
>> keep ALL modules compatible from a version to another.
>>
>> Do you have a specific library or app that you've had a problem with?
>> There were a handful of modules that had a problem with the 3.2 to 3.3
>> conversion, but every one I saw was caused by language and implementation
>> changes, not stdlib changes. I don't think I've seen anything that works
>> with 3.3 but not 3.4. I'm sure it's not impossible for such a thing to
>> happen, but it would be helpful to have at least one real-life example.
>>
>> > what I want to ask is this:
>> > -some "packs" which can, like py-compile, generate .pyc files, but
>> using "old" versions of the default library, and of __builtins__.
>>
>> But how would this work? The same changes that broke a handful of
>> third-party modules between 3.2 and 3.3 probably also mean that the 3.2
>> stdlib wouldn't work in 3.3 without minor changes. And as for builtins,
>> most of those are exposing internals of the implementation, so trying to
>> make the 3.2 builtins work with 3.3 would take a lot more work than just
>> building the 3.2 code against 3.3.
>>
>> > -One will go with every minor version and will be optionnal in the
>> installation
>> > -any imported py file will be able to choose which version it wants
>> with the
>> >    "#! py recommended version X.X" or
>> >    "#! py mandatory version X.X" commentaries at the begining of the
>> file.
>> >
>> > thank you and have a nice day/evening/night.
>> > _______________________________________________
>> > Python-ideas mailing list
>> > Python-ideas at python.org
>> > https://mail.python.org/mailman/listinfo/python-ideas
>> > Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141129/8b82d67d/attachment.html>

From ncoghlan at gmail.com  Sat Nov 29 17:05:11 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 30 Nov 2014 02:05:11 +1000
Subject: [Python-ideas] improve compatibility
In-Reply-To: <00DC45BE-A51D-4EB7-BF90-D4BA2F4B3189@yahoo.com>
References: <CACPPHzu91DjMwRSCv9TnW7LnUkkn2zj7XsXPmN2022e-Ln_29Q@mail.gmail.com>
 <00DC45BE-A51D-4EB7-BF90-D4BA2F4B3189@yahoo.com>
Message-ID: <CADiSq7dX0mGhrir2VBN8b+Duv1cf-znmfcqj6PvSfDgqTSg57g@mail.gmail.com>

On 29 November 2014 at 15:07, Andrew Barnert
<abarnert at yahoo.com.dmarc.invalid> wrote:
> On Nov 28, 2014, at 8:29, Liam Marsh <liam.marsh.home at gmail.com> wrote:
>
>> hello,
>> the problem is that even with extreme precaution, it is impossible to keep ALL modules compatible from a version to another.
>
> Do you have a specific library or app that you've had a problem with? There were a handful of modules that had a problem with the 3.2 to 3.3 conversion, but every one I saw was caused by language and implementation changes, not stdlib changes. I don't think I've seen anything that works with 3.3 but not 3.4. I'm sure it's not impossible for such a thing to happen, but it would be helpful to have at least one real-life example.

The desire for doing selective upgrades on a stable base is actually a
pretty common one (e.g. using modern unittest features on Python 2.6
or 2.7). The hard part is figuring out a way to provide those
selective upgrades cost effectively. (In the context of unpaid
development, the costs are measured in contributor time)

>> what I want to ask is this:
>> -some "packs" which can, like py-compile, generate .pyc files, but using "old" versions of the default library, and of __builtins__.
>
> But how would this work? The same changes that broke a handful of third-party modules between 3.2 and 3.3 probably also mean that the 3.2 stdlib wouldn't work in 3.3 without minor changes. And as for builtins, most of those are exposing internals of the implementation, so trying to make the 3.2 builtins work with 3.3 would take a lot more work than just building the 3.2 code against 3.3.

The feature Liam is looking for effectively already exists, in the
form of updated standard library modules that have been backported to
earlier versions via PyPI. There's a (likely incomplete) list at
https://wiki.python.org/moin/Python2orPython3#Supporting_Python_2_and_Python_3_in_a_common_code_base.

This tackles the problem in the opposite direction, by ensuring
particular modules remain compatible with older versions, rather than
only running on the latest release. One of our reasons for making pip
more readily available to Python 2 users in Python 2.7.9 is to make
those modules easier to access (together with Python 3 migration tools
like six, modernize, future, and caniusepython3).

Cheers,
Nick.

P.S. While some of those backports are maintained directly by core
developers, the PSF license actually allows anyone that wants to (and
has the necessary time available) to backport modules. A number of the
backports originated in users wanting particular features on earlier
versions for their own use, and deciding to make their backport
generally available, rather than keeping it to themselves.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From random832 at fastmail.us  Sun Nov 30 22:30:24 2014
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Sun, 30 Nov 2014 16:30:24 -0500
Subject: [Python-ideas] Extending traceback to (optionally) format and
 show locals.
In-Reply-To: <CAJ3HoZ1A2SDN5NMKRrCjOV35eNFZmWs_d8SzqTt8A_XNfuP2dw@mail.gmail.com>
References: <CAJ3HoZ3it8x2KTyuOS5fz7zdDbjfi0hSat=VNc-NvC8BFa=WwA@mail.gmail.com>
 <A02DE549-9800-41B9-938E-132D570BE226@yahoo.com>
 <CAJ3HoZ1A2SDN5NMKRrCjOV35eNFZmWs_d8SzqTt8A_XNfuP2dw@mail.gmail.com>
Message-ID: <1417383024.1352065.197040833.4FFABE62@webmail.messagingengine.com>



On Thu, Nov 27, 2014, at 19:48, Robert Collins wrote:
> On 27 November 2014 at 14:12, Andrew Barnert <abarnert at yahoo.com> wrote:
> > On Nov 26, 2014, at 15:45, Robert Collins <robertc at robertcollins.net> wrote:
> 
> >> I'm sure there is code out there that depends on the quadruple nature
> >> of extract_stack though, so I think we need to preserve that. Three
> >> strategies occured to me; one is to have parallel functions, one
> >> quadruple, one quintuple. A second one is to have the return value of
> >> extract_stack be a quintuple when a new keyword parameter
> >> include_locals is included. Lastly, and this is my preferred one, if
> >> we return a tuple subclass with an attribute containing a dict with
> >> the rendered data on the locals; this can be present but None, or even
> >> just absent when extract_stack was not asked to include locals.
> >
> > There are lots of other cases in the stdlib where something is usable as a tuple of n fields or as a structseq/namedtuple of >n fields: stat results, struct_tm, etc. So, why not do the same thing here?
> 
> Because backwards compatibility. Moving to a namedtuple is fine -
> changing the length of the tuple is a problem.

Er, but what is being suggested is to do the same backwards-compatible
thing: move to a namedtuple-like object with extra non-tuple fields,
just like those others. I'm confused as to what is the conflict here.