On 2/6/2014, 7:10 PM, Ram Rachum wrote:

Hi,

This time I'm posting on the right list :) Sorry for the mistake in the last thread.

`iter` has a very cool `sentinel` argument. I suggest an additional argument `sentinel_exception`; when it's supplied, instead of waiting for a sentinel value, we wait for a sentinel exception to be raised, and then the iteration is finished.

This'll be useful to construct things like this:

my_iterator = iter(my_deque.popleft, IndexError)

Before starting discussion about the new parameter: how passing 'my_deque.popleft' is supposed to work? Yury

My bad, never used it that way. Since your propose a new parameter, it needs to be a keyword-only one, like this: iter(callback, sentinel_exception=IndexError) Yury On 2/6/2014, 7:17 PM, Ram Rachum wrote:

Looks like someone needs to read the `iter` docs :)

On Fri, Feb 7, 2014 at 2:14 AM, Yury Selivanov wrote:

...

On 2/6/2014, 7:10 PM, Ram Rachum wrote:

...

Hi,

This time I'm posting on the right list :) Sorry for the mistake in the last thread.

`iter` has a very cool `sentinel` argument. I suggest an additional argument `sentinel_exception`; when it's supplied, instead of waiting for a sentinel value, we wait for a sentinel exception to be raised, and then the iteration is finished.

This'll be useful to construct things like this:

my_iterator = iter(my_deque.popleft, IndexError)

Before starting discussion about the new parameter: how passing 'my_deque.popleft' is supposed to work?

Yury _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

--

--- You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group. To unsubscribe from this topic, visit https://groups.google.com/d/ topic/python-ideas/UCaNfAHkBlQ/unsubscribe. To unsubscribe from this group and all its topics, send an email to python-ideas+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On 2/6/2014 7:10 PM, Ram Rachum wrote:

`iter` has a very cool `sentinel` argument. I suggest an additional argument `sentinel_exception`; when it's supplied, instead of waiting for a sentinel value, we wait for a sentinel exception to be raised, and then the iteration is finished.

This'll be useful to construct things like this:

my_iterator = iter(my_deque.popleft, IndexError)

What do you think?

I think this would be a great idea if simplified to reuse the current parameter. It can work in Python because exceptions are objects like anything else and can be passed as arguments. No new parameter is needed. We only need to add 'or raises' to the two-parameter iter definition "In the second form, the callable is called until it returns the sentinel." to get "In the second form, the callable is called until it returns or raises the sentinel." With this version of the proposal, your pop example would work as written. The two-parameter form return a 'callable_iterator' object. Its __next__ method in Python might look like (untested) def __next__(self): x = self.func() if x == self.sentinel: raise StopIteration else: return x The only change needed for the added behavior is to wrap the function call and change an exception matching the sentinel to StopIteration. def __next__(self): try: x = self.func() except Exception as exc: if isinstance(exc, self.sentinel): raise StopIteration from None if x == self.sentinel: raise StopIteration else: return x I do something similar to this in my custom function tester and it is really handy to be able to pass in an expected exception as the expected 'output'. For example, my 'test table' for an int divide function might contain these input and expected output pairs: (2,0), ZeroDevisionError (2,1), 2 (2,2), 1. I consider the threat to backward compatibility, because of the added test for exceptions, theoretical rather than actual. It is very rare to write a function that returns an exception, rarer still to write one that can also raise an instance of the same exception class. Also, using an exception-returning function in two-parameter iter is tricky because exceptions inherit the default equality comparison of only equal by identity.

...

...

ValueError('high') == ValueError('high') False e=ValueError('high') e == e True

The following tested example of code that would be broken by the proposal could well be the first ever written. Warning: it is so twisted and awful that it might to read it. --- from random import random e=ValueError('high value') def crazy(): x = random() if x > .999: return e elif x < .001: raise ValueError('low value') else: return x try: for x in iter(crazy, e): pass else: print(e.args[0]) except ValueError as exc: print(exc.args[0]) # prints 'high value' or 'low value' in less than a second -- Terry Jan Reedy

On 2/6/2014, 9:36 PM, Terry Reedy wrote:

I think this would be a great idea if simplified to reuse the current parameter. It can work in Python because exceptions are objects like anything else and can be passed as arguments. No new parameter is needed. What will the following code print:

d = deque((42, IndexError, 'spam')) print(list(iter(d.popleft, IndexError))) ? Yury

On 2/6/2014 9:45 PM, Chris Angelico wrote:

On Fri, Feb 7, 2014 at 1:36 PM, Terry Reedy wrote:

...

def __next__(self): try: x = self.func() except Exception as exc: if isinstance(exc, self.sentinel): raise StopIteration from None else: raise if x == self.sentinel: raise StopIteration else: return x

Forgive me if this is a stupid question, but wouldn't this suppress any other thrown exception? I'm looking for a bare 'raise' inside the except block,

which should be there. As I said, but you snipped, '(untested)'

such that any exception is guaranteed to raise something, but if it's a subclass of self.sentinel, it raises StopIteration instead.

-- Terry Jan Reedy

On 2/6/2014 11:15 PM, Terry Reedy wrote:

...

On Fri, Feb 7, 2014 at 1:36 PM, Terry Reedy

...

...

def __next__(self): try: x = self.func() except Exception as exc: if isinstance(exc, self.sentinel): raise StopIteration from None else: raise

I just realized that the above is unnecessarily complicated because the expression that follows 'except' is not limited to a builtin exception class name or tuple thereof. (I have never before had reason to dynamically determine the exception to be caught.) So, using a third parameter, replace the 5 lines with 2. except self.stop_exception: raise StopIteration from None

...

...

if x == self.sentinel: raise StopIteration else: return x

-- Terry Jan Reedy

On Fri, Feb 7, 2014 at 5:05 PM, Terry Reedy wrote:

As you showed, it is easy to construct callables that might return an exception before raising it from finite collections with a prev or next method (whether destructive or not).

...

...

...

list(iter(['spam', IndexError, 42].pop, IndexError)) [42] list(iter({'spam', KeyError, 42}.pop, KeyError)) [42, 'spam'] # or [42] or ['spam'], depending on hashing

For these example, I guess Ram is right in suggesting a 3rd parameter. I would, however, just call it something like 'exception' or 'stop_iter'. That would make the description

I honestly wouldn't worry. How often, in production code, will you iterate over something that might return an exception, AND be testing for the raising of that same exception? Or the converse - how often would you be in a position to raise the thing you might want to return, and be annoyed that the raised exception gets squashed into StopIteration? Don't complicate a nice simple API for the sake of that. If you're concerned about security implications of someone manipulating the return values and chopping something off, then just write your own generator: def safe_iter(func, sentinel): try: while True: yield func() except sentinel: pass This will stop when the exception's raised, but not when it's returned. The unusual case can be covered with so little code that the API can afford to ignore it, imo. ChrisA

On Feb 6, 2014, at 22:52, Andrew Barnert wrote:

On Feb 6, 2014, at 22:03, Terry Reedy wrote:

...

On 2/6/2014 11:15 PM, Terry Reedy wrote:

...

...

On Fri, Feb 7, 2014 at 1:36 PM, Terry Reedy

...

...

...

def __next__(self): try: x = self.func() except Exception as exc: if isinstance(exc, self.sentinel): raise StopIteration from None else: raise

I just realized that the above is unnecessarily complicated because the expression that follows 'except' is not limited to a builtin exception class name or tuple thereof. (I have never before had reason to dynamically determine the exception to be caught.) So, using a third parameter, replace the 5 lines with 2.

except self.stop_exception: raise StopIteration from None

Except that you don't have a stop_exception, you have a sentinel, which can be either an object or an exception type.

I'm actually not sure whether it's legal to use, say, 0 or "" as the except expression. In recent 3.4 builds, it seems to be accepted, and to never catch anything. So, if that's guaranteed by the language, it's just a simple typo to fix and your simplified implementation works perfectly.

Reading the docs, it seems like it ought to be ok. In 8.4, it just says: 'For an except clause with an expression, the expression is evaluated, and the clause matches the exception if the resulting object is "compatible" with the exception. An object is compatible with an exception if it is the class or a base class of the exception object or a tuple containing an item compatible with the exception.' So, it seems like 0 is a perfectly valid except expression, which can be checked for compatibility with any exception and will never match. Which is perfect.

On Thu, Feb 06, 2014 at 09:36:19PM -0500, Terry Reedy wrote:

On 2/6/2014 7:10 PM, Ram Rachum wrote:

...

`iter` has a very cool `sentinel` argument. I suggest an additional argument `sentinel_exception`; when it's supplied, instead of waiting for a sentinel value, we wait for a sentinel exception to be raised, and then the iteration is finished.

This'll be useful to construct things like this:

my_iterator = iter(my_deque.popleft, IndexError)

What do you think?

I think this would be a great idea if simplified to reuse the current parameter.

That would be a backwards-incompatible change, for exactly the reason you give below:

It can work in Python because exceptions are objects like anything else and can be passed as arguments.

Right. And there is a big difference between *returning* an exception and *raising* an exception, which is why a new parameter (or a new function) is required. A function might legitimately return exception objects for some reason: exceptions_to_be_tested = iter( [IndexError(msg), ValueError, StopIteration, TypeError] ) def func(): # pre- or post-processing might happen return next(it) for exception in iter(func, StopIteration): # assume the exceptions are caught elsewhere raise exception With the current behaviour, that will raise IndexError and ValueError, then stop. With the suggested change in behaviour, it will raise all four exceptions. We cannot assume that an exception is never a legitimate return result from the callable. "Iterate until this exception is raised" and "iterate until this value is returned" are very different things and it is folly to treat them as the same. [...]

I consider the threat to backward compatibility, because of the added test for exceptions, theoretical rather than actual. It is very rare to write a function that returns an exception,

Rare or not, I've done it, it's allowed by the language, and it is inappropriate to conflate returning a class or instance with raising an exception. It doesn't matter whether it is rare. It is rare to write: iter(func, ({}, {})) nevertheless it would be poor design to have iter treat tuples of exactly two dicts as a special case. Exceptions are first-class values like strings, ints, and tuples containing exactly two dicts. They should be treated exactly the same as any other first-class value. -- Steven

On Fri, Feb 07, 2014 at 05:41:32PM +1100, Chris Angelico wrote:

I honestly wouldn't worry. How often, in production code, will you iterate over something that might return an exception, AND be testing for the raising of that same exception? Or the converse - how often would you be in a position to raise the thing you might want to return, and be annoyed that the raised exception gets squashed into StopIteration?

Doesn't matter how rare it is. Doesn't matter if nobody has ever written production that does it. Or for that matter, if nobody has ever written *non-production* code that does it. "Special cases aren't special enough to break the rules." Exceptions are values just like strings and lists and floats. Probably nobody has ever written this particular piece of code: iter(func, "NOBODY expects the Spanish Inquisition!!!") but would you think it is okay to treat that particular sentinel value differently from every other sentinel value?

Don't complicate a nice simple API for the sake of that.

Your idea of simple is not the same as mine. The current behaviour "the callable is called until it returns the sentinel" is simpler than the proposed behaviour: "the callable is called until it returns the sentinel, unless the sentinel is an exception instance or class, in which case the callable is called until it raises that exception, or one which is compatible with it" This is an ugly API that violates the Zen of Python. I think that is why Raymond has listed this as a recipe rather than modify the function to behave as suggested: http://code.activestate.com/recipes/577155 -- Steven

On 2/7/2014 3:28 AM, Steven D'Aprano wrote:

On Thu, Feb 06, 2014 at 09:36:19PM -0500, Terry Reedy wrote:

...

On 2/6/2014 7:10 PM, Ram Rachum wrote:

...

`iter` has a very cool `sentinel` argument. I suggest an additional argument `sentinel_exception`; when it's supplied, instead of waiting for a sentinel value, we wait for a sentinel exception to be raised, and then the iteration is finished.

This'll be useful to construct things like this:

my_iterator = iter(my_deque.popleft, IndexError)

What do you think?

I think this would be a great idea if simplified to reuse the current parameter.

That would be a backwards-incompatible change, for exactly the reason you give below:

In a later message, I reversed myself, in spite of the actual C signature making it a bit messy. Although help says 'iter(callable, sentinel)', the actual signature is iter(*args), with any attempt to pass an arg by keyword raising TypeError: iter() takes no keyword arguments Let n = len(args). Then the code switches as follows: n=0: TypeError: iter expected at least 1 arguments, got 0 n>2: TypeError: iter expected at most 2 arguments, got 3 n=1: Return __iter__ or __getitem__ wrapper. n=2: if callable(args[0]): return callable_iterator(*args), else: TypeError: iter(v, w): v must be callable If the current signature were merely extended, then I believe the new signature would have to be (if possible) iter(*args, *, stop_iter=<private exception>) But having parameter args[1] (sentinel) be position-only, with no default, while added parameter stop_iter is keyword only, with a (private) default, would be a bit weird. So instead I would suggest making the new signature be iter(iter_or_call, sentinel=<private object>, stop_iter=<private exception>). If sentinel and stop_iter are both default, use current n=1 code, else pass all 3 args to modified callable_iterator that compares sentinel to return values and catches stop_iter exceptions. Either way, the user could choose to only stop on a return value, only stop on an exception, or stop on either with the two values not having to be the same. The only thing that would break is code that depends on a TypeError, but we allow ourselves to do that to extend functions.

...

It can work in Python because exceptions are objects like anything else and can be passed as arguments.

Right. And there is a big difference between *returning* an exception and *raising* an exception, which is why a new parameter (or a new function) is required. A function might legitimately return exception objects for some reason:

exceptions_to_be_tested = iter( [IndexError(msg), ValueError, StopIteration, TypeError] )

def func(): # pre- or post-processing might happen return next(it)

Did you mean next(exceptions_to_be_tested)

for exception in iter(func, StopIteration): # assume the exceptions are caught elsewhere raise exception

With the current behaviour, that will raise IndexError and ValueError, then stop. With the suggested change in behaviour, it will raise all four exceptions.

No, it would still only raise the first two as it would still stop with the return of StopIteration. But the certainty that people would be confused by the double use of one parameter is reason enough not to do it.

We cannot assume that an exception is never a legitimate return result from the callable. "Iterate until this exception is raised" and "iterate until this value is returned" are very different things and it is folly to treat them as the same.

...

[...] I consider the threat to backward compatibility, because of the added test for exceptions, theoretical rather than actual. It is very rare to write a function that returns an exception,

While *writing* such a function might be very rare, Yuri showed how easy it is to create such a callable by binding a collection instance to a method.

Rare or not, I've done it, it's allowed by the language, and it is inappropriate to conflate returning a class or instance with raising an exception.

It doesn't matter whether it is rare. It is rare to write:

iter(func, ({}, {}))

nevertheless it would be poor design to have iter treat tuples of exactly two dicts as a special case.

Exceptions are first-class values like strings, ints, and tuples containing exactly two dicts. They should be treated exactly the same as any other first-class value.

Agreed. -- Terry Jan Reedy

On Friday, February 7, 2014 11:49:12 AM UTC+2, Stephen J. Turnbull wrote:

Steven D'Aprano writes:

...

iter(func, "NOBODY expects the Spanish Inquisition!!!")

+1 Code Snippet of the Week!

A++, would run again :) Now back to serious discussion: I do prefer a separate keyword argument rather than reusing `sentinel`. If `sentinel_exception` is too verbose, then an alternative is `exception`, but just not `stop_iter` which is very undescriptive. Also, I'd support having three separate implementations for the `iter` function, one for just sentinel, one for exception, and one for both, so the iteration would be as quick as possible after the iterator was created. Thanks, Ram.

On 2/7/2014 3:39 AM, Steven D'Aprano wrote:

is simpler than the proposed behaviour:

"the callable is called until it returns the sentinel, unless the sentinel is an exception instance or class, in which case the callable is called until it raises that exception, or one which is compatible with it"

To repeat, for the 3rd or 4th time, this is not the behavior that either I or Ram proposed. I even provided code to be clear as to what I mean at the time. -- Terry Jan Reedy

On 2/7/2014 7:30 PM, Steven D'Aprano wrote:

On Fri, Feb 07, 2014 at 04:54:04PM -0500, Terry Reedy wrote:

...

On 2/7/2014 3:39 AM, Steven D'Aprano wrote:

This

...

...

"the callable is called until it returns the sentinel, unless the sentinel is an exception instance or class, in which case the callable is called until it raises that exception, or one which is compatible with it"

To repeat, for the 3rd or 4th time, this is not the behavior that either I or Ram proposed. I even provided code to be clear as to what I mean at the time.

is critically different from what I actually wrote for how to change the one-line docstring summary

You did write:

No new parameter is needed. We only need to add 'or raises' to the two-parameter iter definition "In the second form, the callable is called until it returns the sentinel." to get "In the second form, the callable is called until it returns or raises the sentinel."

and the code that followed because it adds 'unless the sentinel is an exception instance or class'. Your addtion would change existing behavior before an exception is raised (if ever). I explicitly said in both code and text that I was not proposing to do that, but only to change what happened if and when an exception were raised (at which point there would be no more return values values to worry about.

If you've changed your mind, I'll be glad to hear it,

As I already responded to you at 4:28 EST Friday, I withdrew my reuse-the-parameter proposal, in response to Yuri, 2 hours before the timestamp I have on your first response. In that response to you, I discussed in several paragraphs the actual existing parameter and how to add a third parameter. Since you did not read it, here it is again. ''' In a later message, I reversed myself, in spite of the actual C signature making it a bit messy. Although help says 'iter(callable, sentinel)', the actual signature is iter(*args), with any attempt to pass an arg by keyword raising TypeError: iter() takes no keyword arguments Let n = len(args). Then the code switches as follows: n=0: TypeError: iter expected at least 1 arguments, got 0 n>2: TypeError: iter expected at most 2 arguments, got 3 n=1: Return __iter__ or __getitem__ wrapper. n=2: if callable(args[0]): return callable_iterator(*args), else: TypeError: iter(v, w): v must be callable If the current signature were merely extended, then I believe the new signature would have to be (if possible) iter(*args, *, stop_iter=<private exception>) But having parameter args[1] (sentinel) be position-only, with no default, while added parameter stop_iter is keyword only, with a (private) default, would be a bit weird. So instead I would suggest making the new signature be iter(iter_or_call, sentinel=<private object>, stop_iter=<private exception>). If sentinel and stop_iter are both default, use current n=1 code, else pass all 3 args to modified callable_iterator that compares sentinel to return values and catches stop_iter exceptions. Either way, the user could choose to only stop on a return value, only stop on an exception, or stop on either with the two values not having to be the same. The only thing that would break is code that depends on a TypeError, but we allow ourselves to do that to extend functions. ''' iter(iter_or_call, sentinel=<private object>, stop_iter=<private exception>) is still my current proposal. Serhiy suggested 'stop_iter=StopIteration' and I will explain separately why I think that this would not work. -- Terry Jan Reedy

On 2/7/2014 9:28 AM, Serhiy Storchaka wrote:

07.02.14 10:36, Terry Reedy написав(ла):

...

I wrote the above with the idea that there would be a third parameter to provide an exception. It would have to have a private default like

class _NoException(Exception): pass

Natural default is StopIteration or ().

That does not work because iter has to know if the user called iter with only one argument, an iterable, or with two or (with the proposal) three, with the first argument being a callable. def iter(iter_or_call, sentinel=<private object>, stop_iter=<private exception>): if sentinel == <private object> and stop_iter == <private exception>: # iter_or_call must be an iterable else: # iter_or_call must be a callable return callable_iterator(iter_or_call, sentinel, stop_iter) where callable_iterator has been modified to take the third parameter and raise StopIteration if iter_or_call raises stop_iter. class Callable_iterator: def __next__(self): try: val = self.func() except self.stop_iter: raise StopIteration from None if self.sentinel == value: raise StopIteration else: return val If a user passes sentinel but not stop_iter, the except clause should have no effect and the method should work as it does currently. A default of StopIteration would accomplish that but it does not work for the iter switch. A private exception class also works since no exception raised by self.func should be an instance of such a class (unless the user violates convention). If a user passes stop_iter but not sentinel, the if clause should have no effect and the iteration should continue until there is an exception. So the default sentinel should never compare equal to any value returned by self.func. A private instance of object on the left of == will invoke object.__eq__ which compares by identity. Unless a user violates convention by extracting and using the private instance, the function will never return it. Since value could compare equal to everything, the order for == matters. -- Terry Jan Reedy

On 2/10/2014 4:39 AM, Ram Rachum wrote:

I see that discussion has stalled on this suggestion.

I think of it as more or less finished here and time to open an issue on the tracker, which I plan to do soon, with the specific signature and Python equivalent code I have suggested, as well as your example. If you open one, add me as nosy so I can add the above.

What can we do to move this forward?

Remind me if neither of us have not opened an issue within a week. -- Terry Jan Reedy