Control Flow - Never Executed Loop Body
Hi Python-ideas, this is a follow up from an thread of Python-List: https://mail.python.org/pipermail/python-list/2016-March/705205.html What do you think about the following proposal? *Abstract* Control structures such as "for", "while" and "try" feature some various additional clauses being executed under certain conditions. This proposal adds another extra clause to "for" and "while" in order to provide an easy way to discover whether the body of a loop as never been executed at all. *Motivation** *Coming from the Web application development, we periodically can see a benefit of a hypothetical "empty" clause of Python's for loop like this: for item in my_iterator: # do per item empty: # do something else The same goes for "while" although there the choice of the name "empty" might be questionable. "empty" refers to the involved iterable which is not really existing in case of "while". However, there is not need of introducing inconsistencies among the loop constructs as both feature the "else" clause as well. Better suggestions are welcome; also see the keyword section. *Keyword* Keywords under consideration have been in (an attempt of dispassionate) order of preference: 1) empty -> most obvious, most people responded with a solution to solve the intended problem 2) else -> already taken but to some would be the preferred one 3) or -> as alternative if a new keyword is too much 4) except -> as alternative if a new keyword is too much 5) instead -> less obvious and new keyword *Use-Cases* 1) As humans mostly have a hard time to discover the difference between *nothing**as error* and *nothing**as really empty*, GUI/Web interfaces display an explicit notice in case of empty lists to signify the correct execution of a process/action/search or the other case. 2) Single way and no boilerplate of handling empty loops. As the discussion on Python-List showed are there quite some number of different ways of achieving this kind of functionality. 3) Less errors in development and maintenance. Some of the proposed solutions would not work in some circumstances (either during development or depending on the type of the items) which can hide errors. *Implementation* To be done. *Maintainer* Me. *Issues** *People I talked to suggested "else" as an alternative to "empty" because "empty is not quite clear". Unfortunately, "else" is already taken and is itself not quite clear. "then" has been proposed as an alternative for "else" in an attempt for proper understanding of "else". If that would be an accepted proposal, it would make room for "else" to be used for the usage of the "empty keyword proposed here. *Related Work** *This proposal is partly inspired by template engines like Django or jinja2 which both already provide such facility. Django: {% for item in my_iterator %} ... {% empty %} ... {% endfor %} jinja2: {% for item in my_iterator %} ... {% else %} ... {% endfor %} Best, Sven
On Mar 20, 2016, at 11:12, Sven R. Kunze <srkunze@mail.de> wrote:
Issues People I talked to suggested "else" as an alternative to "empty" because "empty is not quite clear". Unfortunately, "else" is already taken and is itself not quite clear. "then" has been proposed as an alternative for "else" in an attempt for proper understanding of "else". If that would be an accepted proposal, it would make room for "else" to be used for the usage of the "empty keyword proposed here.
Besides the backward compatibility issue, changing "else" to "then" would be horribly confusing. I suspect anyone who thinks that would be an improvement doesn't actually understand or like for...else, and they'd be happier just eliminating it, not renaming it. An else clause is testing that no break was hit inside the loop. Look at a typical example: for elem in seq: if good(elem): break else: raise ValueError The word "then" would make no sense there. "Find the first good element, else raise an exception" makes sense; "find the first good element, then raise an exception" means the opposite of what we want. Anyway, none of this speaks for or against your main proposal, it just means (at least in my opinion) that this alternative option is off the table. More generally, I think if this feature were to be added, "empty" is a reasonable name. The idiomatic way to write it today is something like: elem = empty = object() for elem in seq: do_stuff(elem) if elem is empty: do_empty_seq_stuff() So you're basically looking for syntactic sugar that abbreviated "if ... is empty:", right?
Robert Collins wrote:
oh, ok. sorry, didn't think about the iterator object. ...thinking about it, is there a reason why the boolean interpretation of an empty iterator is true? I know it's not trivial to know if an iterator is empty (still, there are a pair of recipes online, like http://code.activestate.com/recipes/413614-testing-for-an-empty-iterator/ or the first part of the response at http://stackoverflow.com/a/3114423/273593 ) but shoulnd't be more "pythonic" to just have a "falsy" value for an "empty" iterator? -- By ZeD
Not trivial, indeed. Consider the following: def gen(): while random.choice([0, 1]): yield "spam" Is it empty? Is it not? You can't tell when it will be, and while that is a bad example (I hope something like that doesn't get used!), arbitrary generators can have side effects, or be empty under certain circumstances and not under others. That being said, we can maybe make the built-in iterators return a falsey value in such cases, and let third-party iterators handle that on their own, if they feel like doing that. Generators would always be true, but it's an opt-in solution, and custom iterators can already define their own __bool__ if they want. -Emanuel ~ Anything can be a duck if you try hard enough ~
Émanuel Barry writes:
From Vito De Tullio
AFAICS the OP's idea has a trivial and efficient expansion: for item in iterable: # implements __next__ # do for each item empty: # do exactly when iterable raises StopIteration on the first pass becomes item = _sentinel = object() for item in iterable: # do for each item if item is _sentinel: # do exactly when iterable raises StopIteration on the first pass (works in light testing but maybe I've missed something). But to me that means saving one line and a few characters in the trailer clause is not worth syntax.
Is it empty? Is it not? You can't tell when it will be, and while that is a bad example (I hope something like that doesn't get used!),
Replace random with polling an input source or accessing a database, and you get the same nondeterminism. Steve
On Mon, Mar 21, 2016 at 7:01 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
What if 'iterable' is locals().values()? Can you, with perfect reliability, recognize that case? AIUI this is exactly why next() and __next__() are defined to "return a value or raise", rather than "return a value or return a magic no-more-values value", because there's always the possibility that the no-more-values value is a legitimately-yielded value. Maybe this situation isn't important enough or common enough to justify dedicated syntax, but it's definitely a possibility. ChrisA
Chris Angelico wrote:
so, the "correct" way to handle this is something like try: e = next(iterable) except StopIteration: empty_suite() # there is no element else: main_suite(e) # handle the first element for e in iterable: main_suite(e) # handle the rest of the elements ? apart for the readability you need to "copy" the same code two times (or refactor in a function) -- By ZeD
On 21.03.2016 21:57, Vito De Tullio wrote:
Interesting. I didn't think of this solution. However, I assume that when I do "object()" I got something unique among all other objects. So, item = sentinel = object() for item in collection: main_suite(item) if item is sentinel: empty_suite() Is still quite correct, right? Best, Sven
On Tue, Mar 22, 2016 at 11:07 PM, Sven R. Kunze <srkunze@mail.de> wrote:
Sure it is. Perfectly correct.
An iterator can return *any* *object*. That's why Python uses an exception (StopIteration) to signal the absence of an object. More reliable is to use that absence, either by exception or by probing a dictionary's keys: def better(): # Make sure item is unbound try: del item except NameError: pass for item in locals().values(): print("We have:", item) if 'item' not in locals(): print("We have no items.") But now we're getting into the realm of ugly code to deal with edge cases. Like with "yield from", language support can be justified when there's a simple and obvious *but imperfect* way to do something ("yield from x" is not the same as "for item in x: yield item"). ChrisA
On 22.03.2016 13:16, Chris Angelico wrote:
Oh, you are right. That's definitely an ugly edge case. Any proposed solution should just work given the simplicity of the problem.
Interesting.
Exactly. Despite the fact that "yield from" is shorter it can handle edge cases more beautiful. Best, Sven
On Tue, Mar 22, 2016 at 11:27 PM, Sven R. Kunze <srkunze@mail.de> wrote:
It's not enough shorter than the simple 'for' loop to justify the dedicated syntax. But check out its *actual* equivalent code and you'll see that while we're getting toward similar territory, the for-ifempty loop isn't quite here yet :) https://www.python.org/dev/peps/pep-0380/#formal-semantics ChrisA
On 22.03.2016 13:29, Chris Angelico wrote:
I remember that. I was quite surprise to find such a huge amount of equivalent python code to handle all corner and edge cases correctly. If I were to become the one maintaining this, I hope we don't discover even more edge cases. :D Best, Sven
On Tue, Mar 22, 2016 at 01:27:05PM +0100, Sven R. Kunze wrote: [...]
I think we're starting to give far more weight to that case than it deserves. It doesn't deserve a note in the docs, let alone new syntax to "solve it". Nobody is going to do it, except to win a bet. Certainly nobody is going to do it *accidently*.
For plain old iteration, "yield from" does nothing more than iterating over the iterable and yielding. There's no "edge cases" here, there are some pretty major differences. As the PEP says: If yielding of values is the only concern, this can be performed without much difficulty using a loop such as for v in g: yield v However, if the subgenerator is to interact properly with the caller in the case of calls to send() , throw() and close() , things become considerably more difficult. https://www.python.org/dev/peps/pep-0380/ "yield from" doesn't fix some weird edge case with iteration. It provides some significant new functionality which is *very* tricky to get right. That's not the case here with detecting an empty iterator. There are two perfectly adequate and simple ways to detect an empty iterator: # 1 empty = True for x in iterable: empty = False do_stuff() if empty: handle_empty() #2 x = sentinel = object() assert iterable is not locals().values() # *wink* for x in iterable: do_stuff() if x is sentinel: handle_empty() -- Steve
On Tue, Mar 22, 2016 at 11:16:55PM +1100, Chris Angelico wrote:
Are you being sarcastic?
I think that this is a completely artificial edge case that's never going to come up in practice, and is easy to work around: Just Don't Do That. It's perfectly safe if somebody passes you locals() from *another* scope, because it cannot have access to *your* local sentinel. So you only need worry about one case: when *you* pass *your own* locals as the iterable. Hence, *just don't do it*. "But what if I want to iterate over locals() and handle the case where it is empty?" if not locals(): handle_empty else: for value in locals().values(): ... Don't get me wrong, I think it was very clever that you thought of this case. But I don't think this is ever going to come up in practice. -- Steve
On Wed, Mar 23, 2016 at 12:10 AM, Steven D'Aprano <steve@pearwood.info> wrote:
I was, hence the function name.
And make sure the function you call doesn't use sys._getframe to get them for you.
Don't get me wrong, I think it was very clever that you thought of this case. But I don't think this is ever going to come up in practice.
Maybe. But it's an edge case of a form that could come up in other ways - I don't know of a way to enumerate all objects in CPython, but it's a perfectly reasonable way to debug a garbage collector. Point is that iteration can return literally *any* object. The object() sentinel is better than the None sentinel is better than the "if not item" falsiness check, but the only perfect solution is an extremely ugly one (and I'm not guaranteeing that even that is perfect - only that *I* haven't found a flaw in it yet). At some point, the solution is good enough for your code. ChrisA
On Wed, Mar 23, 2016 at 12:21:34AM +1100, Chris Angelico wrote:
Are you saying that I might have code like this: def test(): x = sentinel = object() iterable = some_function() for x in iterable: # you know the rest and some_function() might use _getframe to steal sentinel and return it back to me as iterable = [sentinel]? I'll take my chances with that, thanks very much. I am as concerned by that as I am that some_function might use ctypes to hack the value of integers so that 1 + 1 returns 4. Yes, it could happen. No, I don't care to take any special precautions to avoid it happening. I think a far more likely possibility is that somebody or something has monkey-patched object() to always return the same instance, regardless of who calls it from where, and now my sentinel is the same as everyone else's sentinel. Am I bothered by this? Not in the least. I mean, seriously, we write code all time relying on the behaviour of builtin functions, knowing full well that any piece of code anywhere might monkey-patch them to do something weird. Should we insist on special syntax that returns the len() of sequences because of an edge-case "what if len has been monkey-patched?". No. We just say "well don't monkey-patch len." In case it's not obvious by now, I don't think there's any need for syntax to handle the empty iterable case. -1 on the proposal. -- Steve
Steven D'Aprano wrote:
onestly, I just don't like the sentinel approach... while not perfect I prefer the explicit check on the throws of the StopIteration of the next function iterable = iter(some_function()) try: e = iter(iterable) except StopIteration: # empty else: # stuff with e for e in iterable: # stuff with e -- By ZeD
On Tue, Mar 22, 2016 at 09:36:23PM +0100, Vito De Tullio wrote:
I have no objection to the try...except approach either. Another approach is to call next() with a default: first = next(iterator, sentinel) if first is sentinel: empty else: for x in itertools.chain([first], iterator): process(x) We're spoiled for choice here: there are many ways to process an empty iterable separately from a non-empty one. -- Steve
It’s comparable in purpose to the ``else`` clause on a ``for`` or ``while``. It eliminates the need for a flag variable, like ``was_completed`` or ``is_empty``. It saves two lines of code, but more importantly avoids the accidental flip of True/False when setting the flag variable. Whether that warrants adding new syntax... I’m not convinced by the current proposals, but I can imagine finding a good keyword for the purpose. A keyword as good as ``else`` at least.
On 23 March 2016 at 14:13, Henshaw, Andy <Andy.Henshaw@gtri.gatech.edu> wrote:
For me, after checking all other e-mails, this is the "one and obvious way" of doing it, does not matter if it "seems" boring - it had not to do with being boring, or having one extra variable - it has to do with being readable. Event the variant that would do item = sentinel = object() for ... is less readable, if more elegant. As for adding different syntax for this, I am +0 For the idea of adding "except" clauses to 'with', 'for' and 'while' (along with a "EmptyIteration" exception) I am + 0.5
On 23.03.2016 20:06, Joao S. O. Bueno wrote:
So it seems, you consider a new keyword for the sole purpose of "discovering never executed" less worthwhile than the "except" alternative. What is the reason for this? The potential extensibility? Readability? Btw. I can understand and confirm that the "ifempty" use-case is not as frequent as the length of this thread suggests. So, I would not mind a slightly longer syntax ("except EmptyIteration") as long as it handles ALL corner cases correctly. This actually is my main driver here. I know that there are two or three easy patterns that fail in some cases. If a new syntax would be proposed that cannot handle all of them, it's not worth it. However, the frequency by which people here and on python-list suggest not-always working solutions still gives me a reason to work on this. Best, Sven
On Mon, Mar 21, 2016 at 1:01 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
AFAICS the OP's idea has a trivial and efficient expansion:
I'll take your word for it that this is efficient, and it is trivial in amount of code to write, but it is not the least bit trivial to read or write. It took me a good while to understand what it did, and how it works, and I never would have thought of it myself.
I would have done the klunky thing: _sentinal = False for item in iterable: _sentinel = True # do something if not _sentinel # do something else if the loop never ran almost as compact, less efficient and enough to make me see why the OP want's something new. Howver, the motivating example was "the difference between *nothing**as error* and *nothing** as really empty*, GUI/Web interfaces display an explicit notice in case of empty lists to signify the correct execution of a process/action/search or the other case." Sure -- but this seem to be mixing validation with processing code -- you really want to validate your inputs first. AND: this would mostly be containers, rather than iterables, so the idiom of: if input: for item in input: # do something else: raise EmptyInputError works fine. All that being said: how about "elempty"?, to go with elif? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 21.03.2016 17:31, Chris Barker wrote:
All that being said:
how about "elempty"?, to go with elif?
I am mostly dispassionate about the wording. So, why not elempty? [Btw. most people consider existing keywords for re-using as they cannot be used for variables in the first place] Best, Sven
Chris Barker writes:
By "efficient" I mean only that the loop is unchanged. Bonus is that only one assignment and one test are needed outside of the loop. The only reason I could think of at the time that it might not be trivial to read is that a reader is unfamiliar with the "sentinel = object()" idiom. Otherwise, "for" and "if" are the most basic of Python syntax, so I felt safe in using that word. Writing is another matter, but that's covered under the "sometimes ya gotta be Dutch" clause. :-)
If it's input, yes, but in my experience this would mostly be dealing with internal state such as a database. Then EAFP applies. For example, in paging, an empty page is a natural signal for end of content.
AND: this would mostly be containers,
"Mostly", maybe, but that's not to say that iterators are an unimportant case. And if your application involves a mix of containers and iterators, you may still prefer to use the same idioms for both cases.
All that being said:
how about "elempty"?, to go with elif?
-1 Not to my taste, to say the least.
On 22.03.2016 16:09, Stephen J. Turnbull wrote:
Hmm, it seems there is no easy solution for this. What do you think about an alternative that can handle more than empty and else? for item in collection: # do for item except NeverExecuted: # do if collection is empty It basically merges "try" and "for" and make "for" emit EmptyCollection. So, independent of the initial "never executed loop body" use-case, one could also emulate the "else" clause by: for item in collection: # do for item except StopIteration: # do after the loop Thinking this further, it might enable much more use-cases and stays extensible without the need to invent all kinds of special keywords. So, this "for" might roughly be equivalent to (using the "old for"): class NeverExecuted(StopIteration): pass i = iter(collection) try: e = next(i) except StopIteration: if [NeverExecuted/StopIteration is catched]: raise NeverExecuted else: # do for item e for e in i: # do for item e else: if [StopIteration is catched]: raise StopIteration I hope that's not too convoluted. Best, Sven
Le 22/03/2016 18:21, Sven R. Kunze a écrit :
It's an interesting idea. Generalizing except for more block I mean. Imagine: if user: print(user.permissions[0]) except IndexError: print('anonymous') with open('foo') as f: print(f.read()) except IOError: print('meh') for item in collection: # do for item except NeverExecuted: # do if collection is empty Not only does it allow interesting new usages such as the for loop empty pattern, but it does save a level of indentation for many classic use cases where you would nest try/except.
On Tue, Mar 22, 2016 at 06:21:24PM +0100, Sven R. Kunze wrote:
Possibly with the exception of the three or four previously existing easy solutions :-)
Does this mean that every single for-loop that doesn't catch NeverExecuted (or EmptyCollection) will raise an exception? If not, then how will this work? Is this a special kind of exception-like process that *only* operates inside for loops? What will an explicit "raise NeverExecuted" do?
That doesn't work, for two reasons: (1) Not all for-loops use iterators. The venerable old "sequence protocol" is still supported for sequences that don't support __iter__. So there may not be any StopIteration raised at all. (2) Even if StopIteration is raised, the for-loop catches it (in a manner of speaking) and consumes it. So to have this work, we would need to have the for-loop re-raise StopIteration... but what happens if you don't include an except StopIteration clause? Does every bare for-loop with no "except" now print a traceback and halt processing? If not, why not? -- Steve
On Mar 22, 2016, at 16:21, Steven D'Aprano <steve@pearwood.info> wrote:
The only question is whether any of them are obvious enough (even for novices). If not, we could argue whether any proposed change would be significantly _more_ obvious. And, if so, then the question is whether the change is worth the cost. But I think what we have is already obvious enough. (Especially since 90% of the time, when you need to do something special on empty, you explicitly have a sequence, not any iterable, so it's just "if not seq:".) So really, this proposal is really just asking for syntactic sugar that complicates the language in exchange for making some already-understandable code a little more concise, which doesn't seem worth it.
Elsewhere he mentioned that EmptyCollection would be a subclass of StopIteration. Presumably, every iterator type (or just the builtin ones, and hopefully "many" others?) would have special code to raise EmptyCollection if empty. Like this pseudocode for list_iterator: def __next__(self): if not self.lst: raise EmptyCollection elif self.i >= len(self.lst): raise StopIteration else: i = self.i self.i += 1 return self.lst[self.i] Or, alternatively, for itself would do this. The for loop bytecode would have to change to stash an "any values seen" flag somewhere such that if it sees StopIteration and hasn't seen any values, it converts that to an EmptyCollection. Or any of the other equivalents (e.g., the compiler could unroll the first PyIter_Next from loop from the rest of them to handle it specially). But this seems like it would add a lot of overhead and complexity to every loop whether desired or not.
Presumably that's the same question as what an explicit raise StopIteration does. Just as there's nothing stopping you from writing a __next__ method that raises StopIteration but then yields more values of called again, there's nothing stopping you from raising NeverExecuted pathologically, but you shouldn't do so. M
I think there always is. IIRC, PyObject_Iter (the C API function used by iter and by for loops) actually constructs a sequence iterator object if the object doesn't have tp_iter (__iter__ for Python types) but does have tp_sequence (__getitem__ for Python types, but, e.g., dict has __getitem__ without having tp_sequence). And the for loop doesn't treat that sequence iterator any different from "real" iterators returned by __iter__; it just calls tp_next (__next__) until StopIteration. (And the "other half" of the old-style sequence protocol, that lets old-style sequences be reversed if they have a length, is similarly implemented by the C API underneath the reversed function.) I'm on my phone right now, so I can't double-check any of the details, but I'm 80% sure they're all at least pretty close...
I think this could be made to work: a for loop without an except clause handles StopIteration the same as today (by jumping to the else clause), but one that does have one or more except clauses just treats it like a normal exception. Of course this would mean for/except/else is now legal but useless, which could be confusing ("why does my else clause no longer run when I add an 'except ValueError' clause?"). More generally, I think the fact that for/except StopIteration is almost but not quite identical to plain for would be confusing more often than helpful. But I think it is a coherent proposal, even if it's not one I like. :)
On Mar 23, 2016, at 03:08, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
But I think it is a coherent proposal, even if it's not one I like. :)
And what do you think about adding except clauses to if ? with ? while ?
Well, they don't have the problem that for has (for is already implicitly handling a specific exception in a specific way; none of them are), so they're coherent even without a solution to that problem. (With has the additional problem that it may not be immediately obvious on first glance whether the exit gets calls before or after the except clause, but if so, people can read the docs the first time they come across it and remember it after that.) But I also think they're even less necessary. They'd all be pure syntactic sugar for nesting the statement and a try/except, so we'd be making the language more complicated to learn and remember, and encouraging more code that isn't backward compatible. That's not a huge cost, but the benefit isn't very big either. For _really_ short cases, I think we want except expressions (PEP 463); for longish code, there's nothing wrong with an explicit try; the range for which hiding an implicit try inside if and while would really improve things is small enough that I don't think the benefit outweighs the cost. But maybe some good examples of realistic 3-liners that are significantly improved by the change would convince me (and, more importantly, convince a majority of the others who are skeptical), so I'll keep an open mind.
Le 23/03/2016 16:53, Andrew Barnert a écrit :
Would love to see except expression win, but hasn't been rejected by the BDFL ? Plus it's not really related to the current proposal.
But maybe some good examples of realistic 3-liners that are significantly improved by the change would convince me (and, more importantly, convince a majority of the others who are skeptical), so I'll keep an open mind.
The more I think about it, the more I think it's bad idea. We are mixing semantics of loops and exceptions or conditions and exceptions. I withdraw this part of the "general exceptions" proposal. I'm going back to the simpler and less dangerous proposal: for stuff in foo: bar(stuff) or: default()
On Mar 23, 2016, at 09:48, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
I thought PEP 463 was just stalled waiting on a reference implementation before he'd declare either way? At any rate, the reason it's related is that every case I could think of where I'd want to use if/except is so trivial that I'd much rather use an if expression in an except expression. There may be cases that are not quite trivial enough to demand that, but still small enough that saving one line of code makes a difference; just because I can't think of any certainly doesn't mean they don't exist. :) But, given the rest of your reply, I guess it doesn't matter.
You may want to get everyone back on track by writing a complete proposal (does or come before or after else? what exactly are the semantics? how does it translate to bytecode, if you understand how for already translates?) with just this suggestion some motivating examples. I still think this isn't really necessary. In every case I can think of where I really want to do something special on empty, I definitely have a sequence, and can just do "if not foo:", or I need a count rather than just an empty flag, or I'm already doing some EAFP handling, or... But again, just because I can't think of examples doesn't mean they don't exist. It just implies that maybe the other people you're trying to convince can't think of them either, so it would be better for you to show us.
On 23.03.2016 16:53, Andrew Barnert via Python-ideas wrote:
Could you describe what you mean by coherent?
(With has the additional problem that it may not be immediately obvious on first glance whether the exit gets calls before or after the except clause, but if so, people can read the docs the first time they come across it and remember it after that.)
But I also think they're even less necessary. They'd all be pure syntactic sugar for nesting the statement and a try/except, so we'd be making the language more complicated to learn and remember,
I completely disagree here.
and encouraging more code that isn't backward compatible.
What exactly is the problem with that? Everything discussed on python-ideas would be 3.6+ anyway.
That's not a huge cost, but the benefit isn't very big either. For _really_ short cases, I think we want except expressions (PEP 463); for longish code, there's nothing wrong with an explicit try; the range for which hiding an implicit try inside if and while would really improve things is small enough that I don't think the benefit outweighs the cost.
That's your opinion here. And I find it interesting that others already have thought of the merging of try and if, for and while. It would definitely come in handy. But I understand from your previous responses that you'd rather use Assembler because it is has no unneeded syntactic sugar. ;-) Just kidding.
But maybe some good examples of realistic 3-liners that are significantly improved by the change would convince me (and, more importantly, convince a majority of the others who are skeptical), so I'll keep an open mind.
From what I can see regarding the if-except proposal, the improvement would be constant. So, either you like it (as I do) or you don't see them as necessary (as you do). There is nothing somebody can do to convince you that saving 1 line of code and 1 indentation level across 20 lines of code are good or bad. Best, Sven
On Mar 23, 2016, at 13:26, Sven R. Kunze <srkunze@mail.de> wrote:
Someone earlier claimed that it's impossible to turn the vague proposal into any actual sensible semantics that could be implemented by Python. That would make it incoherent. But I don't believe it's true, so I tried to show how it could be defined well enough to be implementable. For the other three statements, I don't think that argument even comes up; I think the desired semantics are obvious enough that nobody can claim the proposal is incoherent.
Yes, but it's still a cost, that has to be weighed against the benefits. Some new syntax (like "with" or "yield from") is so useful that it's worth encouraging new code to use the new features and not be backward compatible. But that doesn't mean every piece of syntax that might be nice in a brand-new language is so useful that it's worth doing that. It's a high bar, and just ignoring the bar doesn't help you meet it.
You clearly don't understand my precious responses. If you don't see the difference between good syntactic sugar and useless syntactic sugar, then I can't explain to you why Python is better than assembler. :P
So you're refusing to provide examples because you're sure that no examples would convince anyone? In that case, I'm pretty sure your proposal is dead in the water, so I might as well stop responding.
On 23.03.2016 23:17, Andrew Barnert wrote:
You clearly don't understand my precious responses. If you don't see the difference between good syntactic sugar and useless syntactic sugar, then I can't explain to you why Python is better than assembler. :P
Good syntactic sugar is sweet. What means good to you? ;-)
So you're refusing to provide examples because you're sure that no examples would convince anyone?
No, I didn't say that. I don't need examples mostly because I see them each day. But because you asked, here you are: class MyClass: def mymethod(self): try: if my_condition: dosomething() elif another_condition(): makesomething() andanythingelse() else: butdontforgetaboutthis() except SomeException: pass finally: return None Would turn into: class MyClass: def mymethod(self): if my_condition: dosomething() elif another_condition(): makesomething() andanythingelse() else: butdontforgetaboutthis() except SomeException: pass finally: return None 1 line saved and 7 lines with 1 indentation level less (maybe even related to PEP8 and 80 chars line length). You might call it useless, I call it good. Regarding Python syntax, we are constantly going into territory where there is nothing much left to improve. The big "good syntactic sugar" is already done. However, I don't think we should stop here. 10 smaller improvements are as good one big one. Btw. I don't believe we could improve all Python code out there with this change, however when you are in such a situation you really appreciate it if you can do it. Especially when the surrounding source code already blows your mind; you are grateful for every single simplification (less lines, less indentation). I don't know what Python code you maintain but this would our lives easier. Best, Sven
On 23.03.2016 00:56, Andrew Barnert via Python-ideas wrote:
It was reasonable for both Django and for jinja2 to add this. People actually asked for it https://github.com/mitsuhiko/jinja2/issues/77
Repeating that you think it will be a sequence will make it one. We cannot keep track of all possible implementations in our code base. So, if somebody changes from sequence to iterator for whatever reason, it should work without weird errors.
Did you even think what you just said? Almost everything in Python is "just syntactic sugar" compared to most other Turing-complete languages. To put it differently: every construct that abstracts away "goto" is "just syntactic sugar".
Interesting. That would be an alternative approach.
Or, alternatively, for itself would do this.
I think the most important question is: do we want to support the "ifempty" feature in the first place? I can see a lot of discussion around it; that makes it controversial and that means there is half/half support/rejection (support by more people + rejection by less people but those have many good questions we would need to answer before we get this proposal to work). The only real reason against it so far: "it makes the language more complicated because I don't need it". Not entirely compelling but understandable. I further see that people would like this feature GIVEN a very GOOD syntax/approach and I entirely agree with that.
The for loop bytecode would have to change to stash an "any values seen" flag somewhere such that if it sees StopIteration and hasn't seen any values, it converts that to an EmptyCollection. Or any of the other equivalents (e.g., the compiler could unroll the first PyIter_Next from loop from the rest of them to handle it specially).
Something like that.
But this seems like it would add a lot of overhead and complexity to every loop whether desired or not.
If the "for" does not have any empty-equivalent clauses, there is no need to introduce that overhead in the first place. So we can conclude: 1) none overhead for regular "for"s 2) less overhead for "for-ifempty" because it would be done in C and not in Python
I think I would have to deal with the old protocol given the proposal is accepted.
One could disallow "else" in case any "except" is defined.
More generally, I think the fact that for/except StopIteration is almost but not quite identical to plain for would be confusing more often than helpful.
You bet how people think about "else". "So, 'else' is always executed after the for?" "Yes, but only when there is no 'break' executed in the 'for'" "... *thinking* ... okay ..."
But I think it is a coherent proposal, even if it's not one I like. :)
Best, Sven
On Mar 23, 2016, at 13:17, Sven R. Kunze <srkunze@mail.de> wrote:
I've said that if be happy to see any counterexamples, where you really do need this with iterators. So far, nobody has provided one. Of course absence of proof isn't proof of absence, but "you can't prove that it's impossible anyone will ever need this" is not a good rationale for a language change.
There's a big difference between syntactic sugar that makes hard-to-follow code more readable, and syntactic sugar that only makes some already-understandable code a little more concise. The former may belong in the language, the latter very rarely does. You seem to think that there's no inherent cost to adding new features to a language. For example, you later say:
The only real reason against it so far: "it makes the language more complicated because I don't need it". Not entirely compelling but understandable.
The more complicated the language is, the harder it is to keep it all in your head, to spot the control flow while skimming, to trace out the details when necessary, etc. Also, the more things you add, the more places there are for inconsistencies to creep in. (Of course it also makes the language harder to implement and maintain, but those are less important.) Only accepting changes that are actually worth the cost in increased complexity is a big part of what makes Python more readable than "dumping-ground" languages that have C-for, Python-for, while, do-while, until, repeat-until, and loop, some in both statement and expression variants, and some also writable in postfix form.
Not true. The first implementation I suggested, putting EmptyCollection into every iterable, requires the overhead in every case. The second one, changing the way the existing bytecodes work, means making frame objects (or _something_) more complicated to enable stashing the flags, which affects every for loop. The third, unrolling the first PyObject_Iter, has to be done if there's any code that can inspect the current exception state, which can't be statically determined, so it has to be always done. If you have a _different_ implementation, I'm happy to hear it. I supplied all the versions I could think of because another critic (Stephen? I forget...) implied that what you wanted was impossible or incoherent, and it clearly isn't. But it may not be a good idea to let a critic of your idea come up with the implementation. :)
For which of the three implementations? I'm pretty sure all of them would have significant overhead.
No, because, as I just explained, the old protocol is taken care of by wrapping old-style sequences in iterator objects, so as far as the for-loop code is concerned, they look identical to "new-style" iterables.
But that's just an extra rule to implement (and remember) for no real benefit. Why not just document that for/except/else is generally useless and shouldn't be written, and let linters flag it?
More generally, I think the fact that for/except StopIteration is almost but not quite identical to plain for would be confusing more often than helpful.
You bet how people think about "else". "So, 'else' is always executed after the for?" "Yes, but only when there is no 'break' executed in the 'for'" "... *thinking* ... okay ..."
I don't understand your point here. Because there's already something in the language that you find confusing, that gives us free rein to add anything else to the language that people will find confusing?
On 23.03.2016 23:11, Andrew Barnert wrote:
I've said that if be happy to see any counterexamples, where you really do need this with iterators.
1) you conveniently snipped the example I've given: somebody wrote functionality assuming a sequence (as innocently as most of the contributors to python-list and python-ideas); later the input sequence will be changed to an input iterator; "if len(collection)" may hide errors then 2) the colleague who basically initiated the whole discussion needed to implement some sort of "on-the-fly" converter pulling specific items from a third-party system A and pushing the converted items to another third-party system B. In case none items are there to be transferred and converted, a placeholder item still needs be pushed to system B. [yes, that is how somebody can make money; customers sometimes have interesting requirements]
The more complicated the language is, the harder it is to keep it all in your head, to spot the control flow while skimming, to trace out the details when necessary, etc.
And that is exactly why I and other like to see such an addition. It's not just the idea of a single person. It reduces the amount of lines of code and (for the correct implementation of all corner cases) it has less indentation and is presumably faster.
Only accepting changes that are actually worth the cost in increased complexity is a big part of what makes Python more readable than "dumping-ground" languages that have C-for, Python-for, while, do-while, until, repeat-until, and loop, some in both statement and expression variants, and some also writable in postfix form.
I understand that and want to keep it that way as you do. It seems that languages like Django and jinja2, which are 10x more concerned about feature reduction than Python, have {% empty %} or {% else %}. So, Python would not be the first language to support this feature.
Not sure if that can be easily done, but as I understand it, there would be two alternative bytecode implementations available. Depending on whether the "for" has an "empty"-equivalent clause, the "emptified/except/slow for" is used. If the "for" doesn't have such clause, the "original" "for" is used. So, depending on the actual code, the bytecode is different. Does this makes sense?
Oh, then I had interpreted what you said the wrong way. Thanks for explaining again.
But that's just an extra rule to implement (and remember) for no real benefit. Why not just document that for/except/else is generally useless and shouldn't be written, and let linters flag it?
Good idea!
Because there's already something in the language that you find confusing, that gives us free rein to add anything else to the language that people will find confusing?
I am open for suggestions. My argument here is that "except StopIteration" is not better/worse than "else" but the former is extensible. Best, Sven
Andrew Barnert via Python-ideas writes:
I presented a good try at a one-extra-line pure Python implementation (which is still good enough that I'm -1 on syntax, and waiting for a compelling use case for adding "for ... except"). It was Chris Angelico who shot a hole in it with locals().values().
implied that what you wanted was impossible or incoherent, and it clearly isn't.
I don't think there was ever a claim that it was incoherent/impossible or even inefficient at C-level. The point of the mildly obscure[1] item = sentinel idiom was to show that syntax isn't needed to avoid changing the loop body. I don't even know if Chris A would disagree that I succeeded well enough. Footnotes: [1] I know that several people found the item = _sentinel approach unreadable. Which surprised me: I'm usually the one who takes hours to understand the TOOWTDI one-liner. Maybe my newt-ness got bettah!
Let me start by saying I'm not against the proposal, and think it could make a lot of sense when properly implemented. However, I would like to specificially respond to the following. On Wed, Mar 23, 2016 at 09:17:32PM +0100, Sven R. Kunze wrote:
It would be good to remember that both Django and jinja2 are templating languages, and as such they impose different constraints on the user, as well as the sentiment that templates should be (nearly) devoid of all logic. Due to those constraints, adding an `empty` in the template language makes a lot more sense than in a general-purpose programming language. Not to mention the fact that representing 'no results' as an empty page is bad UX, but representing 'no results' from an API as an empty [list, generator, ...] is good design. Again, I like the proposal for the edge-cases it tries to solve (and sometimes I have seen use-cases where it makes sense), but please make sure you have better arguments and examples than 'look at Django and jinja2'.
On Mar 20, 2016, at 16:39, Vito De Tullio <vito.detullio@gmail.com> wrote:
It's not just "not trivial". The only way to do it with full generality (working for, e.g., generators) is to wrap the Iterator in a "peekable iterator" type (or something similar but more powerful, like tee). But that changes the semantics of the iterator: every time you consume element #n, it produces #n+1, not #n. And it's not hard to think of generators where that would be horribly inappropriate. For example, imagine a generator that's reading responses off a network socket. If you try to read response #n+1 when you haven't sent request #n+1 yet (because you're about to look at response #n), it'll fail, or block, or read incorrect information, or something else terrible. And there are other problems. For example, what if the generator needs to be able to handle g.throw; if you've replaced iterable with chain([peeked], iterable) or similar, that won't work. Of course the iterator protocol _could_ have been designed differently (consider C++, where iterators have separate operators to get the current value and to advance to the next one), which would have changed the design of generator functions and generator expressions and various other things that came later, so these particular problems would never have arisen. There are obvious advantages and disadvantages to each design. But Python chose one design a decade and a half ago, and a lot has been built on top of that design, and it's not going to change now. And, given that design, there's no way to make iterators peekable. Which means that it wouldn't be pythonic to make empty iterators falsey, because that's impossible to do without peeking, and I'm pretty sure the only reason the Zen doesn't say possible is better than impossible is that it's so obvious it doesn't need to be said. :) It's also worth noting that what you're asking for is really just a special case of LBYL vs. EAFP. Usually, it's better to just use the object and then see what happened, rather than checking in advance what will happen if you use it. Sometimes you can't avoid LBYL, or it's better because of the specifics of what you're doing, but in general, EAFP is the "default" way of thinking in pythonic code.
On Mar 20, 2016, at 16:54, Andrew Barnert via Python-ideas <python-ideas@python.org> wrote:
Possibly a better way to put this than my other answer: When you really do need to deal with any arbitrary iterable, it's more Pythonic to use the "if element is empty" post-check. When you expect your input to be a collection* (as was clearly your intuition here), go ahead and make use of that, using exactly the "if seq:" code that you wrote. If you need to deal with an iterable that might be an iterator rather than a collection, but if so it's guaranteed to be safely peekable... Well, I don't think that will ever actually come up, but if it does, you can do the peek-and-stuff-back thing instead of the "if element is empty". I think it would probably be simpler and more readable to do the post-test, and it would also avoid having to get across to your reader the idea that you can handle both collections and safely-peekable iterators but not non-safely-peekable iterators, but if you have some reason for that precondition that you wanted to express anyway, maybe using the peek-and-stuff code helps get it across? * Or "reiterable" or "repeatable iterable" or "non-iterator iterable" or any of the other identical or closely parallel concepts. I don't want to restart the argument about which one is more useful here; they all make the right distinction in this case.
On Sun, Mar 20, 2016 at 01:16:50PM -0700, Andrew Barnert via Python-ideas wrote:
"then" is my idea, and not only do I understand "for...else", but I like it and use it, and certainly don't want to eliminate it. I just hate the keyword used. Let me just start by saying that I realise that actually changing "for...else" to "for...then" is at least 8 years too late, so I know this is a non-starter. It would be particularly confusing to change "else" to "then" AND add a new "else" with different semantics at the same time.
An else clause is testing that no break was hit inside the loop. Look at a typical example:
That's one way of thinking about it. But I don't think it is helpful to think of it as setting an invisible flag "a break was hit inside the loop", and then testing it. I think that a more natural way to think about it is that "break" jumps out of the entire for...else compound statement. This has the big advantage that it actually matches what the byte code does in all the versions I've looked at. The "else" block is *unconditionally* executed after the "for" block. There's no "if not flag". Hence "then" is a better name for the construct: "for ... then ...". "break" doesn't set a flag, it jumps right out of the "for...else" statement altogether, not just out of the loop part, but the "else" part as well. (As I said, this matches what the byte code actually does.) As a beginner, I spent a very long time completely perplexed and confused by the behaviour of "for...else" because I understood it to mean "run the for block, *or else* run the else block". In other words, I understood from the keyword that "else" ran *if the for block didn't*, i.e. when the loop iterator is empty. A perfectly natural mistake to make, and I'm not the only person to have made it. This (wrong, incorrect) interpretation matches the most common and familiar use of "else", namely in if...else statements: if ...: a else: b You can get a, or b, but not both. In English, "else" represents an alternative. This does not come even close to matching the behaviour of for...else, which (in the absense of a "break" executes a *and* b, rather than a *or* b: for ...: a else: b I'm not Dutch, but I think that "else" is not a good name for a block of code which unconditionally executes after the for/while/try block. I think it's also unfortunately that it often looks like an indentation error: for x in seq: do_stuff() if condition: break else: do_more() I've seen people "fix" the indentation on code like this and wonder why the code then does the wrong thing. But, like I said, we're stuck with it, and I'm not seriously proposing a change. I think we'd be better off now if the keyword had been "then" from the beginning, but the pain of changing it *now* outweighs the benefit. -- Steve
On Mon, Mar 21, 2016 at 12:32 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Exactly. I came across this situation in a C++ program, and ended up writing the 'break' as a 'goto' - something like this, but with better names: for (int i=0;i<limit;++i) { do_stuff(); if (condition) goto for_else; } more_stuff(); for_else: And the comment against the goto said something along the lines of "this wants to be break-and-a-bit" - it needs to break out of both the loop and the code that follows it. "else" isn't the best keyword, but "finally" might give the wrong impression due to its main use with exceptions. It probably would be a better choice though. ChrisA
On Mar 20, 2016, at 18:32, Steven D'Aprano <steve@pearwood.info> wrote:
The way I've always thought about it is that it's more like a try/else than an if/else: it runs unless you've jumped out of the whole for statement (via break instead of raise). I think Nick Coghlan has a blog post that explains this very nicely, showing both the novice-understandable intuition and how a theoretical Python implementation could implement it (which is close to how CPython actually does, but maybe simpler), so I won't go over the details here. From what you write later, you also don't like the naming of try/else--in which case it's not surprising that you don't like the naming of for/else.
The "else" block is *unconditionally* executed after the "for" block.
The key difference is whether you treat "unless jumped out of" as meaning "unconditionally". I can see the sense of your way of looking at it (there's certainly a sense in which "while True:" loops "unconditionally", even if you have a break or return inside the loop, right?), so I understand where you're coming from. However, I still think _most_ people who don't like for/else don't actually understand it. But of course I can see that as another argument against the naming--if it's confused this many people over the last 20 years, maybe it's inherently confusing? At any rate, as you say at the end, there's no point arguing about this. The "else" clause isn't going to be renamed, and, even if it were, it wouldn't be reused for this new purpose, so that alternative to "empty" isn't worth discussing. (If I were designing a new Python-like language, I might rethink using "else"--but I don't think I'd use "then", or just leave it out the way Boo did.)
On Mar 20, 2016 10:10 PM, "Andrew Barnert via Python-ideas" < python-ideas@python.org> wrote:
On Mar 20, 2016, at 18:32, Steven D'Aprano <steve@pearwood.info> wrote:
On Sun, Mar 20, 2016 at 01:16:50PM -0700, Andrew Barnert via
Python-ideas wrote: than an if/else: it runs unless you've jumped out of the whole for statement (via break instead of raise). I think Nick Coghlan has a blog post that explains this very nicely, showing both the novice-understandable intuition and how a theoretical Python implementation could implement it (which is close to how CPython actually does, but maybe simpler), so I won't go over the details here.
From what you write later, you also don't like the naming of try/else--in
which case it's not surprising that you don't like the naming of for/else. I read the "else:", when attached to "try:", as "alternative to except:" (which makes sense even if no explicit "except" block exists). In "if-else", it means "alternative to if:" (or "if: and elif:"). If "switch-case" existed, "else:" would mean "alternative to case:". But in "for:"/"while:", there is no block keyword that the "else:" is the alternative to. It definitely isn't the alternative to the loop block itself, but that's probably the most common wrong intuition. (Disclaimer: I use "for-else" and "try-else" when I can. I don't dislike them at all. I'm just trying to explain the newbie confusion.)
On 3/20/2016 9:32 PM, Steven D'Aprano wrote:
Why is the truth a mistake?
and I'm not the only person to have made it. This (wrong, incorrect) interpretation
To me, this is disrespectful of other people.
matches the most common and familiar use of "else", namely in if...else statements:
Block a may never be executed. In any case, once it is, the loop starts over, the test is repeated, and either a or b is executed.
I see it differently. For both while and for, block b is the alternative to executing a, when the condition (explicit in 'while', implicit in 'for) is false, just as in an 'if' statement. I know I am not going to convince you, but please don't call me 'wrong' for seeing why 'else' makes sense and for understanding how 'while' is constructed from 'if' and 'jump'. The essential difference between an if statement and a while statement is the jump back that causes a repeat of the test. -- Terry Jan Reedy
On Sun, Mar 20, 2016 at 11:19:54PM -0400, Terry Reedy wrote:
It's not the truth. The else block does *not* only run if the for block doesn't run. It runs regardless of whether the loop iterable is empty or not.
It's a statement of fact. I have seen other people make the same mistake I made: interpreting the "else" clause as only running if the loop iterable is empty. People have been mislead by the keyword. What's disrespectful about pointing this out?
It's an IF...else statement. There's no loop.
That's not what for...else does. Try it with an empty sequence and a non-empty sequence: for x in [1,2]: print("inside for block") else: print("inside else block") for x in []: print("inside for block") else: print("inside else block") BOTH the empty and non-empty cases print "inside else block". It is certainly not "the alternative". -- Steve
I've taught enough and I'm sure everyone else here has too, to know that the "else" in a for loop in non-intuitive to a lot of folks. And maybe a different keyword would have been clearer. But it is what it is, could we keep this discussion to the proposed addition? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Le 21/03/2016 17:34, Chris Barker a écrit :
+1 I would love "else" semantics to be changed as well, but we can't. What about: for x in stuff: foo(x) or: bar() It's a bit weird when you read it the first time, but: - It conveys pretty well the intent. - While it still closes to the original "or" semantics, you can't confuse this syntax with the other syntax for "or". - I can't think of a use case where the parser can find that ambigious. - It's short.
On 21/03/2016 18:37, Michel Desmoulin wrote:
The trouble with "or" or "else" after a for-loop is that it suggests (following English usage) alternative actions: Either execute this for-loop, or do something else. If I were designing Python 4, I might opt for for x in stuff: foo(x) ifnobreak: bar() Pro: "ifnobreak" is at least explicit, and not particularly likely to clash with an already-used variable name. Con: "ifnobreak" is too long and wordy. But I can't think of anything better. "aftercomplete", "whencomplete" are just as bad. Hell, I'm coming round to "then". What about allowing "then" to have an indent between that of the for-statement and the for-body: for x in stuff(): foo(x) then: bar() Of course, you still have to learn it, like all idioms, but having learned it, isn't it a bit more readable? The idea could be extended to other suites, conveying that everything indented is associated with the initial statement of the suite: try: foo() except SomeException: bar() Rob Cliffe
Le 22/03/2016 00:53, Rob Cliffe a écrit :
But that's exactly the goal of "or" here. Run only if the loop is never executed since stuff is empty.
Python 4 won't break compat the way Python 3 did (Guido's words), so Python 4 is not a better opportunity for introducing new stuff than any other release. But I do like the "nobreak" better than else (you can drop the if, it's just noise). However, again, we can't remove else, and "there should be one way to do it" prevents us to add duplicates keywords.
On Mon, Mar 21, 2016 at 12:35 PM Chris Barker <chris.barker@noaa.gov> wrote:
I'm not sure the keyword being "else" is the real problem. It's certainly correlated, but I've noticed a similar confusion for "finally" and "else" in "try" blocks, where the keywords are quite natural. The confusion might in fact simply be a difficulty with the concept "do this if the loop completed without breaking" rather than a difficulty linking that concept to the word "else". Either way, that's a tangent, and I agree that discussion on "else" should be (mostly) separate from a discussion on adding a new keyword.
Itertools is great, and some functions in it are more used than others: - islice; - chain; - dropwhile, takewhile; Unfortunatly many people don't use them because they don't know it exists, but also are not aware of the importance of generators in Python and all in all, the central place iteration has in the language. But I must confess that after 12 years of Python, I always delay the use of it as well: - I have to type the import in every single module and shell session (ok I got PYTHONSTARTUP setup to autoimport, but most people don't). - All functions fell verbose for such a common use case. - If I start to use it, I can say good by to simpler syntaxes such as [] and +. - It always take me a minutes to get dropwhile/takewhile right. They works the opposite way of my brain. The changes I'm going to propose do not add new syntax to Python, but yet would streamline the use of this nice tool and blend it into the language core. Make slicing accept callables ============================= One day someone asked me something similar to: "I got a list of numbers, how do I filter this list so that I stop when numbers are bigger than 4." So I said: print([x for x in numbers if x > 4]) But then he said: "No, I want to stop reading any number I encounter after the first x > 4." "Oh". Then: import itertools def stop(element): return not element > 4 print(list(itertools.takewhile(stop, numbers)) I actually got it wrong 2 times, first I forgot the "not", then I mixed up the parameters in takewhile. I was going to then introduce lambda but my colleagues looked at me in a sad way after glancing at the code and I backed up. So my first proposal is to be able to do: def stop(element): return element > 4 print(numbers[:stop]) It's quite pythonic, easy to understand : the end of the slice is when this condition is met. Any not the strange way takewhile work, which is "carry on as long as this condition is met". We could also extend itertools.islice to accept such parameter. Slicing any iterable ====================== Now, while I do like islice, I miss the straigthforwardness of [:]: from itertools import islice def func_accepting_any_iterable(foo): return bar(islice(foo, 3, 7)) It's verbose, and renders the [3:7] syntaxe almost useless if you don't have control over the code creating the iterable you are going to process since you don't know what it's going to be. So the second proposal is to allow: def func_accepting_any_iterable(foo): return bar(foo[3:7]) The slicing would then return a list if it's a list, a typle if it's a tuple, and a islice(generator) if it's a generator. If somebody uses a negative index, it would then raises a ValueError like islice. This would make duck typing and iteration even easier in Python. Chaining iterable ================== Iterating on heterogenous iterable is not clear. You can add lists with lists and tuples with tuples, but if you need more, then you need itertools.chain. Few people know about it, so I usually see duplicate loops and conversion to lists/tuples. So My first proposal is to overload the "&" operator so that anything defining __iter__ can be used with it. Then you can just do: chaining = "abc" & [True, False] & (x * x for x in range(10)) for element in chaining: print(element) Instead of: from itertools import chain chaining = chain("abc", [True, False], (x * x for x in range(10))) for element in chaining: print(element)
On Tue, Mar 22, 2016 at 10:06 AM, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
You're not the first to ask for something like this :) Let's get *really* specific about semantics, though - and particularly about the difference between iterables, iterators, and generators.
This cannot be defined for arbitrary iterables, unless you're proposing to mandate it in some way. (It conflicts with the way a list handles slicing, for instance.) Even for arbitrary iterators, it may be quite tricky (since iterators are based on a protocol, not a type); but maybe it would be worth proposing an "iterator mixin" that handles this for you, eg: class IteratorOperations: def __getitem__(self, thing): if isinstance(thing, slice): if has_function_in_criteria(slice): return self.takeuntil(s.start, s.stop) return itertools.islice(...) def takeuntil(self, start, stop): val = next(self) while start is not None and not start(val): val = next(self) while stop is None or not stop(val): yield val val = next(self) As long as you inherit from that, you get these operations made available to you. Now, if you're asking this about generators specifically, then it might be possible to add this (since all generators are of the same type). It wouldn't be as broad as the itertools functions (which can operate on any iterable), but could be handy if you do a lot with gens, plus it's hard to subclass them.
Again, while I am sympathetic to the problem, it's actually very hard; islice always returns the same kind of thing, but slicing syntax can return all manner of different things, because it's up to the object on the left:
You don't want these to start returning islice objects. You mentioned lists, but other types will also return themselves when sliced. Possibly the solution here is actually to redefine object.__getitem__? Currently, it simply raises TypeError - not subscriptable. Instead, it could call iter() on itself, and then attempt to islice it. That would mean that the TypeError would change to "is not iterable" (insignificant difference), anything that already defines __getitem__ will be unaffected (good), and anything that's iterable but not subscriptable would automatically islice itself (potentially a trap, if people don't know what they're doing).
Again, anything involving operators is tricky, since anything can override its handling. But if you require that the first one be a specific iterator class, you can simply add __and__ to it to do what you want: class iter: iter = iter # snapshot the default 'iter' def __init__(self, *args): self.iter = self.iter(*args) # break people's minds def __iter__(self): return self def __next__(self): return next(self.iter) def __and__(self, other): yield from self.iter yield from other Okay, so you'd probably do it without the naughty bits, but still :) As long as you call iter() on the first thing in the chain, everything else will work. ChrisA
The solution I'm currently using involves having a class called g, and everytime I want to manipulate an iterable, I just wrap it in g(). Then I got an object with all those semantics (and actually a lot more). Maybe we can make only those things apply to the objects returned by iter() ? (Also, about slicing accepting callable, actually g() goes a bit overboard and accept any object. If the object is not an int or callable, then it's used as a sentinel value. Not sure if I should speak about that here.) Le 22/03/2016 00:36, Chris Angelico a écrit :
On Tue, Mar 22, 2016 at 10:59 AM, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
Oh, that's easy then. iter = g :)
Nothing wrong with that. The semantics are still the same; you simply have several ways of defining "the end has been reached". ChrisA
On 2016-03-21 16:59, Michel Desmoulin wrote:
I'm not sure I see the value in new syntax, but I think what you describe here would be a useful addition to itertools. Yes, you would still have to import itertools, but you could just import this one "convenient iterator" class and use that, which could allow for concise but readable code that otherwise would have to use many different itertools functions. It would require some thought about what sorts of overloaded operators (if any) would be appropriate for various itertools functions, but I think it could still be a gain even if the main advantage was just being able to write stuff like "niceIterator(x)[10:20] + niceIterator(y)[20:30]". Since it would just be a convenience wrapper, there need be no worries about it not working for particular kinds of iterables (e.g., ones that already define their own behavior for certain operators); you just wouldn't use it for ones where it masked behavior you cared about on the underlying iterable. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Mar 21, 2016, at 16:06, Michel Desmoulin <desmoulinmichel@gmail.com> wrote: In addition to all the issues Chris raised...
The first issue is pretty minor compared to the later ones, but already shows the problems of thinking about only lists and iterators, so I won't skip it: Applying this to a sequence that copies when slicing makes sense. Applying it to an iterator (together with your #2) makes sense. Applying it to a type that returns views on slicing, like a NumPy array, doesn't necessarily make sense, especially f that type is mutable, like a NumPy array. (Should the view change if the first element > 4 changes?)
And what if it's a dict? Returning an islice of an iterator over the dict _works_, but it's almost certainly not what you want, because an iterator over the dict gives you the dict's keys, not its values. If d[3:7] means anything, I'd expect it to mean something like {k: v for (k, v) in d.items() if 3<=k<7}, or {v for ...same...}, not 4 arbitrary keys out of the dict. (Imagine that d's keys are all integers. Surely you'd want it to include d[3], d[4], d[5], and d[6], right?) And what if it's one of the collections on PyPI that already provides a non-islice-like meaning for slice syntax? For example, many of the sorted-dict types do key slicing, which returns you something like {k: v for (k, v) in d.items() if 3<=k<7} but still sorted, in log rather than linear time, and sometimes a view rather than a copy. And what if it's a collection for which indexing makes no sense, not even the wrong kind of sense, like a set? It'll slice out 4 values in arbitrary order iff there are at least 7 values. What's the good in that? And what if it's an iterator that isn't a generator? Does it just return some arbitrary new iterator type, even if the input type provided some additional interface on top of Iterator (as generators do)? Also, even with iterators a lot of things you do with slicing no longer make sense, but would run and silently do the wrong thing. For example: if matches_header(seq[:6]): handle_body(seq[6:]) If seq is an iterator, that first line is going to consume the first 6 elements, which means the second is now going to start on the 12th element rather than the 6th. It will be a lot less fun to debug "why do some of my messages lose their first word or so, but others work fine?" than the current "why do I sometimes get a TypeError telling me that type list_iterator isn't indexable?"
So, what does {1} & {1, 2, 3} do? This is a trick question: sets already define the & operator, as do other set-like collections. So this proposal either has to treat sets as not iterable, or break the set interface. All of these are part of the same problem: you're assuming that all iterables are either sequences or generators, but many of them--including two very important built-in types, not to mention all of the builtin types' iterators--are not.
Le 22/03/2016 01:45, Andrew Barnert a écrit :
Numpy is not part of the stdlib. We should not prevent adding a feature in Python because it will not immediately benefit an 3rd party lib, even a famous one. The proposal doesn't hurt Numpy : they override __getitem__ and choose not to accept the default behavior anyway, and are not affected by the default behavior. Besides, you generally try to not mix Numpy and non Numpy manipulation code as it has it's own semantics (no for loop, special slicing, etc.).
This is a point to discuss. I would raise a ValueError, trying to slice a dict is almost always a mistake. E.G: if you design a function that needs an argument to be sliceable (event with islice), you usually don't want people to pass in dicts, and when you strangely do (to sample maybe ?), then you would cast it manually. It would be a rare use case, compared to the multiple occasions you need a more generic slicing.
And what if it's one of the collections on PyPI that already provides a non-islice-like meaning for slice syntax? For example, many of the sorted-dict types do key slicing, which returns you something like {k: v for (k, v) in d.items() if 3<=k<7} but still sorted, in log rather than linear time, and sometimes a view rather than a copy.
See the point about Numpy.
And what if it's a collection for which indexing makes no sense, not even the wrong kind of sense, like a set? It'll slice out 4 values in arbitrary order iff there are at least 7 values. What's the good in that?
See the point about dict.
And what if it's an iterator that isn't a generator? Does it just return some arbitrary new iterator type, even if the input type provided some additional interface on top of Iterator (as generators do)?
This can be discussed and is more about the proper implementation, but does not discard the validity of the idea.
Either you use generators or you don't. If you use generators, you know they will be consumed when you pass them around. This has nothing to do with the slicing syntax. The one problem I can see is when: - seq is a generator you didn't produce; - you don't know it's a generator. - you get a surprising behavior because slicing cause no errors. It's an edge case and is worth considering if it's going to be a blocker or not. Also, one alternative is to only add slicing to all objects returned by iter() in the stdlib. This would force people to explicitly mark that they know what they are doing, and while less convenient, remains very handy.
Indeed I forgot about sets. Maybe there is another operator that would do the trick, such as "<<". We can't use "+" as it would be confusing, and sets() overide a LOT of operators. Anyway, let's not throw the baby with the water. This is the least important part of the proposal, and any part can be changed, improved or ditched. That's what Python-ideas is for.
All of these are part of the same problem: you're assuming that all iterables are either sequences or generators, but many of them--including two very important built-in types, not to mention all of the builtin types' iterators--are not.
"Slicing any iterable" was a bad title. It should have been "extend the slicing application". I'm not assuming any iterable is a sequence or a generator, I think we can come up with a reasonable behavior for the case slicing doesn't make sense, adding more power to this handy tool. Please also consider the iter() proposal as a more verbose alternative, yet still powerful alternative. Maybe even easier to implement ?
On 22 March 2016 at 16:19, Michael Selik <mike@selik.org> wrote:
And consider the other side of the coin. If I read code using a function of the itertools module, its easy to look up the definition; if I read code featuring a rarely used syntax, how easy is it to come up with search engine terms to describe the syntax, and thus find documentation for the feature?
On 3/22/2016 03:34, Graham Gower wrote:
This is a pure devil's advocate response, as I don't feel there is a need to add itertools functionality to the language itself. Googling for 'Python with keyword' or 'Python lambda keyword', for example, give very good results (documentation or examples in the first 5 results on both). Any proposal involving adding a new keyword would probably be as discoverable as the standard library. ... Which is the big gotcha with arguments involving new keywords. If it makes it exactly as discoverable, then nothing is gained by new syntax, and we are back where we started. Your point is accurate for any syntax that does not involve new keywords, or for new uses of existing keywords.
Le 22/03/2016 08:34, Graham Gower a écrit :
I see your point. Again, this is the weakest point of my proposal, we should not discard all of it just because of that. And maybe we can come up with something better. E.G: I suggested before that iter() returns objects with advanced semantics for [:] as an alternative to add slicing to generators. Maybe the same objects can come with a chain() method, which is explicit. Or provide an additional builtin function, which does help with those issues if changing iter() is not possible.
On Tue, Mar 22, 2016 at 11:58 PM, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
E.G: I suggested before that iter() returns objects with advanced semantics for [:] as an alternative to add slicing to generators.
The builtin iter() function simply returns whatever __iter__() returns. I've offered a way for you to easily shadow the builtin with your own class (which can then provide whatever it likes), but it's going to have to work by wrapping the original iterator, which will break generators (unless you explicitly pass everything through, and even then you'll break things that check if isinstance(x, generator), because you can't subclass generator). ChrisA
What if itertools.iter was added with the expanded semantics? Possibly even a new name, but similar use to the built-in func. Top-posted from my Windows Phone -----Original Message----- From: "Chris Angelico" <rosuav@gmail.com> Sent: 3/22/2016 6:06 Cc: "python-ideas" <python-ideas@python.org> Subject: Re: [Python-ideas] Integrate some itertools into the Python syntax On Tue, Mar 22, 2016 at 11:58 PM, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
E.G: I suggested before that iter() returns objects with advanced semantics for [:] as an alternative to add slicing to generators.
The builtin iter() function simply returns whatever __iter__() returns. I've offered a way for you to easily shadow the builtin with your own class (which can then provide whatever it likes), but it's going to have to work by wrapping the original iterator, which will break generators (unless you explicitly pass everything through, and even then you'll break things that check if isinstance(x, generator), because you can't subclass generator). ChrisA _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On 22 March 2016 at 16:01, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
Note that sys, os, re, math, datetime are all "an import in every file you want to use it". The bar for changing Python is higher than just avoiding an import. Paul
Le 22/03/2016 17:18, Paul Moore a écrit :
This is an appeal to consider islice & Co as important as normal slicing. Indeed, you will have most certainly iterables in any code that use datetime, re or math. You have no certainty of having datetime, re or math imported in any code dealing with iterables. Think about how annoying it would be to do:
For every slice. We don't, because slicing is part of our standard data processing toolkit. And I think we can make it even better. We already do in some places. E.G: range(10)[3:5] works while range() generate values on the fly. Why ? Because it's convenient, expressive and Pythonic. Well, if you process a file and you want to limit it to all lines after the first "BEGIN SECTION" (there can be other) and before the first "STOP" (there can be others), but only 10000 lines max, and not load the whole file in memory, you could do: def foo(p): with open(p) as f: def begin: return x == "BEGIN SECTION" def end: return x == "STOP" return f[begin, end][:10000] It's very clean, very convenient, very natural, and memory efficient. Now compare it with itertools: from itertools import takewhile, dropwhile, islice def foo(p): with open(p) as f: def begin: return x != "BEGIN SECTION" def end: return x != "STOP" return islice(takewhile(end, dropwhile(begin, f)), 0, 10000) It's ugly, hard to read, hard to write. In Python, you are always iterating on something, it makes sense to make sure we have the best tooling to do at our fingertips.
On 03/22/2016 10:51 AM, Michel Desmoulin wrote:
Except the 10,000 limit doesn't happen until /after/ the end block is reached -- which could be a million lines later.
Yes, it is. I would make a function: def take_between(begin=0, end=None, limit=None): """ return a list with all items between `begin` and `end`, but no more than `limit` begin -> an `int`, or function that returns `True` on first line to keep end -> None (for no end), an `int` relative to start of iterable, or a function that returns True on last item to keep limit -> max number of items to keep, or None for all items """
In Python, you are always iterating on something, it makes sense to make sure we have the best tooling to do at our fingertips.
In Python, you are always using data containers -- yet we don't have a plethora of different tree types built in. A simple import is "at our fingertips". On a more supportive note: Are you aware of `more_itertools`[1] ? If it doesn't already have `takeuntil` and `dropuntil` and your (pretty cool) `iter`, perhaps you could get your ideas included there. -- ~Ethan~ [1] https://pypi.python.org/pypi/more-itertools/
On Tue, Mar 22, 2016 at 8:41 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
if f[begin, end] is a generator, the 10000 limit may happen before the end block is reached, which I think was the point. Python 3 does give generators and iterators a more central role than Python 2. One thought that comes to mind is to add some stuff from itertools as attributes of iter. For example: iter.slice(...). Kind of like a factory method, although iter is not really a type. - Koos
Le 22/03/2016 20:23, Ethan Furman a écrit :
[begin, end] and [:10000] are applied one next() at a time. begin, then end, then :10000 for the first next(), then again in that order for the following next() call, etc. That's the whole point.
On 03/22/2016 12:26 PM, Michel Desmoulin wrote:
That may be the point, but that is not what the above code does. Since you don't believe me, let's break it down: f[begin:end] -> grabs a section of p. This could be 5 lines or 50000000 [:10000] -> take the first 10000 of the previous result return -> send those (up to) 10000 lines back -- ~Ethan~
No you didn't store the lines, the whole points is that they are returning generators. In that context, f[begin, end] does itertools.takewhile and dropwhile and [:10000] is doing islice(0, 10000). None of those functions store anything. Please read the begining of thread. Le 22/03/2016 20:41, Ethan Furman a écrit :
On 03/22/2016 12:43 PM, Michel Desmoulin wrote:
I presume you are referring to this line:
The slicing would then return a list if it's a list, a tuple if it's a tuple, and a islice(generator) if it's a generator.
Of which the big problem is that the object returned from an open call is not a list, not a tuple, and not a generator. For the sake of argument let's pretend you meant "and an islice(iter) for everything else". So the actual proposal is to add `__getitem__` to `object` (that way all classes that define their own `__getitem__` such as list and tuple are not affected), and the substance of this new `__getitem__` is to: - check for an `__iter__` method (if none, raise TypeError) - check the start, stop, step arguments to determine whether dropwhile, takewhile, or islice is needed - return the appropriate object Okay, it's an interesting proposal. I would suggest you make the changes, run the test suite, see if anything blows up. As a short-cut maybe you could add the code to the appropriate ABC and test that way. -- ~Ethan~
2016-03-22 20:43 GMT+01:00, Michel Desmoulin <desmoulinmichel@gmail.com>:
It would be true if f is generator. If f is list then we could probably want to have another operator to avoid memory consumption return list(f{begin,end}{:10000}) #return list(f{begin,end}[:10000]) # this could be same
If your point is that generators can sometimes be hard, then your point is clear :-). Maybe using them should not be made to look more straightforward than it actually is. Which brings me back to... One thought that comes to mind is to add some stuff from itertools as attributes of iter. For example: iter.slice(...). Kind of like a factory method, although iter is not really a type. - Koos
Le 22/03/2016 20:52, Koos Zevenhoven a écrit :
If your point is that generators can sometimes be hard, then your point is clear :-).
Maybe using them should not be made to look more
straightforward than it actually is.
I would have made the same mistake with itertools, let be frank. Which brings me back to...
It's an interesting though. Actually we could also make something like iter.wraps(iterable) to return and object with the properties discussed in that thread. It would be fully backward compatible, explicit, yet convenient enough.
On Mar 22, 2016, at 12:52, Koos Zevenhoven <k7hoven@gmail.com> wrote:
The one thing from itertools that I think people most miss out on by not knowing itertools is chain.from_iterable, with chain itself a close second. Would calling them iter.chain.from_iterable and iter.chain help? I'm not sure. Other people have proposed promoting one or both to builtins (maybe named flatten and chain, respectively), but that hasn't gone over very well. Making them "class methods" of iter seems a lot less disruptive. The other thing that I think might help is a way of writing "slice literals" more easily. Just having islice easier to find doesn't help much, because the main hurdle in writing and reading the code seems to be translating between the natural slice syntax and islice's (range-ish) unpythonic signature. Maybe islice(it, slice[:10000])? Or even islice(it)[:10000]?
On Tue, Mar 22, 2016 at 9:10 PM, Koos Zevenhoven <k7hoven@gmail.com> wrote:
I should add, though, that the the function should probably return list(f[begin, end][:10000]) or something, before the file f is closed. - Koos
Le 22/03/2016 20:26, Koos Zevenhoven a écrit :
Yeah the semantic of my function is wrong. It should rather expect a file like object as a parameter to avoid this bug. Plus, the begin() and end() functions are actually syntax errors ^^
On Tue, Mar 22, 2016 at 9:45 PM, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
Yeah, and perhaps be a generator function and yield from f[begin:end][:10000] instead of returning it. But beginners may end up trying this. It may be better to have them as functions (either in itertools or perhaps as attributes of iter), which may help them see what's actually going on. Or help them see that they actually don't know how it works and should just make a for loop for this to be more clear to them. - Koos
On Mar 22, 2016, at 10:51, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
Range is a sequence, just like list and tuple, not an iterator, like generator and list_iterator. So it supports slicing because sequences generally support slicing. Not because it's so convenient and expressive that it's worth making a special case, just because it follows the normal rules. It's amazing to me that so many people still describe xrange/3.x range as an iterator or generator, and then draw further incorrect lessons from that. Yes, the docs were a bit confusing up to about 2.4, but how long ago is that now? Lazy sequences are still sequences. Strict non-sequences (like dict) are still not sequences. Lazy vs. strict has almost[1] nothing to do with sequence vs. iterator (or sequence vs. non-sequence collection, or collection vs. iterator, or anything else.) --- [1]: Just "almost nothing" because you can't really write a strict generator--or, rather, you sort of can in a way, but it still has to look lazy to the user, so there's no point.
Le 22/03/2016 21:03, Andrew Barnert a écrit :
Yes, I worded my title with "all iterables", it was a mistake, I didn't think people would raise the debate on dict, set, etc because it was obvious to me that nobody would imagine I would try to make slicing work on those. I was wrong, but please don't make drown the debate into a battle of semantics. Ok, let's recenter to lazy sequences, and how convenient it would be to be able to limit them to a size or a condition without needing itertools. And in that context, the fact is that range(1000000) doesn't create 1000000 items in memory, and it's very convenient to be able to slice it. And it would would be as convenient to be able to do so on generators (which are sequences) instead of using islice.
On Mar 22, 2016, at 13:11, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
This is not just a battle of (meaningless) semantics. You seem to be completely misinterpreting the actual language semantics of Python, and therefore making proposals that don't make any sense. Case in point: you can _already_ limit lazy sequences to a size without needing itertools. That's exactly what you're doing in your range example. So we don't need to fix the language to make something possible when that something is already possible.
No, generators are _not_ sequences. And that's exactly the point you're missing. You have to understand the difference between iterators and other iterables to use Python effectively. That's just fundamental to so much of Python that there's no way around that. Maybe that was a language design mistake, but trying to muddy the waters further and confuse even more people is hardly a solution. Meanwhile, the range type points to a ready solution: when you want slicing (or other sequence behavior) with laziness, you can always write a lazy sequence type, usually in only a few lines of code. (I've got a couple blog posts on it, if you can't figure it out yourself.) And that solves the problem without introducing any confusion about what it should mean to slice an iterator, or any problems of the "if header(spam[:6]): process_body(spam[6:])" type.
On 3/22/2016 4:03 PM, Andrew Barnert via Python-ideas wrote:
An example I recently ran across is http://python4kids.brendanscott.com/2016/03/19/python-for-kids-python-3-proj... Python4Kids uses range-as-list for the 2.x version. For the 3.x version, he is agonizing over explaing 'range-as-generator'. -- Terry Jan Reedy
Le 22/03/2016 06:49, Michael Selik a écrit :
It's a gut feeling really, coming from the fact I train people in Python for a living: - most of my colleagues and myself always show the builtins way before anything else. I rarely have the time to even show itertools. - same goes for tutorials online. Itertools is rarerly demonstrated. - a whole module is more impressive than a few behavior and my student will usually delay reading about any doc related to one module. They don't even know where to start or how to express what they are looking for - it's something they will try (or discover by mistake): random playing with operators and data is something I witness a lot. Random playing with imports, not so much. I may be wrong, I can't finance a study with a big representative sample to prove it. But again, nobody can do so for anything on this list.
On Tue, Mar 22, 2016 at 8:52 AM Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
Luckily we have the case of set.union and set.intersection versus the pipe and ampersand operators. One way you could provide some evidence is comparing the frequency of usage of the method vs operator in public projects. We might need to then discard code examples written by experts from our analysis, but it'd be a start.
On 22 March 2016 at 09:06, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
iter() already has a two-argument form to accept a callable+sentinel value, so it may not be unreasonable to offer an "until" callback to say when to stop iterating. That is: def stop(element): return element >4 print(list(iter(numbers, until=stop))) Slicing arbitrary iterables may be amenable to a str.join style solution, by putting the functionality on slice objects, rather than on the iterables: slice(3, 7).iter(iterable) (Whether or not to make slice notation usable outside subscript operations could then be tackled as an independent question) For itertools.chain, it may make sense to simply promote it to the builtins. I'm not the least bit sure about the wisdom of the first two ideas, but the last one seems straightforward enough, and I'd be comfortable with us putting "chain" on the same tier as existing builtin iteration related tools like enumerate and zip. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le 23/03/2016 06:13, Nick Coghlan a écrit :
Do you propose to make a "since" param too ?
Those solutions are already much more convenient than using itertools, but it's going to look very strange: def foo(p): with open(p) as f: def begin(): return x.startswith('#') def end(): return x != "STOP" yield from slice(0, 10).iter(iter(f, since=begin, until=en)) The last line is really not Pythonic. Compared to: def foo(p): with open(p) as f: def begin(): return x.startswith('#') def end(): return x != "STOP" yield from f[begin:end][:10] But I beleive there is an idea here. Attaching things to slice is cleaner than to iter(): def foo(p): with open(p) as f: def begin(): return x.startswith('#') def end(): return x != "STOP" yield from slice.wraps(f)[begin:end][:10]
Same problem as with new keywords : it can be a problem with people using chain as a var name.
To be fair, I don't nearly use chain as much as zip or enumerate, so I'm not going to push for something as drastic.
Regards, Nick.
On Wed, Mar 23, 2016 at 9:04 PM, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
Less of a problem though - it'd only be an issue for people who (a) use chain as a var name, and (b) want to use the new shorthand. Their code will continue to work identically with itertools.chain (or not using it at all). With a new keyword, their code would instantly fail. ChrisA
I enjoy using ``chain`` as a variable name. It has many meanings outside of iteration tools. Three cheers for namespaces. Many of Python 3’s changes versus Python 2 were to take the pure stance -- change special statement syntax to function calls, move a few builtins into appropriate modules. If chain were a builtin, we’d have someone suggesting it move into itertools.
On Mar 23, 2016, at 10:13, Michael Selik <mike@selik.org> wrote:
As Chris just explained in the message you're replying to, this wouldn't affect you. I've used "vars" and "compile" and "dir" as local variables many times; as long as I don't need to call the builtin in the same scope, it's not a problem. The same would be true if you keep using "chain". Unless you want to chain iterables of your chains together, it'll never arise. Also, putting things in builtins isn't _always_ bad. It's just a high standard that has to be met. I don't think anyone believes any, all, and enumerate fail to meet that standard. So the question is just whether chain meets it. My problem is that I'm not sure chain really does meet it. It's chain.from_iterable that I often see people reaching for and not finding, and moving chain to builtins won't help those people find it. (This is compounded by the fact that even a novice can figure out how to do chain given chain.from_iterable, but not the other way around.) Also, for something we expect novices to start using as soon as they discover iterators, it seems worrisome that we'd be expecting them to understand the idea of a static method on something that looks like a function but is actually a type before they can have a clue of what it means. In another thread a year or two ago, someone suggested making chain.from_iterable into a builtin with a different name, maybe "flatten". But that now means we're adding _two_ builtins rather than one, and they're very closely related but don't appear so in the docs, which obviously increases the negatives... Still, I like adding chain (and/or flatten) to builtins a lot more than I like adding sequence behavior to some iterators, or adding a whole new kind of function-like slicing syntax to all iterables, or any of the other proposals here.
On Wed, Mar 23, 2016 at 1:39 PM Andrew Barnert <abarnert@yahoo.com> wrote:
Depends what you mean by "affect". It'll affect how I read my colleagues' code. I want to see ``from itertools import chain`` at the top of their modules if they're using chain in the sense of chaining iterables.
Part of why ``any`` and ``all`` succeed as builtins is their natural language meanings are quite close to their Python meaning. ``enumerate`` succeeds because it is unusual in natural language. For many people, Python might be the most frequent context for using that word. Some evidence that ``chain`` should be promoted to builtin would be that ``chain`` is used more often than the rest of the itertools library and that the word "chain" is rarely used except as the itertools chain. Luckily one could search some public projects on GitHub to provide that evidence. If no one bothers to do so, I'm guessing the desire isn't so great.
I like the LazyList you wrote. If the need for operator-style slicing on iterators were great, I think we'd find a decent amount of usage of such a wrapper. Its absence is evidence that ``from itertools import islice`` is more pleasant. As has been discussed many times, lazy sequences like range cause some confusion by providing both generator behavior and getitem/slicing. Perhaps it's better to keep the two varieties of iterables, lazy and non-lazy, more firmly separated by having ``lst[a:b]`` used exclusively for true sequences and ``islice(g, a, b)`` for generators. Just yesterday I got frustrated by the difference of return types from subscripting bytes versus str while trying to keep my code 2/3 compatible: ``bytestring[0]`` gives an integer while ``textstring[0]`` gives a 1-length str. I had to resort to the awkwardness of ``s[0:1]`` not knowing whether I'll have a Python 3 bytes or Python 2 str. I prefer imports to inconsistency. A couple questions to help clarify the situation: 1. Do you prefer ``a.union(b)`` or ``a | b``? 2. Do you prefer ``datetime.now() + timedelta(days=5)`` or ``5.days.from_now``? I think the answer to #2 is clear, we prefer Python to Ruby (more specifically the Rails sub-community). The answer to #1 is more difficult. I'm often tempted to say ``a | b`` for its elegance, but I keep coming back to ``a.union(b)`` as clunky but readable, easy to explain, etc.
The advantage of having a small set of builtins is that you know the entire set of builtins. If chain really is useful enough that it belongs as a builtin, you will very quickly adapt to reading that code, and it won't bother you or slow down your comprehension at all, any more than any other builtins do. Adding dozens of builtins would break that; adding one wouldn't. There's only room for a handful more builtins in the entire future life of Python, and the question of whether chain deserves to be one of them is a big question (and I suspect the answer is no). But I think it's the only real question, and adding more on top of it doesn't really help us get to the answer.
Some evidence...
Agreed. And again, my own anecdotal experience is that chain.from_iterable is actually used (or sadly missed) more than chain itself, especially among novices, so I'm not advocating making chain a builtin unless someone proves me wrong on that.
Still, I like adding chain (and/or flatten) to builtins a lot more than I like adding sequence behavior to some iterators, or adding a whole new kind of function-like slicing syntax to all iterables, or any of the other proposals here.
I like the LazyList you wrote. If the need for operator-style slicing on iterators were great, I think we'd find a decent amount of usage of such a wrapper. Its absence is evidence that ``from itertools import islice`` is more pleasant.
The one on my blog? I've never found a use for that in real code[0], hence why (IIRC) I never even put it on PyPI. But I use range all the time, and various other lazy sequences. The problem with LazyList isn't that it's lazy, but that it's a recursive cons-like structure, which is ultimately a different way to solve (mostly) the same set of problems that Python already has one obvious solution for (iterators), and an ill-fitting one at that (given that Python discourages unnecessary recursive algorithms). I've also written a more Python-style lazy list that wraps up caching an iterator in a list-like object. It's basically just a list and an iterator, along with a method to self.lst.extend(islice(self.it, index - len(self.lst)). The point of that is to show how easy it is to write, but how many different API decisions come up that could be reasonably resolved either way, so there's probably no one-size-fits-all design. Which implies that if there are apps that need something like that, they probably write it themselves. Anyway, if your point is that iterators having slicing would be a bad thing, I agree with you. But that doesn't necessarily mean that chain and islice as they are today is the best possible answer, just that it's better than adding operators to the iterator type (even if there were such a thing as "the iterator type", which there isn't). Using islice can still be clumsy, and I'm happy to see what alternatives or improvements people come up with.
As has been discussed many times, lazy sequences like range cause some confusion by providing both generator behavior and getitem/slicing. Perhaps it's better to keep the two varieties of iterables, lazy and non-lazy, more firmly separated by having ``lst[a:b]`` used exclusively for true sequences and ``islice(g, a, b)`` for generators.
Definitely not. Even if breaking range and friends weren't a huge backward compat issue, it would weaken the language. And, meanwhile, you would gain absolutely nothing. You'd still have sets and dicts and third-party sorted trees and list iterators and itertools iterators and key views and so on, none of which are either sequences or generators. (Also, think about this parallel: do you want to say that dict key views shouldn't have set operators because they're lazy and therefore not "real sets"? If not, what's the difference?) The problem isn't that Python has things that are neither sequences nor generators, it's that the Python community has people who think it doesn't. Any time someone says range is (like) a generator, or any similar misleading myth, they need to be corrected with extreme prejudice before they set another batch of novices up for confusion. So, again: range does not provide anything like "generator behavior". It's not only repeatably iterable, it's also randomly-accessible, container-testable, reversible, and all the other things that are true of sequences.
Just yesterday I got frustrated by the difference of return types from subscripting bytes versus str while trying to keep my code 2/3 compatible:
That's a completely separate problem. I think everyone agrees that there are design mistakes in the bytes class. As of 3.5, most of the ones that can be fixed have been, but some of them we're just unfortunately stuck with. None of those problems have to do with the iteration or sequence protocols.
A couple questions to help clarify the situation: 1. Do you prefer ``a.union(b)`` or ``a | b``?
When I'm doing stuff that's inherently mathematical-set-like, I use the operator. When I'm using sets purely as an optimization, I use the method.[1] But how does that relate to this thread? I think the implied assumption in the proposal is that people want to do "inherently sequence-like stuff" with iterators; if so, they _should_ want to spell it with brackets. I think the problem is that they're wrong to want that, not that they're trying to spell it wrong.
2. Do you prefer ``datetime.now() + timedelta(days=5)`` or ``5.days.from_now``?
Of course the former.[2] But the problem with "5.days" is nothing to do with this discussion. It's that it implies a bizarre ontology where natural numbers are things that know about timespans as intimately as they know about counting,[3] which is ridiculous.[4] If you really wanted numbers to know how to do "inherently day-like stuff", the Ruby way would make sense; you just shouldn't want that. --- [0]: I have found uses for it in comparing translated Lisp/ML/Haskell idioms to native Python ones, or in demonstrating things to other people, which is why I wrote it in the first place. [1]: Or sometimes even the classmethod set.union(a, b). In fact, sometimes I even write it out more explicitly--e.g., {x for x in a if x in b} in place of a & b emphasizes that I could easily change the {...} to [...] if I need to preserve order or duplicates in a. [2]: I like now() + days(5), or maybe even now() + 5*days, even more. But that's easy to add myself without modifying either int or timedelta, and without confusing my readers. [3]: Or, only slightly less accurately, it's like saying "5's days" instead of "5 days" in English. [4]: The Ruby answer to that is that writing an application is really about writing an app-specific language that lets you write the app itself a trivial single-page program, and in an OO context, that means extending types in app-specific ways. The number 5 in general doesn't know how to construct a timespan, but the number 5 in the context of a calendaring web application does. That's an arguably reasonable stance (Paul Graham has some great blog posts making the non-OO version of the same argument for Lisp/Arc), but it's blatantly obvious that it's directly opposed to the Python stance that the language should be simple enough that we can all immediately read each other's code. Python doesn't encourage indefinitely expanding core types for the same reason it doesn't encourage writing your own flow control statements.
On Thu, Mar 24, 2016 at 8:46 AM, Andrew Barnert <abarnert@yahoo.com> wrote:
The advantage of having a small set of builtins is that you know the entire set of builtins.
Python 3.6 has len(dir(builtins))==150, which is hardly small. Even removing the ones that don't begin with a lower-case letter (which removes a small number of special names like __package__, and a large number of exception classes) leaves 72. Can you name them all? ChrisA
On Mar 23, 2016, at 15:01, Chris Angelico <rosuav@gmail.com> wrote:
Can you name all 25000 words of English that you know? There are 68 builtins listed in a nice table at the top of the builtin functions chapter in the docs. Take a look at that table. They're all (except maybe format and iter[1]) functions[2] that you can immediately recognize them when reading them, and recall when they're relevant to code you're writing, including know exactly what their interface is and what they do. If you see "any()" or "int()" in code, you don't need to turn to the docs to look up what it does. If you need to convert something to a debug string representation, you know to call "repr()". The fact that you might leave out "any", "int", or "repr" in a pop quiz demanding that you list all 68 of them in alphabetical order doesn't mean you don't know them. You certainly can't say the same is true for all functions and methods in the stdlib. --- [1]: I think everyone knows iter _exists_, but not everyone knows about its two-argument form. [2]: ... or types...
On Thu, Mar 24, 2016 at 9:38 AM, Andrew Barnert <abarnert@yahoo.com> wrote:
Fair enough, naming them all is a poor definition of "know". But the flip side (knowing what one means when you meet it) is also a poor definition. The real trick is knowing, when you're faced with a problem, what the solution is; and while most people here will be able to do that, I'm not sure that every Python programmer can. Some builtins are well known, like int and len; others, not so much - zip, hash, divmod. And then there are some (compile, bytearray, callable) that, if they were elsewhere in the stdlib, would barely change people's code. How many non-experts know exactly which function decorators are builtins and which are imported from functools? If you have to look them up, (a) there's not a lot of difference between built-in and stdlib, and (b) the builtins are already a bit crowded. My conclusion: itertools.chain is just as discoverable as builtins.chain, so the only real benefit would be playing around at the interactive prompt, which you can improve on with an auto script. -0.5 on making chain a builtin. ChrisA
On 03/23/2016 04:17 PM, Michel Desmoulin wrote:
Le 24/03/2016 00:15, Chris Angelico a écrit :
You know, that's a good point! It just has one problem: I use datetime.datetime and datetime.date a heck of a lot more than chain and chain.from_iterable; and os.getuid and os.setuid! and sys.argv, and collections.OrderedDict, and sys.version_info, and sys.modules, and most of the things in logging... So all those things should be builtin first. Then we can talk about chain. Chaining-all-my-favorite-imports-ly yrs, -- ~Ethan~
On Mar 23, 2016, at 16:00, Chris Angelico <rosuav@gmail.com> wrote:
Well, readability is at least as important as writability. But yes, writability is also important, which is why I covered it: 'If you need to convert something to a debug string representation, you know to call "repr()".'
and while most people here will be able to do that, I'm not sure that every Python programmer can.
Sure, there's a certain level of competence below which you can't say people know all the builtins. There's also a certain level of competence below which you can't say people know every kind of statement in the syntax. But the point is that the level is low enough to be reasonably achievable. It's not only the hardcore experts that know almost all of the builtins. If we tripled the number of builtins, I don't think that would be true anymore. (Secondarily, the alphabetical chart in the docs is good enough at 68; at 200, we'd almost certainly need to reorganize it to make it useful.)
I don't think anyone would be horribly bothered by moving compile outside of builtins, but it has to be there for bootstrapping reasons. (Maybe that isn't true anymore, but it apparently was in 3.0.) For callable, that actually _was_ removed from builtins in 3.0, and added back pretty late in the 3.2 cycle because, as it turns out, people really did miss it more than you'd expect a priori. But anyway, the presence of 2 or 3 or even 10 functions out of those 68 that maybe shouldn't be there isn't license to add 100 more. If anything, it means we have to be a tiny bit more careful what we add there in the future.
I don't think it's _just_ as discoverable. But I do think it's discoverable _enough_. (Plus, as I already mentioned, I think chain.from_iterable is more of a problem than chain, and moving chain to builtins wouldn't help that problem enough to be worth doing.) So I'm also -0.5.
On Mon, Mar 21, 2016 at 8:06 PM, Michel Desmoulin <desmoulinmichel@gmail.com
wrote:
I like how dropwhile and takewhile could be easily integrated with list comprehensions / generator expressions: [x for x in range(10) while x < 5] # takewhile [x for x in range(10) not while x < 5] # dropwhile [x for x in range(10) from x >= 5] # forward thinking dropwhile I believe this is almost plain english without creating new keywords. Regards
On Wed, Mar 23, 2016, 5:27 PM João Bernardo <jbvsmo@gmail.com> wrote:
They read well, except for the square brackets which to me imply consuming the entire iterator. Itertools takewhile will early exit, possibly leaving some values on the iterator. If these do consume the entire iterator, what's the difference with the ``if`` clause in a comprehension?
On Wed, Mar 23, 2016, 5:55 PM Sven R. Kunze <srkunze@mail.de> wrote:
After the execution of the list comprehension using "while" if the condition triggered halfway, would there be anything left in the original iterator? If it was a generator expression, we'd say "Yes" immediately. As a list comprehension, I'm not sure.
João Bernardo On Wed, Mar 23, 2016 at 7:18 PM, Michael Selik <mike@selik.org> wrote:
Dropwhile alone would consume the entire iterator for obvious reasons. Takewhile will stop when necessary, otherwise why implement it? You could nest them and still be readable: [x for x in range(10) from x > 3 while x < 7] Still quite readable. If this were an iterator like "foo = iter(range(10))", there will be items left in the end.
On 3/21/2016 9:29 AM, Steven D'Aprano wrote:
Why is the truth a mistake?
It's not the truth.
Yes it is. When the iterator is empty, possibly after many iterations, and the iterator is tested, the else clause runs.
The else block does *not* only run if the for block doesn't run.
I have never, ever, claimed that. Initially empty and empty after all items have been processed are different.
It becomes a while loop when 'go to if-test' is added after 'a'.
Each time next(iterable) is called by the for loop machinery, executing b is the alternative to executing a. You and I view the overall looping process differently. The way I see it, as repeating if-else whenever if is true, makes 'else' perfectly sensible. You see it differently, in a way that makes 'else' less sensible. Here is an alternative way to construct 'while condition' from if-else. Imagine that we only have a loop-forever primative (implemented internally as an unconditional jump). Python currently spells this as 'while True'. Then 'while condition' could written as the following (which works today). while True: if condition: true_part() else: false_part() break The false_part is the alternative to the true_part. -- Terry Jan Reedy
On 3/22/2016 3:34 PM, Ethan Furman wrote:
On 03/22/2016 11:49 AM, Terry Reedy wrote:
from the perspective of a particular loop.
but for/else and while/else treat them the same.
Which is to say, each test is a test of current state, independent of history. It does not matter if the iterater was created empty, was emptied by previous code out or inside the current function, or emptied by the current loop. People who want to condition on history should record it. The idiom presented previously is a form of recording history. item = marker = object() # could be earlier in function body for item in iterable: ... else: if item = marker: # started loop empty ... -- Terry Jan Reedy
On 03/23/2016 03:25 PM, Terry Reedy wrote:
On 3/22/2016 3:34 PM, Ethan Furman wrote:
From the perspective of many, many (most?) people. I use for loops and while loops *alot*, and I can count the times I have wanted to use the `else` clause in the meaning of "all my tests failed, so do this block" on the fingers of one hand.
By the way, I appreciate your explanations -- they help make a little more sense out of it -- but the behaviour of `for/else` and `while/else` is definitely a learned exception to the general rule of "truthy -> this thing is something" / "falsey -> there is nothing here". -- ~Ethan~
I"ve only used for .. else a handful of times, and honestly, probably forgot to use it when it would have been the right way to do something far more times :-) but I can't think of a SINGLE time when I wanted to know that the loop didn't run because it's iterable was empty. Which is my way of saying: I think adding something would make this use case more clear and easy to do, but it's a rare enough use case that it's not worth it. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Thu, Mar 24, 2016 at 5:34 PM Chris Barker <chris.barker@noaa.gov> wrote:
More specifically, a time when the loop didn't run because the iterable was empty AND the iterable didn't have a len to make that a simple check. Which suddenly gives me an idea... class Peeking: def __init__(self, iterator): self.it = iter(iterator) self.sentinel = object() self.buf = next(self.it, self.sentinel) def __iter__(self): return self def __next__(self): if self.buf is self.sentinel: raise StopIteration value = self.buf self.buf = next(self.it, self.sentinel) return value def __bool__(self): return self.buf is not self.sentinel If you find yourself in a situation where you want to check empty as if your iterator was a sequence (ie. len(seq) == 0 means False), give this wrapper a shot just before you do the loop. >>> bool(Peeking(range(0))) False >>> bool(Peeking(range(1))) True We could even make it a context manager to monkey-patch a buffering inside the context for the bool check, then undo the patch on leaving the context to stop buffering.
On 03/24/2016 03:14 PM, Michael Selik wrote:
You mean like this answer? http://stackoverflow.com/a/7976272/208880 -- ~Ethan~
On Tue, Mar 22, 2016 at 02:49:44PM -0400, Terry Reedy wrote:
I just realised I typo-ed that. I meant "iterable", not iterator. For-loops can iterate over any iterable (sequences, sets, dicts etc) not just iterators (as I'm sure you know). I may have inadvertently given you the wrong impression, in which case, I apologise.
I can only guess that you thought I was referring to the invisible, implementation-dependent iterator which is used internally by the for-loop. (See byte code below for what I mean.) That's fine, but it isn't what the topic of this discussion is about. It's irrelevent. Nobody should care about whether *that* iterator is exhausted or not, because that iterator is unreachable from Python code. What we might care about is the iterable "seq" used in (for example) "for x in seq". Sometimes people want to run a block of code *only* if the body of the for loop never runs, and some people mistakenly thought, or think, that the "else" clause does that. I know this for a fact because I was one of those people, and I have seen others make the same mistake. I can prove it is a mistake by running this code: seq = [2, 4] for x in seq: print("seq is not empty") # this is correct else: print("seq is empty") # this is wrong assert list(seq) == [] The assertion won't *always* fail, e.g. try passing "seq = []") but the fact that it *sometimes* fails proves that it is a mistake to interpret the "else" clause as "run only when the for loop doesn't run". Take this for-loop: def f(): for x in seq: print(x) else: print(1) In Python 2.7, dis.dis(f) gives: 2 0 SETUP_LOOP 24 (to 27) 3 LOAD_GLOBAL 0 (seq) 6 GET_ITER >> 7 FOR_ITER 11 (to 21) 10 STORE_FAST 0 (x) 3 13 LOAD_FAST 0 (x) 16 PRINT_ITEM 17 PRINT_NEWLINE 18 JUMP_ABSOLUTE 7 >> 21 POP_BLOCK 5 22 LOAD_CONST 1 (1) 25 PRINT_ITEM 26 PRINT_NEWLINE >> 27 LOAD_CONST 0 (None) 30 RETURN_VALUE My guess is that you thought I was referring to the iterator which is created by the call to GET_ITER, and iterated over by FOR_ITER. Mea culpa -- I was not. I meant to say iterable, not iterator, and in this example I would mean "seq". If you look at the byte code, you will see that there is no test for whether this internal iterator, or seq for that matter, is empty. When the for loop, the code unconditionally executes the else block. If the for loop contains a break (or a return, or a raise) execution may transfer to outside of the for...else altogether, skipping the else block, but there is still no test for emptiness. It's an unconditional jump: if you reach the break, you jump past the end of the for...else.
Then I hope that we now are on the same page, and agree that: (1) "else" does NOT mean "execute this block only if the for block doesn't run", even if some people mistakenly did, or do, think so. (2) Some people want such a statement, and that's the topic for this discussion. (3) As for...else now exists, there is no testing whether or not the for-loop ITERABLE is empty, or whether the internal (inaccessible) ITERATOR is empty; furthermore, other implementations or versions may not even use an internal ITERATOR (e.g. Python 1.5 certainly doesn't). (4) "break" doesn't set a flag, it just jumps to the outside of for...else (although in principle this is implementation dependent, there is nothing in the semantics of for...else that *requires* a flag be set), or the iterable be tested for emptiness. -- Steve
2016-03-21 2:32 GMT+01:00, Steven D'Aprano <steve@pearwood.info>: [...]
A. Just a little analyze about this theoretical possibility for x in seq: if good(x):break # (1) then: all_bad() # (2) else: empty() # (3) (3) seems to be alternative to (2) like "if then else" , but is not , it is alternative to (1) "union" (2) Also empty is not contradictory to all_bad! 'else' seems to be not ideal but yes we have what we have B. if we don't want to define new keywords and want to map all (?) possibilities (empty vs non empty, break vs no break) and don't want to break backward compatibility then we could analyze this: for i in seq: if cond(i): print('non empty & break') # (1) break and: print('non empty & all') # (2) or: print('no elements') # (3) else: print('no break') # (4) (1) is exclusive but (4) could be with (2) or<exclusive> (3) (2) and (3) are contradictory I have not opinion about order of and, or, else in this moment.
On Sun, Mar 20, 2016 at 07:12:58PM +0100, Sven R. Kunze wrote:
"empty" is a poor choice, as I expect that it will break a lot of code that already uses it as a variable. if empty: ...
I'm not too keen on the look of "or" for this: for x in iterable: block or: alternative but I think it's the least worst of the proposed keywords.
4) except -> as alternative if a new keyword is too much
I think that "except" completely fails to describe what the keyword does, and it will give some people the misunderstanding that it catches errors in the body of the for-loop. -- Steve
On 21.03.2016 02:44, Steven D'Aprano wrote:
So, "empty" is off the table I suppose.
Got it but I am open for suggestions. :-)
Now, that you mention it. This might give birth to a completely different idea. What about? for item in collection: # do for item except EmptyCollection: # do if collection is empty except StopIteration: # do after the loop Which basically merges try and for? Best, Sven
On Tue, Mar 22, 2016 at 01:20:05PM +0100, Sven R. Kunze wrote:
From your statement 'merges try and for', it would seem as you now want
What worries me about this is that the next question is going to be: Why not have this on `with` suites, `if` suites, and so on. I've already seen remarks symmilar to that as replies. Especially because of your remark 'Which basically merges try and for', which adds to my worries. Consider the following: for item in [0, 1, 2]: raise StopIteration except StopIteration: # do after the loop # Also after a `raise` inside the loop body? the exception to be caught in this case as well. Now, maybe you did not intend it as such, and really intended to only catch the exceptions raised by the iterable itself. But that was not quite clear to begin with. Because of this (possible) confusion, I'd really suggest slapping on another keyword to the `for` statement over allowing `except` clauses with limited use. Another reason is that for item in [0, 1, 2]: # do something except StopIteration: pass would do exactly the same as the `for` loop without the `except` clause, (due to the `pass`). Now this does not break the principle of 'one obvious way to do it', because for Pythonistas with experience, the `except` is superfluous. But would a novice know? (Assuming he started not by reading the Python docs, but by working on code written by another programmer)? Yes, RTFM, but more often you learn by reading other peoples code.
On 24.03.2016 07:54, Sjoerd Job Postmus wrote:
My initial motivation is to support the "empty" use-case. I am not attached to any proposal as long as that is achieved. For one reason, I dislike to explain how to emulate the missing feature to others (especially when Django and jinja2 supports it) and for another I found usage for it for myself recently. [offtopic] I am curious now. What is the problem with providing exception handling to more than just try? Is it just because it complicates the language which you are used to? Or is there something really problematic I miss here?
Especially because of your remark 'Which basically merges try and for', which adds to my worries.
This was to make people understand it more easily. :-)
That's interesting. Yes, that could be confusing. But what can we do? I cannot believe that we run out of ideas just for something that some simpler languages can handle easily. :-/
Noted but I wouldn't give too much weight to this issue. As others noted "else" is equally confusing. Best, Sven
On 20/03/2016 18:12, Sven R. Kunze wrote:
I don't do hypothetical. Practicality beats purity. So having followed this thread I do not believe that language changes are needed. So why not add a paragraph to this https://docs.python.org/3/tutorial/controlflow.html#break-and-continue-state... giving the cleanest solution from this thread for the way to write this Pythonically. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence
Am 20.03.2016 um 19:12 schrieb Sven R. Kunze:
Yes, this looks good. What about this? {{{ for item in my_iterator: # do per item on empty: # this code gets executed if iterator was empty on break: # this code gets executed if the iteration was left by a "break" on notempty: # ... }}} Is there a case which I have forgotten? ... I have no clue how to implement this :-) Regards, Thomas -- Thomas Guettler http://www.thomas-guettler.de/
On 24.03.2016 14:54, Thomas Güttler wrote:
Hmm, interesting. "on" would indeed be a distinguishing keyword to "except". So, "except" handles exceptions and "on" handles internal control flow (the iter protocol). Nice idea actually.
}}}
Is there a case which I have forgotten?
"on notbreak" I suppose. I think the downside here is that we would need to register another keyword "on". Is this used as a normal variable often? Not sure if "on empty" can be registered as a single keyword since it contains whitespace. Best, Sven
Am 24.03.2016 um 18:44 schrieb Sven R. Kunze:
I thank you, because you had the idea to extend the loop syntax :-) But I guess it is impossible to get "on empty" implemented. Regards, Thomas Güttler -- Thomas Guettler http://www.thomas-guettler.de/
What about:
The empty marker would be set to True but the for loop and set to False when iterating on the loop. The perf cost of this is only added if you explicitly requires it with the "as" keyword. You don't handle break, which is handle by else anyway. And maybe having access to the iterator could open the door to more features. Le 06/04/2016 16:16, Thomas Güttler a écrit :
On Mar 20, 2016, at 11:12, Sven R. Kunze <srkunze@mail.de> wrote:
Issues People I talked to suggested "else" as an alternative to "empty" because "empty is not quite clear". Unfortunately, "else" is already taken and is itself not quite clear. "then" has been proposed as an alternative for "else" in an attempt for proper understanding of "else". If that would be an accepted proposal, it would make room for "else" to be used for the usage of the "empty keyword proposed here.
Besides the backward compatibility issue, changing "else" to "then" would be horribly confusing. I suspect anyone who thinks that would be an improvement doesn't actually understand or like for...else, and they'd be happier just eliminating it, not renaming it. An else clause is testing that no break was hit inside the loop. Look at a typical example: for elem in seq: if good(elem): break else: raise ValueError The word "then" would make no sense there. "Find the first good element, else raise an exception" makes sense; "find the first good element, then raise an exception" means the opposite of what we want. Anyway, none of this speaks for or against your main proposal, it just means (at least in my opinion) that this alternative option is off the table. More generally, I think if this feature were to be added, "empty" is a reasonable name. The idiomatic way to write it today is something like: elem = empty = object() for elem in seq: do_stuff(elem) if elem is empty: do_empty_seq_stuff() So you're basically looking for syntactic sugar that abbreviated "if ... is empty:", right?
Robert Collins wrote:
oh, ok. sorry, didn't think about the iterator object. ...thinking about it, is there a reason why the boolean interpretation of an empty iterator is true? I know it's not trivial to know if an iterator is empty (still, there are a pair of recipes online, like http://code.activestate.com/recipes/413614-testing-for-an-empty-iterator/ or the first part of the response at http://stackoverflow.com/a/3114423/273593 ) but shoulnd't be more "pythonic" to just have a "falsy" value for an "empty" iterator? -- By ZeD
Not trivial, indeed. Consider the following: def gen(): while random.choice([0, 1]): yield "spam" Is it empty? Is it not? You can't tell when it will be, and while that is a bad example (I hope something like that doesn't get used!), arbitrary generators can have side effects, or be empty under certain circumstances and not under others. That being said, we can maybe make the built-in iterators return a falsey value in such cases, and let third-party iterators handle that on their own, if they feel like doing that. Generators would always be true, but it's an opt-in solution, and custom iterators can already define their own __bool__ if they want. -Emanuel ~ Anything can be a duck if you try hard enough ~
Émanuel Barry writes:
From Vito De Tullio
AFAICS the OP's idea has a trivial and efficient expansion: for item in iterable: # implements __next__ # do for each item empty: # do exactly when iterable raises StopIteration on the first pass becomes item = _sentinel = object() for item in iterable: # do for each item if item is _sentinel: # do exactly when iterable raises StopIteration on the first pass (works in light testing but maybe I've missed something). But to me that means saving one line and a few characters in the trailer clause is not worth syntax.
Is it empty? Is it not? You can't tell when it will be, and while that is a bad example (I hope something like that doesn't get used!),
Replace random with polling an input source or accessing a database, and you get the same nondeterminism. Steve
On Mon, Mar 21, 2016 at 7:01 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
What if 'iterable' is locals().values()? Can you, with perfect reliability, recognize that case? AIUI this is exactly why next() and __next__() are defined to "return a value or raise", rather than "return a value or return a magic no-more-values value", because there's always the possibility that the no-more-values value is a legitimately-yielded value. Maybe this situation isn't important enough or common enough to justify dedicated syntax, but it's definitely a possibility. ChrisA
Chris Angelico wrote:
so, the "correct" way to handle this is something like try: e = next(iterable) except StopIteration: empty_suite() # there is no element else: main_suite(e) # handle the first element for e in iterable: main_suite(e) # handle the rest of the elements ? apart for the readability you need to "copy" the same code two times (or refactor in a function) -- By ZeD
On 21.03.2016 21:57, Vito De Tullio wrote:
Interesting. I didn't think of this solution. However, I assume that when I do "object()" I got something unique among all other objects. So, item = sentinel = object() for item in collection: main_suite(item) if item is sentinel: empty_suite() Is still quite correct, right? Best, Sven
On Tue, Mar 22, 2016 at 11:07 PM, Sven R. Kunze <srkunze@mail.de> wrote:
Sure it is. Perfectly correct.
An iterator can return *any* *object*. That's why Python uses an exception (StopIteration) to signal the absence of an object. More reliable is to use that absence, either by exception or by probing a dictionary's keys: def better(): # Make sure item is unbound try: del item except NameError: pass for item in locals().values(): print("We have:", item) if 'item' not in locals(): print("We have no items.") But now we're getting into the realm of ugly code to deal with edge cases. Like with "yield from", language support can be justified when there's a simple and obvious *but imperfect* way to do something ("yield from x" is not the same as "for item in x: yield item"). ChrisA
On 22.03.2016 13:16, Chris Angelico wrote:
Oh, you are right. That's definitely an ugly edge case. Any proposed solution should just work given the simplicity of the problem.
Interesting.
Exactly. Despite the fact that "yield from" is shorter it can handle edge cases more beautiful. Best, Sven
On Tue, Mar 22, 2016 at 11:27 PM, Sven R. Kunze <srkunze@mail.de> wrote:
It's not enough shorter than the simple 'for' loop to justify the dedicated syntax. But check out its *actual* equivalent code and you'll see that while we're getting toward similar territory, the for-ifempty loop isn't quite here yet :) https://www.python.org/dev/peps/pep-0380/#formal-semantics ChrisA
On 22.03.2016 13:29, Chris Angelico wrote:
I remember that. I was quite surprise to find such a huge amount of equivalent python code to handle all corner and edge cases correctly. If I were to become the one maintaining this, I hope we don't discover even more edge cases. :D Best, Sven
On Tue, Mar 22, 2016 at 01:27:05PM +0100, Sven R. Kunze wrote: [...]
I think we're starting to give far more weight to that case than it deserves. It doesn't deserve a note in the docs, let alone new syntax to "solve it". Nobody is going to do it, except to win a bet. Certainly nobody is going to do it *accidently*.
For plain old iteration, "yield from" does nothing more than iterating over the iterable and yielding. There's no "edge cases" here, there are some pretty major differences. As the PEP says: If yielding of values is the only concern, this can be performed without much difficulty using a loop such as for v in g: yield v However, if the subgenerator is to interact properly with the caller in the case of calls to send() , throw() and close() , things become considerably more difficult. https://www.python.org/dev/peps/pep-0380/ "yield from" doesn't fix some weird edge case with iteration. It provides some significant new functionality which is *very* tricky to get right. That's not the case here with detecting an empty iterator. There are two perfectly adequate and simple ways to detect an empty iterator: # 1 empty = True for x in iterable: empty = False do_stuff() if empty: handle_empty() #2 x = sentinel = object() assert iterable is not locals().values() # *wink* for x in iterable: do_stuff() if x is sentinel: handle_empty() -- Steve
On Tue, Mar 22, 2016 at 11:16:55PM +1100, Chris Angelico wrote:
Are you being sarcastic?
I think that this is a completely artificial edge case that's never going to come up in practice, and is easy to work around: Just Don't Do That. It's perfectly safe if somebody passes you locals() from *another* scope, because it cannot have access to *your* local sentinel. So you only need worry about one case: when *you* pass *your own* locals as the iterable. Hence, *just don't do it*. "But what if I want to iterate over locals() and handle the case where it is empty?" if not locals(): handle_empty else: for value in locals().values(): ... Don't get me wrong, I think it was very clever that you thought of this case. But I don't think this is ever going to come up in practice. -- Steve
On Wed, Mar 23, 2016 at 12:10 AM, Steven D'Aprano <steve@pearwood.info> wrote:
I was, hence the function name.
And make sure the function you call doesn't use sys._getframe to get them for you.
Don't get me wrong, I think it was very clever that you thought of this case. But I don't think this is ever going to come up in practice.
Maybe. But it's an edge case of a form that could come up in other ways - I don't know of a way to enumerate all objects in CPython, but it's a perfectly reasonable way to debug a garbage collector. Point is that iteration can return literally *any* object. The object() sentinel is better than the None sentinel is better than the "if not item" falsiness check, but the only perfect solution is an extremely ugly one (and I'm not guaranteeing that even that is perfect - only that *I* haven't found a flaw in it yet). At some point, the solution is good enough for your code. ChrisA
On Wed, Mar 23, 2016 at 12:21:34AM +1100, Chris Angelico wrote:
Are you saying that I might have code like this: def test(): x = sentinel = object() iterable = some_function() for x in iterable: # you know the rest and some_function() might use _getframe to steal sentinel and return it back to me as iterable = [sentinel]? I'll take my chances with that, thanks very much. I am as concerned by that as I am that some_function might use ctypes to hack the value of integers so that 1 + 1 returns 4. Yes, it could happen. No, I don't care to take any special precautions to avoid it happening. I think a far more likely possibility is that somebody or something has monkey-patched object() to always return the same instance, regardless of who calls it from where, and now my sentinel is the same as everyone else's sentinel. Am I bothered by this? Not in the least. I mean, seriously, we write code all time relying on the behaviour of builtin functions, knowing full well that any piece of code anywhere might monkey-patch them to do something weird. Should we insist on special syntax that returns the len() of sequences because of an edge-case "what if len has been monkey-patched?". No. We just say "well don't monkey-patch len." In case it's not obvious by now, I don't think there's any need for syntax to handle the empty iterable case. -1 on the proposal. -- Steve
Steven D'Aprano wrote:
onestly, I just don't like the sentinel approach... while not perfect I prefer the explicit check on the throws of the StopIteration of the next function iterable = iter(some_function()) try: e = iter(iterable) except StopIteration: # empty else: # stuff with e for e in iterable: # stuff with e -- By ZeD
On Tue, Mar 22, 2016 at 09:36:23PM +0100, Vito De Tullio wrote:
I have no objection to the try...except approach either. Another approach is to call next() with a default: first = next(iterator, sentinel) if first is sentinel: empty else: for x in itertools.chain([first], iterator): process(x) We're spoiled for choice here: there are many ways to process an empty iterable separately from a non-empty one. -- Steve
It’s comparable in purpose to the ``else`` clause on a ``for`` or ``while``. It eliminates the need for a flag variable, like ``was_completed`` or ``is_empty``. It saves two lines of code, but more importantly avoids the accidental flip of True/False when setting the flag variable. Whether that warrants adding new syntax... I’m not convinced by the current proposals, but I can imagine finding a good keyword for the purpose. A keyword as good as ``else`` at least.
On 23 March 2016 at 14:13, Henshaw, Andy <Andy.Henshaw@gtri.gatech.edu> wrote:
For me, after checking all other e-mails, this is the "one and obvious way" of doing it, does not matter if it "seems" boring - it had not to do with being boring, or having one extra variable - it has to do with being readable. Event the variant that would do item = sentinel = object() for ... is less readable, if more elegant. As for adding different syntax for this, I am +0 For the idea of adding "except" clauses to 'with', 'for' and 'while' (along with a "EmptyIteration" exception) I am + 0.5
On 23.03.2016 20:06, Joao S. O. Bueno wrote:
So it seems, you consider a new keyword for the sole purpose of "discovering never executed" less worthwhile than the "except" alternative. What is the reason for this? The potential extensibility? Readability? Btw. I can understand and confirm that the "ifempty" use-case is not as frequent as the length of this thread suggests. So, I would not mind a slightly longer syntax ("except EmptyIteration") as long as it handles ALL corner cases correctly. This actually is my main driver here. I know that there are two or three easy patterns that fail in some cases. If a new syntax would be proposed that cannot handle all of them, it's not worth it. However, the frequency by which people here and on python-list suggest not-always working solutions still gives me a reason to work on this. Best, Sven
On Mon, Mar 21, 2016 at 1:01 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
AFAICS the OP's idea has a trivial and efficient expansion:
I'll take your word for it that this is efficient, and it is trivial in amount of code to write, but it is not the least bit trivial to read or write. It took me a good while to understand what it did, and how it works, and I never would have thought of it myself.
I would have done the klunky thing: _sentinal = False for item in iterable: _sentinel = True # do something if not _sentinel # do something else if the loop never ran almost as compact, less efficient and enough to make me see why the OP want's something new. Howver, the motivating example was "the difference between *nothing**as error* and *nothing** as really empty*, GUI/Web interfaces display an explicit notice in case of empty lists to signify the correct execution of a process/action/search or the other case." Sure -- but this seem to be mixing validation with processing code -- you really want to validate your inputs first. AND: this would mostly be containers, rather than iterables, so the idiom of: if input: for item in input: # do something else: raise EmptyInputError works fine. All that being said: how about "elempty"?, to go with elif? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 21.03.2016 17:31, Chris Barker wrote:
All that being said:
how about "elempty"?, to go with elif?
I am mostly dispassionate about the wording. So, why not elempty? [Btw. most people consider existing keywords for re-using as they cannot be used for variables in the first place] Best, Sven
Chris Barker writes:
By "efficient" I mean only that the loop is unchanged. Bonus is that only one assignment and one test are needed outside of the loop. The only reason I could think of at the time that it might not be trivial to read is that a reader is unfamiliar with the "sentinel = object()" idiom. Otherwise, "for" and "if" are the most basic of Python syntax, so I felt safe in using that word. Writing is another matter, but that's covered under the "sometimes ya gotta be Dutch" clause. :-)
If it's input, yes, but in my experience this would mostly be dealing with internal state such as a database. Then EAFP applies. For example, in paging, an empty page is a natural signal for end of content.
AND: this would mostly be containers,
"Mostly", maybe, but that's not to say that iterators are an unimportant case. And if your application involves a mix of containers and iterators, you may still prefer to use the same idioms for both cases.
All that being said:
how about "elempty"?, to go with elif?
-1 Not to my taste, to say the least.
On 22.03.2016 16:09, Stephen J. Turnbull wrote:
Hmm, it seems there is no easy solution for this. What do you think about an alternative that can handle more than empty and else? for item in collection: # do for item except NeverExecuted: # do if collection is empty It basically merges "try" and "for" and make "for" emit EmptyCollection. So, independent of the initial "never executed loop body" use-case, one could also emulate the "else" clause by: for item in collection: # do for item except StopIteration: # do after the loop Thinking this further, it might enable much more use-cases and stays extensible without the need to invent all kinds of special keywords. So, this "for" might roughly be equivalent to (using the "old for"): class NeverExecuted(StopIteration): pass i = iter(collection) try: e = next(i) except StopIteration: if [NeverExecuted/StopIteration is catched]: raise NeverExecuted else: # do for item e for e in i: # do for item e else: if [StopIteration is catched]: raise StopIteration I hope that's not too convoluted. Best, Sven
Le 22/03/2016 18:21, Sven R. Kunze a écrit :
It's an interesting idea. Generalizing except for more block I mean. Imagine: if user: print(user.permissions[0]) except IndexError: print('anonymous') with open('foo') as f: print(f.read()) except IOError: print('meh') for item in collection: # do for item except NeverExecuted: # do if collection is empty Not only does it allow interesting new usages such as the for loop empty pattern, but it does save a level of indentation for many classic use cases where you would nest try/except.
On Tue, Mar 22, 2016 at 06:21:24PM +0100, Sven R. Kunze wrote:
Possibly with the exception of the three or four previously existing easy solutions :-)
Does this mean that every single for-loop that doesn't catch NeverExecuted (or EmptyCollection) will raise an exception? If not, then how will this work? Is this a special kind of exception-like process that *only* operates inside for loops? What will an explicit "raise NeverExecuted" do?
That doesn't work, for two reasons: (1) Not all for-loops use iterators. The venerable old "sequence protocol" is still supported for sequences that don't support __iter__. So there may not be any StopIteration raised at all. (2) Even if StopIteration is raised, the for-loop catches it (in a manner of speaking) and consumes it. So to have this work, we would need to have the for-loop re-raise StopIteration... but what happens if you don't include an except StopIteration clause? Does every bare for-loop with no "except" now print a traceback and halt processing? If not, why not? -- Steve
On Mar 22, 2016, at 16:21, Steven D'Aprano <steve@pearwood.info> wrote:
The only question is whether any of them are obvious enough (even for novices). If not, we could argue whether any proposed change would be significantly _more_ obvious. And, if so, then the question is whether the change is worth the cost. But I think what we have is already obvious enough. (Especially since 90% of the time, when you need to do something special on empty, you explicitly have a sequence, not any iterable, so it's just "if not seq:".) So really, this proposal is really just asking for syntactic sugar that complicates the language in exchange for making some already-understandable code a little more concise, which doesn't seem worth it.
Elsewhere he mentioned that EmptyCollection would be a subclass of StopIteration. Presumably, every iterator type (or just the builtin ones, and hopefully "many" others?) would have special code to raise EmptyCollection if empty. Like this pseudocode for list_iterator: def __next__(self): if not self.lst: raise EmptyCollection elif self.i >= len(self.lst): raise StopIteration else: i = self.i self.i += 1 return self.lst[self.i] Or, alternatively, for itself would do this. The for loop bytecode would have to change to stash an "any values seen" flag somewhere such that if it sees StopIteration and hasn't seen any values, it converts that to an EmptyCollection. Or any of the other equivalents (e.g., the compiler could unroll the first PyIter_Next from loop from the rest of them to handle it specially). But this seems like it would add a lot of overhead and complexity to every loop whether desired or not.
Presumably that's the same question as what an explicit raise StopIteration does. Just as there's nothing stopping you from writing a __next__ method that raises StopIteration but then yields more values of called again, there's nothing stopping you from raising NeverExecuted pathologically, but you shouldn't do so. M
I think there always is. IIRC, PyObject_Iter (the C API function used by iter and by for loops) actually constructs a sequence iterator object if the object doesn't have tp_iter (__iter__ for Python types) but does have tp_sequence (__getitem__ for Python types, but, e.g., dict has __getitem__ without having tp_sequence). And the for loop doesn't treat that sequence iterator any different from "real" iterators returned by __iter__; it just calls tp_next (__next__) until StopIteration. (And the "other half" of the old-style sequence protocol, that lets old-style sequences be reversed if they have a length, is similarly implemented by the C API underneath the reversed function.) I'm on my phone right now, so I can't double-check any of the details, but I'm 80% sure they're all at least pretty close...
I think this could be made to work: a for loop without an except clause handles StopIteration the same as today (by jumping to the else clause), but one that does have one or more except clauses just treats it like a normal exception. Of course this would mean for/except/else is now legal but useless, which could be confusing ("why does my else clause no longer run when I add an 'except ValueError' clause?"). More generally, I think the fact that for/except StopIteration is almost but not quite identical to plain for would be confusing more often than helpful. But I think it is a coherent proposal, even if it's not one I like. :)
On Mar 23, 2016, at 03:08, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
But I think it is a coherent proposal, even if it's not one I like. :)
And what do you think about adding except clauses to if ? with ? while ?
Well, they don't have the problem that for has (for is already implicitly handling a specific exception in a specific way; none of them are), so they're coherent even without a solution to that problem. (With has the additional problem that it may not be immediately obvious on first glance whether the exit gets calls before or after the except clause, but if so, people can read the docs the first time they come across it and remember it after that.) But I also think they're even less necessary. They'd all be pure syntactic sugar for nesting the statement and a try/except, so we'd be making the language more complicated to learn and remember, and encouraging more code that isn't backward compatible. That's not a huge cost, but the benefit isn't very big either. For _really_ short cases, I think we want except expressions (PEP 463); for longish code, there's nothing wrong with an explicit try; the range for which hiding an implicit try inside if and while would really improve things is small enough that I don't think the benefit outweighs the cost. But maybe some good examples of realistic 3-liners that are significantly improved by the change would convince me (and, more importantly, convince a majority of the others who are skeptical), so I'll keep an open mind.
Le 23/03/2016 16:53, Andrew Barnert a écrit :
Would love to see except expression win, but hasn't been rejected by the BDFL ? Plus it's not really related to the current proposal.
But maybe some good examples of realistic 3-liners that are significantly improved by the change would convince me (and, more importantly, convince a majority of the others who are skeptical), so I'll keep an open mind.
The more I think about it, the more I think it's bad idea. We are mixing semantics of loops and exceptions or conditions and exceptions. I withdraw this part of the "general exceptions" proposal. I'm going back to the simpler and less dangerous proposal: for stuff in foo: bar(stuff) or: default()
On Mar 23, 2016, at 09:48, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
I thought PEP 463 was just stalled waiting on a reference implementation before he'd declare either way? At any rate, the reason it's related is that every case I could think of where I'd want to use if/except is so trivial that I'd much rather use an if expression in an except expression. There may be cases that are not quite trivial enough to demand that, but still small enough that saving one line of code makes a difference; just because I can't think of any certainly doesn't mean they don't exist. :) But, given the rest of your reply, I guess it doesn't matter.
You may want to get everyone back on track by writing a complete proposal (does or come before or after else? what exactly are the semantics? how does it translate to bytecode, if you understand how for already translates?) with just this suggestion some motivating examples. I still think this isn't really necessary. In every case I can think of where I really want to do something special on empty, I definitely have a sequence, and can just do "if not foo:", or I need a count rather than just an empty flag, or I'm already doing some EAFP handling, or... But again, just because I can't think of examples doesn't mean they don't exist. It just implies that maybe the other people you're trying to convince can't think of them either, so it would be better for you to show us.
On 23.03.2016 16:53, Andrew Barnert via Python-ideas wrote:
Could you describe what you mean by coherent?
(With has the additional problem that it may not be immediately obvious on first glance whether the exit gets calls before or after the except clause, but if so, people can read the docs the first time they come across it and remember it after that.)
But I also think they're even less necessary. They'd all be pure syntactic sugar for nesting the statement and a try/except, so we'd be making the language more complicated to learn and remember,
I completely disagree here.
and encouraging more code that isn't backward compatible.
What exactly is the problem with that? Everything discussed on python-ideas would be 3.6+ anyway.
That's not a huge cost, but the benefit isn't very big either. For _really_ short cases, I think we want except expressions (PEP 463); for longish code, there's nothing wrong with an explicit try; the range for which hiding an implicit try inside if and while would really improve things is small enough that I don't think the benefit outweighs the cost.
That's your opinion here. And I find it interesting that others already have thought of the merging of try and if, for and while. It would definitely come in handy. But I understand from your previous responses that you'd rather use Assembler because it is has no unneeded syntactic sugar. ;-) Just kidding.
But maybe some good examples of realistic 3-liners that are significantly improved by the change would convince me (and, more importantly, convince a majority of the others who are skeptical), so I'll keep an open mind.
From what I can see regarding the if-except proposal, the improvement would be constant. So, either you like it (as I do) or you don't see them as necessary (as you do). There is nothing somebody can do to convince you that saving 1 line of code and 1 indentation level across 20 lines of code are good or bad. Best, Sven
On Mar 23, 2016, at 13:26, Sven R. Kunze <srkunze@mail.de> wrote:
Someone earlier claimed that it's impossible to turn the vague proposal into any actual sensible semantics that could be implemented by Python. That would make it incoherent. But I don't believe it's true, so I tried to show how it could be defined well enough to be implementable. For the other three statements, I don't think that argument even comes up; I think the desired semantics are obvious enough that nobody can claim the proposal is incoherent.
Yes, but it's still a cost, that has to be weighed against the benefits. Some new syntax (like "with" or "yield from") is so useful that it's worth encouraging new code to use the new features and not be backward compatible. But that doesn't mean every piece of syntax that might be nice in a brand-new language is so useful that it's worth doing that. It's a high bar, and just ignoring the bar doesn't help you meet it.
You clearly don't understand my precious responses. If you don't see the difference between good syntactic sugar and useless syntactic sugar, then I can't explain to you why Python is better than assembler. :P
So you're refusing to provide examples because you're sure that no examples would convince anyone? In that case, I'm pretty sure your proposal is dead in the water, so I might as well stop responding.
On 23.03.2016 23:17, Andrew Barnert wrote:
You clearly don't understand my precious responses. If you don't see the difference between good syntactic sugar and useless syntactic sugar, then I can't explain to you why Python is better than assembler. :P
Good syntactic sugar is sweet. What means good to you? ;-)
So you're refusing to provide examples because you're sure that no examples would convince anyone?
No, I didn't say that. I don't need examples mostly because I see them each day. But because you asked, here you are: class MyClass: def mymethod(self): try: if my_condition: dosomething() elif another_condition(): makesomething() andanythingelse() else: butdontforgetaboutthis() except SomeException: pass finally: return None Would turn into: class MyClass: def mymethod(self): if my_condition: dosomething() elif another_condition(): makesomething() andanythingelse() else: butdontforgetaboutthis() except SomeException: pass finally: return None 1 line saved and 7 lines with 1 indentation level less (maybe even related to PEP8 and 80 chars line length). You might call it useless, I call it good. Regarding Python syntax, we are constantly going into territory where there is nothing much left to improve. The big "good syntactic sugar" is already done. However, I don't think we should stop here. 10 smaller improvements are as good one big one. Btw. I don't believe we could improve all Python code out there with this change, however when you are in such a situation you really appreciate it if you can do it. Especially when the surrounding source code already blows your mind; you are grateful for every single simplification (less lines, less indentation). I don't know what Python code you maintain but this would our lives easier. Best, Sven
On 23.03.2016 00:56, Andrew Barnert via Python-ideas wrote:
It was reasonable for both Django and for jinja2 to add this. People actually asked for it https://github.com/mitsuhiko/jinja2/issues/77
Repeating that you think it will be a sequence will make it one. We cannot keep track of all possible implementations in our code base. So, if somebody changes from sequence to iterator for whatever reason, it should work without weird errors.
Did you even think what you just said? Almost everything in Python is "just syntactic sugar" compared to most other Turing-complete languages. To put it differently: every construct that abstracts away "goto" is "just syntactic sugar".
Interesting. That would be an alternative approach.
Or, alternatively, for itself would do this.
I think the most important question is: do we want to support the "ifempty" feature in the first place? I can see a lot of discussion around it; that makes it controversial and that means there is half/half support/rejection (support by more people + rejection by less people but those have many good questions we would need to answer before we get this proposal to work). The only real reason against it so far: "it makes the language more complicated because I don't need it". Not entirely compelling but understandable. I further see that people would like this feature GIVEN a very GOOD syntax/approach and I entirely agree with that.
The for loop bytecode would have to change to stash an "any values seen" flag somewhere such that if it sees StopIteration and hasn't seen any values, it converts that to an EmptyCollection. Or any of the other equivalents (e.g., the compiler could unroll the first PyIter_Next from loop from the rest of them to handle it specially).
Something like that.
But this seems like it would add a lot of overhead and complexity to every loop whether desired or not.
If the "for" does not have any empty-equivalent clauses, there is no need to introduce that overhead in the first place. So we can conclude: 1) none overhead for regular "for"s 2) less overhead for "for-ifempty" because it would be done in C and not in Python
I think I would have to deal with the old protocol given the proposal is accepted.
One could disallow "else" in case any "except" is defined.
More generally, I think the fact that for/except StopIteration is almost but not quite identical to plain for would be confusing more often than helpful.
You bet how people think about "else". "So, 'else' is always executed after the for?" "Yes, but only when there is no 'break' executed in the 'for'" "... *thinking* ... okay ..."
But I think it is a coherent proposal, even if it's not one I like. :)
Best, Sven
On Mar 23, 2016, at 13:17, Sven R. Kunze <srkunze@mail.de> wrote:
I've said that if be happy to see any counterexamples, where you really do need this with iterators. So far, nobody has provided one. Of course absence of proof isn't proof of absence, but "you can't prove that it's impossible anyone will ever need this" is not a good rationale for a language change.
There's a big difference between syntactic sugar that makes hard-to-follow code more readable, and syntactic sugar that only makes some already-understandable code a little more concise. The former may belong in the language, the latter very rarely does. You seem to think that there's no inherent cost to adding new features to a language. For example, you later say:
The only real reason against it so far: "it makes the language more complicated because I don't need it". Not entirely compelling but understandable.
The more complicated the language is, the harder it is to keep it all in your head, to spot the control flow while skimming, to trace out the details when necessary, etc. Also, the more things you add, the more places there are for inconsistencies to creep in. (Of course it also makes the language harder to implement and maintain, but those are less important.) Only accepting changes that are actually worth the cost in increased complexity is a big part of what makes Python more readable than "dumping-ground" languages that have C-for, Python-for, while, do-while, until, repeat-until, and loop, some in both statement and expression variants, and some also writable in postfix form.
Not true. The first implementation I suggested, putting EmptyCollection into every iterable, requires the overhead in every case. The second one, changing the way the existing bytecodes work, means making frame objects (or _something_) more complicated to enable stashing the flags, which affects every for loop. The third, unrolling the first PyObject_Iter, has to be done if there's any code that can inspect the current exception state, which can't be statically determined, so it has to be always done. If you have a _different_ implementation, I'm happy to hear it. I supplied all the versions I could think of because another critic (Stephen? I forget...) implied that what you wanted was impossible or incoherent, and it clearly isn't. But it may not be a good idea to let a critic of your idea come up with the implementation. :)
For which of the three implementations? I'm pretty sure all of them would have significant overhead.
No, because, as I just explained, the old protocol is taken care of by wrapping old-style sequences in iterator objects, so as far as the for-loop code is concerned, they look identical to "new-style" iterables.
But that's just an extra rule to implement (and remember) for no real benefit. Why not just document that for/except/else is generally useless and shouldn't be written, and let linters flag it?
More generally, I think the fact that for/except StopIteration is almost but not quite identical to plain for would be confusing more often than helpful.
You bet how people think about "else". "So, 'else' is always executed after the for?" "Yes, but only when there is no 'break' executed in the 'for'" "... *thinking* ... okay ..."
I don't understand your point here. Because there's already something in the language that you find confusing, that gives us free rein to add anything else to the language that people will find confusing?
On 23.03.2016 23:11, Andrew Barnert wrote:
I've said that if be happy to see any counterexamples, where you really do need this with iterators.
1) you conveniently snipped the example I've given: somebody wrote functionality assuming a sequence (as innocently as most of the contributors to python-list and python-ideas); later the input sequence will be changed to an input iterator; "if len(collection)" may hide errors then 2) the colleague who basically initiated the whole discussion needed to implement some sort of "on-the-fly" converter pulling specific items from a third-party system A and pushing the converted items to another third-party system B. In case none items are there to be transferred and converted, a placeholder item still needs be pushed to system B. [yes, that is how somebody can make money; customers sometimes have interesting requirements]
The more complicated the language is, the harder it is to keep it all in your head, to spot the control flow while skimming, to trace out the details when necessary, etc.
And that is exactly why I and other like to see such an addition. It's not just the idea of a single person. It reduces the amount of lines of code and (for the correct implementation of all corner cases) it has less indentation and is presumably faster.
Only accepting changes that are actually worth the cost in increased complexity is a big part of what makes Python more readable than "dumping-ground" languages that have C-for, Python-for, while, do-while, until, repeat-until, and loop, some in both statement and expression variants, and some also writable in postfix form.
I understand that and want to keep it that way as you do. It seems that languages like Django and jinja2, which are 10x more concerned about feature reduction than Python, have {% empty %} or {% else %}. So, Python would not be the first language to support this feature.
Not sure if that can be easily done, but as I understand it, there would be two alternative bytecode implementations available. Depending on whether the "for" has an "empty"-equivalent clause, the "emptified/except/slow for" is used. If the "for" doesn't have such clause, the "original" "for" is used. So, depending on the actual code, the bytecode is different. Does this makes sense?
Oh, then I had interpreted what you said the wrong way. Thanks for explaining again.
But that's just an extra rule to implement (and remember) for no real benefit. Why not just document that for/except/else is generally useless and shouldn't be written, and let linters flag it?
Good idea!
Because there's already something in the language that you find confusing, that gives us free rein to add anything else to the language that people will find confusing?
I am open for suggestions. My argument here is that "except StopIteration" is not better/worse than "else" but the former is extensible. Best, Sven
Andrew Barnert via Python-ideas writes:
I presented a good try at a one-extra-line pure Python implementation (which is still good enough that I'm -1 on syntax, and waiting for a compelling use case for adding "for ... except"). It was Chris Angelico who shot a hole in it with locals().values().
implied that what you wanted was impossible or incoherent, and it clearly isn't.
I don't think there was ever a claim that it was incoherent/impossible or even inefficient at C-level. The point of the mildly obscure[1] item = sentinel idiom was to show that syntax isn't needed to avoid changing the loop body. I don't even know if Chris A would disagree that I succeeded well enough. Footnotes: [1] I know that several people found the item = _sentinel approach unreadable. Which surprised me: I'm usually the one who takes hours to understand the TOOWTDI one-liner. Maybe my newt-ness got bettah!
Let me start by saying I'm not against the proposal, and think it could make a lot of sense when properly implemented. However, I would like to specificially respond to the following. On Wed, Mar 23, 2016 at 09:17:32PM +0100, Sven R. Kunze wrote:
It would be good to remember that both Django and jinja2 are templating languages, and as such they impose different constraints on the user, as well as the sentiment that templates should be (nearly) devoid of all logic. Due to those constraints, adding an `empty` in the template language makes a lot more sense than in a general-purpose programming language. Not to mention the fact that representing 'no results' as an empty page is bad UX, but representing 'no results' from an API as an empty [list, generator, ...] is good design. Again, I like the proposal for the edge-cases it tries to solve (and sometimes I have seen use-cases where it makes sense), but please make sure you have better arguments and examples than 'look at Django and jinja2'.
On Mar 20, 2016, at 16:39, Vito De Tullio <vito.detullio@gmail.com> wrote:
It's not just "not trivial". The only way to do it with full generality (working for, e.g., generators) is to wrap the Iterator in a "peekable iterator" type (or something similar but more powerful, like tee). But that changes the semantics of the iterator: every time you consume element #n, it produces #n+1, not #n. And it's not hard to think of generators where that would be horribly inappropriate. For example, imagine a generator that's reading responses off a network socket. If you try to read response #n+1 when you haven't sent request #n+1 yet (because you're about to look at response #n), it'll fail, or block, or read incorrect information, or something else terrible. And there are other problems. For example, what if the generator needs to be able to handle g.throw; if you've replaced iterable with chain([peeked], iterable) or similar, that won't work. Of course the iterator protocol _could_ have been designed differently (consider C++, where iterators have separate operators to get the current value and to advance to the next one), which would have changed the design of generator functions and generator expressions and various other things that came later, so these particular problems would never have arisen. There are obvious advantages and disadvantages to each design. But Python chose one design a decade and a half ago, and a lot has been built on top of that design, and it's not going to change now. And, given that design, there's no way to make iterators peekable. Which means that it wouldn't be pythonic to make empty iterators falsey, because that's impossible to do without peeking, and I'm pretty sure the only reason the Zen doesn't say possible is better than impossible is that it's so obvious it doesn't need to be said. :) It's also worth noting that what you're asking for is really just a special case of LBYL vs. EAFP. Usually, it's better to just use the object and then see what happened, rather than checking in advance what will happen if you use it. Sometimes you can't avoid LBYL, or it's better because of the specifics of what you're doing, but in general, EAFP is the "default" way of thinking in pythonic code.
On Mar 20, 2016, at 16:54, Andrew Barnert via Python-ideas <python-ideas@python.org> wrote:
Possibly a better way to put this than my other answer: When you really do need to deal with any arbitrary iterable, it's more Pythonic to use the "if element is empty" post-check. When you expect your input to be a collection* (as was clearly your intuition here), go ahead and make use of that, using exactly the "if seq:" code that you wrote. If you need to deal with an iterable that might be an iterator rather than a collection, but if so it's guaranteed to be safely peekable... Well, I don't think that will ever actually come up, but if it does, you can do the peek-and-stuff-back thing instead of the "if element is empty". I think it would probably be simpler and more readable to do the post-test, and it would also avoid having to get across to your reader the idea that you can handle both collections and safely-peekable iterators but not non-safely-peekable iterators, but if you have some reason for that precondition that you wanted to express anyway, maybe using the peek-and-stuff code helps get it across? * Or "reiterable" or "repeatable iterable" or "non-iterator iterable" or any of the other identical or closely parallel concepts. I don't want to restart the argument about which one is more useful here; they all make the right distinction in this case.
On Sun, Mar 20, 2016 at 01:16:50PM -0700, Andrew Barnert via Python-ideas wrote:
"then" is my idea, and not only do I understand "for...else", but I like it and use it, and certainly don't want to eliminate it. I just hate the keyword used. Let me just start by saying that I realise that actually changing "for...else" to "for...then" is at least 8 years too late, so I know this is a non-starter. It would be particularly confusing to change "else" to "then" AND add a new "else" with different semantics at the same time.
An else clause is testing that no break was hit inside the loop. Look at a typical example:
That's one way of thinking about it. But I don't think it is helpful to think of it as setting an invisible flag "a break was hit inside the loop", and then testing it. I think that a more natural way to think about it is that "break" jumps out of the entire for...else compound statement. This has the big advantage that it actually matches what the byte code does in all the versions I've looked at. The "else" block is *unconditionally* executed after the "for" block. There's no "if not flag". Hence "then" is a better name for the construct: "for ... then ...". "break" doesn't set a flag, it jumps right out of the "for...else" statement altogether, not just out of the loop part, but the "else" part as well. (As I said, this matches what the byte code actually does.) As a beginner, I spent a very long time completely perplexed and confused by the behaviour of "for...else" because I understood it to mean "run the for block, *or else* run the else block". In other words, I understood from the keyword that "else" ran *if the for block didn't*, i.e. when the loop iterator is empty. A perfectly natural mistake to make, and I'm not the only person to have made it. This (wrong, incorrect) interpretation matches the most common and familiar use of "else", namely in if...else statements: if ...: a else: b You can get a, or b, but not both. In English, "else" represents an alternative. This does not come even close to matching the behaviour of for...else, which (in the absense of a "break" executes a *and* b, rather than a *or* b: for ...: a else: b I'm not Dutch, but I think that "else" is not a good name for a block of code which unconditionally executes after the for/while/try block. I think it's also unfortunately that it often looks like an indentation error: for x in seq: do_stuff() if condition: break else: do_more() I've seen people "fix" the indentation on code like this and wonder why the code then does the wrong thing. But, like I said, we're stuck with it, and I'm not seriously proposing a change. I think we'd be better off now if the keyword had been "then" from the beginning, but the pain of changing it *now* outweighs the benefit. -- Steve
On Mon, Mar 21, 2016 at 12:32 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Exactly. I came across this situation in a C++ program, and ended up writing the 'break' as a 'goto' - something like this, but with better names: for (int i=0;i<limit;++i) { do_stuff(); if (condition) goto for_else; } more_stuff(); for_else: And the comment against the goto said something along the lines of "this wants to be break-and-a-bit" - it needs to break out of both the loop and the code that follows it. "else" isn't the best keyword, but "finally" might give the wrong impression due to its main use with exceptions. It probably would be a better choice though. ChrisA
On Mar 20, 2016, at 18:32, Steven D'Aprano <steve@pearwood.info> wrote:
The way I've always thought about it is that it's more like a try/else than an if/else: it runs unless you've jumped out of the whole for statement (via break instead of raise). I think Nick Coghlan has a blog post that explains this very nicely, showing both the novice-understandable intuition and how a theoretical Python implementation could implement it (which is close to how CPython actually does, but maybe simpler), so I won't go over the details here. From what you write later, you also don't like the naming of try/else--in which case it's not surprising that you don't like the naming of for/else.
The "else" block is *unconditionally* executed after the "for" block.
The key difference is whether you treat "unless jumped out of" as meaning "unconditionally". I can see the sense of your way of looking at it (there's certainly a sense in which "while True:" loops "unconditionally", even if you have a break or return inside the loop, right?), so I understand where you're coming from. However, I still think _most_ people who don't like for/else don't actually understand it. But of course I can see that as another argument against the naming--if it's confused this many people over the last 20 years, maybe it's inherently confusing? At any rate, as you say at the end, there's no point arguing about this. The "else" clause isn't going to be renamed, and, even if it were, it wouldn't be reused for this new purpose, so that alternative to "empty" isn't worth discussing. (If I were designing a new Python-like language, I might rethink using "else"--but I don't think I'd use "then", or just leave it out the way Boo did.)
On Mar 20, 2016 10:10 PM, "Andrew Barnert via Python-ideas" < python-ideas@python.org> wrote:
On Mar 20, 2016, at 18:32, Steven D'Aprano <steve@pearwood.info> wrote:
On Sun, Mar 20, 2016 at 01:16:50PM -0700, Andrew Barnert via
Python-ideas wrote: than an if/else: it runs unless you've jumped out of the whole for statement (via break instead of raise). I think Nick Coghlan has a blog post that explains this very nicely, showing both the novice-understandable intuition and how a theoretical Python implementation could implement it (which is close to how CPython actually does, but maybe simpler), so I won't go over the details here.
From what you write later, you also don't like the naming of try/else--in
which case it's not surprising that you don't like the naming of for/else. I read the "else:", when attached to "try:", as "alternative to except:" (which makes sense even if no explicit "except" block exists). In "if-else", it means "alternative to if:" (or "if: and elif:"). If "switch-case" existed, "else:" would mean "alternative to case:". But in "for:"/"while:", there is no block keyword that the "else:" is the alternative to. It definitely isn't the alternative to the loop block itself, but that's probably the most common wrong intuition. (Disclaimer: I use "for-else" and "try-else" when I can. I don't dislike them at all. I'm just trying to explain the newbie confusion.)
On 3/20/2016 9:32 PM, Steven D'Aprano wrote:
Why is the truth a mistake?
and I'm not the only person to have made it. This (wrong, incorrect) interpretation
To me, this is disrespectful of other people.
matches the most common and familiar use of "else", namely in if...else statements:
Block a may never be executed. In any case, once it is, the loop starts over, the test is repeated, and either a or b is executed.
I see it differently. For both while and for, block b is the alternative to executing a, when the condition (explicit in 'while', implicit in 'for) is false, just as in an 'if' statement. I know I am not going to convince you, but please don't call me 'wrong' for seeing why 'else' makes sense and for understanding how 'while' is constructed from 'if' and 'jump'. The essential difference between an if statement and a while statement is the jump back that causes a repeat of the test. -- Terry Jan Reedy
On Sun, Mar 20, 2016 at 11:19:54PM -0400, Terry Reedy wrote:
It's not the truth. The else block does *not* only run if the for block doesn't run. It runs regardless of whether the loop iterable is empty or not.
It's a statement of fact. I have seen other people make the same mistake I made: interpreting the "else" clause as only running if the loop iterable is empty. People have been mislead by the keyword. What's disrespectful about pointing this out?
It's an IF...else statement. There's no loop.
That's not what for...else does. Try it with an empty sequence and a non-empty sequence: for x in [1,2]: print("inside for block") else: print("inside else block") for x in []: print("inside for block") else: print("inside else block") BOTH the empty and non-empty cases print "inside else block". It is certainly not "the alternative". -- Steve
I've taught enough and I'm sure everyone else here has too, to know that the "else" in a for loop in non-intuitive to a lot of folks. And maybe a different keyword would have been clearer. But it is what it is, could we keep this discussion to the proposed addition? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Le 21/03/2016 17:34, Chris Barker a écrit :
+1 I would love "else" semantics to be changed as well, but we can't. What about: for x in stuff: foo(x) or: bar() It's a bit weird when you read it the first time, but: - It conveys pretty well the intent. - While it still closes to the original "or" semantics, you can't confuse this syntax with the other syntax for "or". - I can't think of a use case where the parser can find that ambigious. - It's short.
On 21/03/2016 18:37, Michel Desmoulin wrote:
The trouble with "or" or "else" after a for-loop is that it suggests (following English usage) alternative actions: Either execute this for-loop, or do something else. If I were designing Python 4, I might opt for for x in stuff: foo(x) ifnobreak: bar() Pro: "ifnobreak" is at least explicit, and not particularly likely to clash with an already-used variable name. Con: "ifnobreak" is too long and wordy. But I can't think of anything better. "aftercomplete", "whencomplete" are just as bad. Hell, I'm coming round to "then". What about allowing "then" to have an indent between that of the for-statement and the for-body: for x in stuff(): foo(x) then: bar() Of course, you still have to learn it, like all idioms, but having learned it, isn't it a bit more readable? The idea could be extended to other suites, conveying that everything indented is associated with the initial statement of the suite: try: foo() except SomeException: bar() Rob Cliffe
Le 22/03/2016 00:53, Rob Cliffe a écrit :
But that's exactly the goal of "or" here. Run only if the loop is never executed since stuff is empty.
Python 4 won't break compat the way Python 3 did (Guido's words), so Python 4 is not a better opportunity for introducing new stuff than any other release. But I do like the "nobreak" better than else (you can drop the if, it's just noise). However, again, we can't remove else, and "there should be one way to do it" prevents us to add duplicates keywords.
On Mon, Mar 21, 2016 at 12:35 PM Chris Barker <chris.barker@noaa.gov> wrote:
I'm not sure the keyword being "else" is the real problem. It's certainly correlated, but I've noticed a similar confusion for "finally" and "else" in "try" blocks, where the keywords are quite natural. The confusion might in fact simply be a difficulty with the concept "do this if the loop completed without breaking" rather than a difficulty linking that concept to the word "else". Either way, that's a tangent, and I agree that discussion on "else" should be (mostly) separate from a discussion on adding a new keyword.
Itertools is great, and some functions in it are more used than others: - islice; - chain; - dropwhile, takewhile; Unfortunatly many people don't use them because they don't know it exists, but also are not aware of the importance of generators in Python and all in all, the central place iteration has in the language. But I must confess that after 12 years of Python, I always delay the use of it as well: - I have to type the import in every single module and shell session (ok I got PYTHONSTARTUP setup to autoimport, but most people don't). - All functions fell verbose for such a common use case. - If I start to use it, I can say good by to simpler syntaxes such as [] and +. - It always take me a minutes to get dropwhile/takewhile right. They works the opposite way of my brain. The changes I'm going to propose do not add new syntax to Python, but yet would streamline the use of this nice tool and blend it into the language core. Make slicing accept callables ============================= One day someone asked me something similar to: "I got a list of numbers, how do I filter this list so that I stop when numbers are bigger than 4." So I said: print([x for x in numbers if x > 4]) But then he said: "No, I want to stop reading any number I encounter after the first x > 4." "Oh". Then: import itertools def stop(element): return not element > 4 print(list(itertools.takewhile(stop, numbers)) I actually got it wrong 2 times, first I forgot the "not", then I mixed up the parameters in takewhile. I was going to then introduce lambda but my colleagues looked at me in a sad way after glancing at the code and I backed up. So my first proposal is to be able to do: def stop(element): return element > 4 print(numbers[:stop]) It's quite pythonic, easy to understand : the end of the slice is when this condition is met. Any not the strange way takewhile work, which is "carry on as long as this condition is met". We could also extend itertools.islice to accept such parameter. Slicing any iterable ====================== Now, while I do like islice, I miss the straigthforwardness of [:]: from itertools import islice def func_accepting_any_iterable(foo): return bar(islice(foo, 3, 7)) It's verbose, and renders the [3:7] syntaxe almost useless if you don't have control over the code creating the iterable you are going to process since you don't know what it's going to be. So the second proposal is to allow: def func_accepting_any_iterable(foo): return bar(foo[3:7]) The slicing would then return a list if it's a list, a typle if it's a tuple, and a islice(generator) if it's a generator. If somebody uses a negative index, it would then raises a ValueError like islice. This would make duck typing and iteration even easier in Python. Chaining iterable ================== Iterating on heterogenous iterable is not clear. You can add lists with lists and tuples with tuples, but if you need more, then you need itertools.chain. Few people know about it, so I usually see duplicate loops and conversion to lists/tuples. So My first proposal is to overload the "&" operator so that anything defining __iter__ can be used with it. Then you can just do: chaining = "abc" & [True, False] & (x * x for x in range(10)) for element in chaining: print(element) Instead of: from itertools import chain chaining = chain("abc", [True, False], (x * x for x in range(10))) for element in chaining: print(element)
On Tue, Mar 22, 2016 at 10:06 AM, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
You're not the first to ask for something like this :) Let's get *really* specific about semantics, though - and particularly about the difference between iterables, iterators, and generators.
This cannot be defined for arbitrary iterables, unless you're proposing to mandate it in some way. (It conflicts with the way a list handles slicing, for instance.) Even for arbitrary iterators, it may be quite tricky (since iterators are based on a protocol, not a type); but maybe it would be worth proposing an "iterator mixin" that handles this for you, eg: class IteratorOperations: def __getitem__(self, thing): if isinstance(thing, slice): if has_function_in_criteria(slice): return self.takeuntil(s.start, s.stop) return itertools.islice(...) def takeuntil(self, start, stop): val = next(self) while start is not None and not start(val): val = next(self) while stop is None or not stop(val): yield val val = next(self) As long as you inherit from that, you get these operations made available to you. Now, if you're asking this about generators specifically, then it might be possible to add this (since all generators are of the same type). It wouldn't be as broad as the itertools functions (which can operate on any iterable), but could be handy if you do a lot with gens, plus it's hard to subclass them.
Again, while I am sympathetic to the problem, it's actually very hard; islice always returns the same kind of thing, but slicing syntax can return all manner of different things, because it's up to the object on the left:
You don't want these to start returning islice objects. You mentioned lists, but other types will also return themselves when sliced. Possibly the solution here is actually to redefine object.__getitem__? Currently, it simply raises TypeError - not subscriptable. Instead, it could call iter() on itself, and then attempt to islice it. That would mean that the TypeError would change to "is not iterable" (insignificant difference), anything that already defines __getitem__ will be unaffected (good), and anything that's iterable but not subscriptable would automatically islice itself (potentially a trap, if people don't know what they're doing).
Again, anything involving operators is tricky, since anything can override its handling. But if you require that the first one be a specific iterator class, you can simply add __and__ to it to do what you want: class iter: iter = iter # snapshot the default 'iter' def __init__(self, *args): self.iter = self.iter(*args) # break people's minds def __iter__(self): return self def __next__(self): return next(self.iter) def __and__(self, other): yield from self.iter yield from other Okay, so you'd probably do it without the naughty bits, but still :) As long as you call iter() on the first thing in the chain, everything else will work. ChrisA
The solution I'm currently using involves having a class called g, and everytime I want to manipulate an iterable, I just wrap it in g(). Then I got an object with all those semantics (and actually a lot more). Maybe we can make only those things apply to the objects returned by iter() ? (Also, about slicing accepting callable, actually g() goes a bit overboard and accept any object. If the object is not an int or callable, then it's used as a sentinel value. Not sure if I should speak about that here.) Le 22/03/2016 00:36, Chris Angelico a écrit :
On Tue, Mar 22, 2016 at 10:59 AM, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
Oh, that's easy then. iter = g :)
Nothing wrong with that. The semantics are still the same; you simply have several ways of defining "the end has been reached". ChrisA
On 2016-03-21 16:59, Michel Desmoulin wrote:
I'm not sure I see the value in new syntax, but I think what you describe here would be a useful addition to itertools. Yes, you would still have to import itertools, but you could just import this one "convenient iterator" class and use that, which could allow for concise but readable code that otherwise would have to use many different itertools functions. It would require some thought about what sorts of overloaded operators (if any) would be appropriate for various itertools functions, but I think it could still be a gain even if the main advantage was just being able to write stuff like "niceIterator(x)[10:20] + niceIterator(y)[20:30]". Since it would just be a convenience wrapper, there need be no worries about it not working for particular kinds of iterables (e.g., ones that already define their own behavior for certain operators); you just wouldn't use it for ones where it masked behavior you cared about on the underlying iterable. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Mar 21, 2016, at 16:06, Michel Desmoulin <desmoulinmichel@gmail.com> wrote: In addition to all the issues Chris raised...
The first issue is pretty minor compared to the later ones, but already shows the problems of thinking about only lists and iterators, so I won't skip it: Applying this to a sequence that copies when slicing makes sense. Applying it to an iterator (together with your #2) makes sense. Applying it to a type that returns views on slicing, like a NumPy array, doesn't necessarily make sense, especially f that type is mutable, like a NumPy array. (Should the view change if the first element > 4 changes?)
And what if it's a dict? Returning an islice of an iterator over the dict _works_, but it's almost certainly not what you want, because an iterator over the dict gives you the dict's keys, not its values. If d[3:7] means anything, I'd expect it to mean something like {k: v for (k, v) in d.items() if 3<=k<7}, or {v for ...same...}, not 4 arbitrary keys out of the dict. (Imagine that d's keys are all integers. Surely you'd want it to include d[3], d[4], d[5], and d[6], right?) And what if it's one of the collections on PyPI that already provides a non-islice-like meaning for slice syntax? For example, many of the sorted-dict types do key slicing, which returns you something like {k: v for (k, v) in d.items() if 3<=k<7} but still sorted, in log rather than linear time, and sometimes a view rather than a copy. And what if it's a collection for which indexing makes no sense, not even the wrong kind of sense, like a set? It'll slice out 4 values in arbitrary order iff there are at least 7 values. What's the good in that? And what if it's an iterator that isn't a generator? Does it just return some arbitrary new iterator type, even if the input type provided some additional interface on top of Iterator (as generators do)? Also, even with iterators a lot of things you do with slicing no longer make sense, but would run and silently do the wrong thing. For example: if matches_header(seq[:6]): handle_body(seq[6:]) If seq is an iterator, that first line is going to consume the first 6 elements, which means the second is now going to start on the 12th element rather than the 6th. It will be a lot less fun to debug "why do some of my messages lose their first word or so, but others work fine?" than the current "why do I sometimes get a TypeError telling me that type list_iterator isn't indexable?"
So, what does {1} & {1, 2, 3} do? This is a trick question: sets already define the & operator, as do other set-like collections. So this proposal either has to treat sets as not iterable, or break the set interface. All of these are part of the same problem: you're assuming that all iterables are either sequences or generators, but many of them--including two very important built-in types, not to mention all of the builtin types' iterators--are not.
Le 22/03/2016 01:45, Andrew Barnert a écrit :
Numpy is not part of the stdlib. We should not prevent adding a feature in Python because it will not immediately benefit an 3rd party lib, even a famous one. The proposal doesn't hurt Numpy : they override __getitem__ and choose not to accept the default behavior anyway, and are not affected by the default behavior. Besides, you generally try to not mix Numpy and non Numpy manipulation code as it has it's own semantics (no for loop, special slicing, etc.).
This is a point to discuss. I would raise a ValueError, trying to slice a dict is almost always a mistake. E.G: if you design a function that needs an argument to be sliceable (event with islice), you usually don't want people to pass in dicts, and when you strangely do (to sample maybe ?), then you would cast it manually. It would be a rare use case, compared to the multiple occasions you need a more generic slicing.
And what if it's one of the collections on PyPI that already provides a non-islice-like meaning for slice syntax? For example, many of the sorted-dict types do key slicing, which returns you something like {k: v for (k, v) in d.items() if 3<=k<7} but still sorted, in log rather than linear time, and sometimes a view rather than a copy.
See the point about Numpy.
And what if it's a collection for which indexing makes no sense, not even the wrong kind of sense, like a set? It'll slice out 4 values in arbitrary order iff there are at least 7 values. What's the good in that?
See the point about dict.
And what if it's an iterator that isn't a generator? Does it just return some arbitrary new iterator type, even if the input type provided some additional interface on top of Iterator (as generators do)?
This can be discussed and is more about the proper implementation, but does not discard the validity of the idea.
Either you use generators or you don't. If you use generators, you know they will be consumed when you pass them around. This has nothing to do with the slicing syntax. The one problem I can see is when: - seq is a generator you didn't produce; - you don't know it's a generator. - you get a surprising behavior because slicing cause no errors. It's an edge case and is worth considering if it's going to be a blocker or not. Also, one alternative is to only add slicing to all objects returned by iter() in the stdlib. This would force people to explicitly mark that they know what they are doing, and while less convenient, remains very handy.
Indeed I forgot about sets. Maybe there is another operator that would do the trick, such as "<<". We can't use "+" as it would be confusing, and sets() overide a LOT of operators. Anyway, let's not throw the baby with the water. This is the least important part of the proposal, and any part can be changed, improved or ditched. That's what Python-ideas is for.
All of these are part of the same problem: you're assuming that all iterables are either sequences or generators, but many of them--including two very important built-in types, not to mention all of the builtin types' iterators--are not.
"Slicing any iterable" was a bad title. It should have been "extend the slicing application". I'm not assuming any iterable is a sequence or a generator, I think we can come up with a reasonable behavior for the case slicing doesn't make sense, adding more power to this handy tool. Please also consider the iter() proposal as a more verbose alternative, yet still powerful alternative. Maybe even easier to implement ?
On 22 March 2016 at 16:19, Michael Selik <mike@selik.org> wrote:
And consider the other side of the coin. If I read code using a function of the itertools module, its easy to look up the definition; if I read code featuring a rarely used syntax, how easy is it to come up with search engine terms to describe the syntax, and thus find documentation for the feature?
On 3/22/2016 03:34, Graham Gower wrote:
This is a pure devil's advocate response, as I don't feel there is a need to add itertools functionality to the language itself. Googling for 'Python with keyword' or 'Python lambda keyword', for example, give very good results (documentation or examples in the first 5 results on both). Any proposal involving adding a new keyword would probably be as discoverable as the standard library. ... Which is the big gotcha with arguments involving new keywords. If it makes it exactly as discoverable, then nothing is gained by new syntax, and we are back where we started. Your point is accurate for any syntax that does not involve new keywords, or for new uses of existing keywords.
Le 22/03/2016 08:34, Graham Gower a écrit :
I see your point. Again, this is the weakest point of my proposal, we should not discard all of it just because of that. And maybe we can come up with something better. E.G: I suggested before that iter() returns objects with advanced semantics for [:] as an alternative to add slicing to generators. Maybe the same objects can come with a chain() method, which is explicit. Or provide an additional builtin function, which does help with those issues if changing iter() is not possible.
On Tue, Mar 22, 2016 at 11:58 PM, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
E.G: I suggested before that iter() returns objects with advanced semantics for [:] as an alternative to add slicing to generators.
The builtin iter() function simply returns whatever __iter__() returns. I've offered a way for you to easily shadow the builtin with your own class (which can then provide whatever it likes), but it's going to have to work by wrapping the original iterator, which will break generators (unless you explicitly pass everything through, and even then you'll break things that check if isinstance(x, generator), because you can't subclass generator). ChrisA
What if itertools.iter was added with the expanded semantics? Possibly even a new name, but similar use to the built-in func. Top-posted from my Windows Phone -----Original Message----- From: "Chris Angelico" <rosuav@gmail.com> Sent: 3/22/2016 6:06 Cc: "python-ideas" <python-ideas@python.org> Subject: Re: [Python-ideas] Integrate some itertools into the Python syntax On Tue, Mar 22, 2016 at 11:58 PM, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
E.G: I suggested before that iter() returns objects with advanced semantics for [:] as an alternative to add slicing to generators.
The builtin iter() function simply returns whatever __iter__() returns. I've offered a way for you to easily shadow the builtin with your own class (which can then provide whatever it likes), but it's going to have to work by wrapping the original iterator, which will break generators (unless you explicitly pass everything through, and even then you'll break things that check if isinstance(x, generator), because you can't subclass generator). ChrisA _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On 22 March 2016 at 16:01, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
Note that sys, os, re, math, datetime are all "an import in every file you want to use it". The bar for changing Python is higher than just avoiding an import. Paul
Le 22/03/2016 17:18, Paul Moore a écrit :
This is an appeal to consider islice & Co as important as normal slicing. Indeed, you will have most certainly iterables in any code that use datetime, re or math. You have no certainty of having datetime, re or math imported in any code dealing with iterables. Think about how annoying it would be to do:
For every slice. We don't, because slicing is part of our standard data processing toolkit. And I think we can make it even better. We already do in some places. E.G: range(10)[3:5] works while range() generate values on the fly. Why ? Because it's convenient, expressive and Pythonic. Well, if you process a file and you want to limit it to all lines after the first "BEGIN SECTION" (there can be other) and before the first "STOP" (there can be others), but only 10000 lines max, and not load the whole file in memory, you could do: def foo(p): with open(p) as f: def begin: return x == "BEGIN SECTION" def end: return x == "STOP" return f[begin, end][:10000] It's very clean, very convenient, very natural, and memory efficient. Now compare it with itertools: from itertools import takewhile, dropwhile, islice def foo(p): with open(p) as f: def begin: return x != "BEGIN SECTION" def end: return x != "STOP" return islice(takewhile(end, dropwhile(begin, f)), 0, 10000) It's ugly, hard to read, hard to write. In Python, you are always iterating on something, it makes sense to make sure we have the best tooling to do at our fingertips.
On 03/22/2016 10:51 AM, Michel Desmoulin wrote:
Except the 10,000 limit doesn't happen until /after/ the end block is reached -- which could be a million lines later.
Yes, it is. I would make a function: def take_between(begin=0, end=None, limit=None): """ return a list with all items between `begin` and `end`, but no more than `limit` begin -> an `int`, or function that returns `True` on first line to keep end -> None (for no end), an `int` relative to start of iterable, or a function that returns True on last item to keep limit -> max number of items to keep, or None for all items """
In Python, you are always iterating on something, it makes sense to make sure we have the best tooling to do at our fingertips.
In Python, you are always using data containers -- yet we don't have a plethora of different tree types built in. A simple import is "at our fingertips". On a more supportive note: Are you aware of `more_itertools`[1] ? If it doesn't already have `takeuntil` and `dropuntil` and your (pretty cool) `iter`, perhaps you could get your ideas included there. -- ~Ethan~ [1] https://pypi.python.org/pypi/more-itertools/
On Tue, Mar 22, 2016 at 8:41 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
if f[begin, end] is a generator, the 10000 limit may happen before the end block is reached, which I think was the point. Python 3 does give generators and iterators a more central role than Python 2. One thought that comes to mind is to add some stuff from itertools as attributes of iter. For example: iter.slice(...). Kind of like a factory method, although iter is not really a type. - Koos
Le 22/03/2016 20:23, Ethan Furman a écrit :
[begin, end] and [:10000] are applied one next() at a time. begin, then end, then :10000 for the first next(), then again in that order for the following next() call, etc. That's the whole point.
On 03/22/2016 12:26 PM, Michel Desmoulin wrote:
That may be the point, but that is not what the above code does. Since you don't believe me, let's break it down: f[begin:end] -> grabs a section of p. This could be 5 lines or 50000000 [:10000] -> take the first 10000 of the previous result return -> send those (up to) 10000 lines back -- ~Ethan~
No you didn't store the lines, the whole points is that they are returning generators. In that context, f[begin, end] does itertools.takewhile and dropwhile and [:10000] is doing islice(0, 10000). None of those functions store anything. Please read the begining of thread. Le 22/03/2016 20:41, Ethan Furman a écrit :
On 03/22/2016 12:43 PM, Michel Desmoulin wrote:
I presume you are referring to this line:
The slicing would then return a list if it's a list, a tuple if it's a tuple, and a islice(generator) if it's a generator.
Of which the big problem is that the object returned from an open call is not a list, not a tuple, and not a generator. For the sake of argument let's pretend you meant "and an islice(iter) for everything else". So the actual proposal is to add `__getitem__` to `object` (that way all classes that define their own `__getitem__` such as list and tuple are not affected), and the substance of this new `__getitem__` is to: - check for an `__iter__` method (if none, raise TypeError) - check the start, stop, step arguments to determine whether dropwhile, takewhile, or islice is needed - return the appropriate object Okay, it's an interesting proposal. I would suggest you make the changes, run the test suite, see if anything blows up. As a short-cut maybe you could add the code to the appropriate ABC and test that way. -- ~Ethan~
2016-03-22 20:43 GMT+01:00, Michel Desmoulin <desmoulinmichel@gmail.com>:
It would be true if f is generator. If f is list then we could probably want to have another operator to avoid memory consumption return list(f{begin,end}{:10000}) #return list(f{begin,end}[:10000]) # this could be same
If your point is that generators can sometimes be hard, then your point is clear :-). Maybe using them should not be made to look more straightforward than it actually is. Which brings me back to... One thought that comes to mind is to add some stuff from itertools as attributes of iter. For example: iter.slice(...). Kind of like a factory method, although iter is not really a type. - Koos
Le 22/03/2016 20:52, Koos Zevenhoven a écrit :
If your point is that generators can sometimes be hard, then your point is clear :-).
Maybe using them should not be made to look more
straightforward than it actually is.
I would have made the same mistake with itertools, let be frank. Which brings me back to...
It's an interesting though. Actually we could also make something like iter.wraps(iterable) to return and object with the properties discussed in that thread. It would be fully backward compatible, explicit, yet convenient enough.
On Mar 22, 2016, at 12:52, Koos Zevenhoven <k7hoven@gmail.com> wrote:
The one thing from itertools that I think people most miss out on by not knowing itertools is chain.from_iterable, with chain itself a close second. Would calling them iter.chain.from_iterable and iter.chain help? I'm not sure. Other people have proposed promoting one or both to builtins (maybe named flatten and chain, respectively), but that hasn't gone over very well. Making them "class methods" of iter seems a lot less disruptive. The other thing that I think might help is a way of writing "slice literals" more easily. Just having islice easier to find doesn't help much, because the main hurdle in writing and reading the code seems to be translating between the natural slice syntax and islice's (range-ish) unpythonic signature. Maybe islice(it, slice[:10000])? Or even islice(it)[:10000]?
On Tue, Mar 22, 2016 at 9:10 PM, Koos Zevenhoven <k7hoven@gmail.com> wrote:
I should add, though, that the the function should probably return list(f[begin, end][:10000]) or something, before the file f is closed. - Koos
Le 22/03/2016 20:26, Koos Zevenhoven a écrit :
Yeah the semantic of my function is wrong. It should rather expect a file like object as a parameter to avoid this bug. Plus, the begin() and end() functions are actually syntax errors ^^
On Tue, Mar 22, 2016 at 9:45 PM, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
Yeah, and perhaps be a generator function and yield from f[begin:end][:10000] instead of returning it. But beginners may end up trying this. It may be better to have them as functions (either in itertools or perhaps as attributes of iter), which may help them see what's actually going on. Or help them see that they actually don't know how it works and should just make a for loop for this to be more clear to them. - Koos
On Mar 22, 2016, at 10:51, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
Range is a sequence, just like list and tuple, not an iterator, like generator and list_iterator. So it supports slicing because sequences generally support slicing. Not because it's so convenient and expressive that it's worth making a special case, just because it follows the normal rules. It's amazing to me that so many people still describe xrange/3.x range as an iterator or generator, and then draw further incorrect lessons from that. Yes, the docs were a bit confusing up to about 2.4, but how long ago is that now? Lazy sequences are still sequences. Strict non-sequences (like dict) are still not sequences. Lazy vs. strict has almost[1] nothing to do with sequence vs. iterator (or sequence vs. non-sequence collection, or collection vs. iterator, or anything else.) --- [1]: Just "almost nothing" because you can't really write a strict generator--or, rather, you sort of can in a way, but it still has to look lazy to the user, so there's no point.
Le 22/03/2016 21:03, Andrew Barnert a écrit :
Yes, I worded my title with "all iterables", it was a mistake, I didn't think people would raise the debate on dict, set, etc because it was obvious to me that nobody would imagine I would try to make slicing work on those. I was wrong, but please don't make drown the debate into a battle of semantics. Ok, let's recenter to lazy sequences, and how convenient it would be to be able to limit them to a size or a condition without needing itertools. And in that context, the fact is that range(1000000) doesn't create 1000000 items in memory, and it's very convenient to be able to slice it. And it would would be as convenient to be able to do so on generators (which are sequences) instead of using islice.
On Mar 22, 2016, at 13:11, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
This is not just a battle of (meaningless) semantics. You seem to be completely misinterpreting the actual language semantics of Python, and therefore making proposals that don't make any sense. Case in point: you can _already_ limit lazy sequences to a size without needing itertools. That's exactly what you're doing in your range example. So we don't need to fix the language to make something possible when that something is already possible.
No, generators are _not_ sequences. And that's exactly the point you're missing. You have to understand the difference between iterators and other iterables to use Python effectively. That's just fundamental to so much of Python that there's no way around that. Maybe that was a language design mistake, but trying to muddy the waters further and confuse even more people is hardly a solution. Meanwhile, the range type points to a ready solution: when you want slicing (or other sequence behavior) with laziness, you can always write a lazy sequence type, usually in only a few lines of code. (I've got a couple blog posts on it, if you can't figure it out yourself.) And that solves the problem without introducing any confusion about what it should mean to slice an iterator, or any problems of the "if header(spam[:6]): process_body(spam[6:])" type.
On 3/22/2016 4:03 PM, Andrew Barnert via Python-ideas wrote:
An example I recently ran across is http://python4kids.brendanscott.com/2016/03/19/python-for-kids-python-3-proj... Python4Kids uses range-as-list for the 2.x version. For the 3.x version, he is agonizing over explaing 'range-as-generator'. -- Terry Jan Reedy
Le 22/03/2016 06:49, Michael Selik a écrit :
It's a gut feeling really, coming from the fact I train people in Python for a living: - most of my colleagues and myself always show the builtins way before anything else. I rarely have the time to even show itertools. - same goes for tutorials online. Itertools is rarerly demonstrated. - a whole module is more impressive than a few behavior and my student will usually delay reading about any doc related to one module. They don't even know where to start or how to express what they are looking for - it's something they will try (or discover by mistake): random playing with operators and data is something I witness a lot. Random playing with imports, not so much. I may be wrong, I can't finance a study with a big representative sample to prove it. But again, nobody can do so for anything on this list.
On Tue, Mar 22, 2016 at 8:52 AM Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
Luckily we have the case of set.union and set.intersection versus the pipe and ampersand operators. One way you could provide some evidence is comparing the frequency of usage of the method vs operator in public projects. We might need to then discard code examples written by experts from our analysis, but it'd be a start.
On 22 March 2016 at 09:06, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
iter() already has a two-argument form to accept a callable+sentinel value, so it may not be unreasonable to offer an "until" callback to say when to stop iterating. That is: def stop(element): return element >4 print(list(iter(numbers, until=stop))) Slicing arbitrary iterables may be amenable to a str.join style solution, by putting the functionality on slice objects, rather than on the iterables: slice(3, 7).iter(iterable) (Whether or not to make slice notation usable outside subscript operations could then be tackled as an independent question) For itertools.chain, it may make sense to simply promote it to the builtins. I'm not the least bit sure about the wisdom of the first two ideas, but the last one seems straightforward enough, and I'd be comfortable with us putting "chain" on the same tier as existing builtin iteration related tools like enumerate and zip. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le 23/03/2016 06:13, Nick Coghlan a écrit :
Do you propose to make a "since" param too ?
Those solutions are already much more convenient than using itertools, but it's going to look very strange: def foo(p): with open(p) as f: def begin(): return x.startswith('#') def end(): return x != "STOP" yield from slice(0, 10).iter(iter(f, since=begin, until=en)) The last line is really not Pythonic. Compared to: def foo(p): with open(p) as f: def begin(): return x.startswith('#') def end(): return x != "STOP" yield from f[begin:end][:10] But I beleive there is an idea here. Attaching things to slice is cleaner than to iter(): def foo(p): with open(p) as f: def begin(): return x.startswith('#') def end(): return x != "STOP" yield from slice.wraps(f)[begin:end][:10]
Same problem as with new keywords : it can be a problem with people using chain as a var name.
To be fair, I don't nearly use chain as much as zip or enumerate, so I'm not going to push for something as drastic.
Regards, Nick.
On Wed, Mar 23, 2016 at 9:04 PM, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
Less of a problem though - it'd only be an issue for people who (a) use chain as a var name, and (b) want to use the new shorthand. Their code will continue to work identically with itertools.chain (or not using it at all). With a new keyword, their code would instantly fail. ChrisA
I enjoy using ``chain`` as a variable name. It has many meanings outside of iteration tools. Three cheers for namespaces. Many of Python 3’s changes versus Python 2 were to take the pure stance -- change special statement syntax to function calls, move a few builtins into appropriate modules. If chain were a builtin, we’d have someone suggesting it move into itertools.
On Mar 23, 2016, at 10:13, Michael Selik <mike@selik.org> wrote:
As Chris just explained in the message you're replying to, this wouldn't affect you. I've used "vars" and "compile" and "dir" as local variables many times; as long as I don't need to call the builtin in the same scope, it's not a problem. The same would be true if you keep using "chain". Unless you want to chain iterables of your chains together, it'll never arise. Also, putting things in builtins isn't _always_ bad. It's just a high standard that has to be met. I don't think anyone believes any, all, and enumerate fail to meet that standard. So the question is just whether chain meets it. My problem is that I'm not sure chain really does meet it. It's chain.from_iterable that I often see people reaching for and not finding, and moving chain to builtins won't help those people find it. (This is compounded by the fact that even a novice can figure out how to do chain given chain.from_iterable, but not the other way around.) Also, for something we expect novices to start using as soon as they discover iterators, it seems worrisome that we'd be expecting them to understand the idea of a static method on something that looks like a function but is actually a type before they can have a clue of what it means. In another thread a year or two ago, someone suggested making chain.from_iterable into a builtin with a different name, maybe "flatten". But that now means we're adding _two_ builtins rather than one, and they're very closely related but don't appear so in the docs, which obviously increases the negatives... Still, I like adding chain (and/or flatten) to builtins a lot more than I like adding sequence behavior to some iterators, or adding a whole new kind of function-like slicing syntax to all iterables, or any of the other proposals here.
On Wed, Mar 23, 2016 at 1:39 PM Andrew Barnert <abarnert@yahoo.com> wrote:
Depends what you mean by "affect". It'll affect how I read my colleagues' code. I want to see ``from itertools import chain`` at the top of their modules if they're using chain in the sense of chaining iterables.
Part of why ``any`` and ``all`` succeed as builtins is their natural language meanings are quite close to their Python meaning. ``enumerate`` succeeds because it is unusual in natural language. For many people, Python might be the most frequent context for using that word. Some evidence that ``chain`` should be promoted to builtin would be that ``chain`` is used more often than the rest of the itertools library and that the word "chain" is rarely used except as the itertools chain. Luckily one could search some public projects on GitHub to provide that evidence. If no one bothers to do so, I'm guessing the desire isn't so great.
I like the LazyList you wrote. If the need for operator-style slicing on iterators were great, I think we'd find a decent amount of usage of such a wrapper. Its absence is evidence that ``from itertools import islice`` is more pleasant. As has been discussed many times, lazy sequences like range cause some confusion by providing both generator behavior and getitem/slicing. Perhaps it's better to keep the two varieties of iterables, lazy and non-lazy, more firmly separated by having ``lst[a:b]`` used exclusively for true sequences and ``islice(g, a, b)`` for generators. Just yesterday I got frustrated by the difference of return types from subscripting bytes versus str while trying to keep my code 2/3 compatible: ``bytestring[0]`` gives an integer while ``textstring[0]`` gives a 1-length str. I had to resort to the awkwardness of ``s[0:1]`` not knowing whether I'll have a Python 3 bytes or Python 2 str. I prefer imports to inconsistency. A couple questions to help clarify the situation: 1. Do you prefer ``a.union(b)`` or ``a | b``? 2. Do you prefer ``datetime.now() + timedelta(days=5)`` or ``5.days.from_now``? I think the answer to #2 is clear, we prefer Python to Ruby (more specifically the Rails sub-community). The answer to #1 is more difficult. I'm often tempted to say ``a | b`` for its elegance, but I keep coming back to ``a.union(b)`` as clunky but readable, easy to explain, etc.
The advantage of having a small set of builtins is that you know the entire set of builtins. If chain really is useful enough that it belongs as a builtin, you will very quickly adapt to reading that code, and it won't bother you or slow down your comprehension at all, any more than any other builtins do. Adding dozens of builtins would break that; adding one wouldn't. There's only room for a handful more builtins in the entire future life of Python, and the question of whether chain deserves to be one of them is a big question (and I suspect the answer is no). But I think it's the only real question, and adding more on top of it doesn't really help us get to the answer.
Some evidence...
Agreed. And again, my own anecdotal experience is that chain.from_iterable is actually used (or sadly missed) more than chain itself, especially among novices, so I'm not advocating making chain a builtin unless someone proves me wrong on that.
Still, I like adding chain (and/or flatten) to builtins a lot more than I like adding sequence behavior to some iterators, or adding a whole new kind of function-like slicing syntax to all iterables, or any of the other proposals here.
I like the LazyList you wrote. If the need for operator-style slicing on iterators were great, I think we'd find a decent amount of usage of such a wrapper. Its absence is evidence that ``from itertools import islice`` is more pleasant.
The one on my blog? I've never found a use for that in real code[0], hence why (IIRC) I never even put it on PyPI. But I use range all the time, and various other lazy sequences. The problem with LazyList isn't that it's lazy, but that it's a recursive cons-like structure, which is ultimately a different way to solve (mostly) the same set of problems that Python already has one obvious solution for (iterators), and an ill-fitting one at that (given that Python discourages unnecessary recursive algorithms). I've also written a more Python-style lazy list that wraps up caching an iterator in a list-like object. It's basically just a list and an iterator, along with a method to self.lst.extend(islice(self.it, index - len(self.lst)). The point of that is to show how easy it is to write, but how many different API decisions come up that could be reasonably resolved either way, so there's probably no one-size-fits-all design. Which implies that if there are apps that need something like that, they probably write it themselves. Anyway, if your point is that iterators having slicing would be a bad thing, I agree with you. But that doesn't necessarily mean that chain and islice as they are today is the best possible answer, just that it's better than adding operators to the iterator type (even if there were such a thing as "the iterator type", which there isn't). Using islice can still be clumsy, and I'm happy to see what alternatives or improvements people come up with.
As has been discussed many times, lazy sequences like range cause some confusion by providing both generator behavior and getitem/slicing. Perhaps it's better to keep the two varieties of iterables, lazy and non-lazy, more firmly separated by having ``lst[a:b]`` used exclusively for true sequences and ``islice(g, a, b)`` for generators.
Definitely not. Even if breaking range and friends weren't a huge backward compat issue, it would weaken the language. And, meanwhile, you would gain absolutely nothing. You'd still have sets and dicts and third-party sorted trees and list iterators and itertools iterators and key views and so on, none of which are either sequences or generators. (Also, think about this parallel: do you want to say that dict key views shouldn't have set operators because they're lazy and therefore not "real sets"? If not, what's the difference?) The problem isn't that Python has things that are neither sequences nor generators, it's that the Python community has people who think it doesn't. Any time someone says range is (like) a generator, or any similar misleading myth, they need to be corrected with extreme prejudice before they set another batch of novices up for confusion. So, again: range does not provide anything like "generator behavior". It's not only repeatably iterable, it's also randomly-accessible, container-testable, reversible, and all the other things that are true of sequences.
Just yesterday I got frustrated by the difference of return types from subscripting bytes versus str while trying to keep my code 2/3 compatible:
That's a completely separate problem. I think everyone agrees that there are design mistakes in the bytes class. As of 3.5, most of the ones that can be fixed have been, but some of them we're just unfortunately stuck with. None of those problems have to do with the iteration or sequence protocols.
A couple questions to help clarify the situation: 1. Do you prefer ``a.union(b)`` or ``a | b``?
When I'm doing stuff that's inherently mathematical-set-like, I use the operator. When I'm using sets purely as an optimization, I use the method.[1] But how does that relate to this thread? I think the implied assumption in the proposal is that people want to do "inherently sequence-like stuff" with iterators; if so, they _should_ want to spell it with brackets. I think the problem is that they're wrong to want that, not that they're trying to spell it wrong.
2. Do you prefer ``datetime.now() + timedelta(days=5)`` or ``5.days.from_now``?
Of course the former.[2] But the problem with "5.days" is nothing to do with this discussion. It's that it implies a bizarre ontology where natural numbers are things that know about timespans as intimately as they know about counting,[3] which is ridiculous.[4] If you really wanted numbers to know how to do "inherently day-like stuff", the Ruby way would make sense; you just shouldn't want that. --- [0]: I have found uses for it in comparing translated Lisp/ML/Haskell idioms to native Python ones, or in demonstrating things to other people, which is why I wrote it in the first place. [1]: Or sometimes even the classmethod set.union(a, b). In fact, sometimes I even write it out more explicitly--e.g., {x for x in a if x in b} in place of a & b emphasizes that I could easily change the {...} to [...] if I need to preserve order or duplicates in a. [2]: I like now() + days(5), or maybe even now() + 5*days, even more. But that's easy to add myself without modifying either int or timedelta, and without confusing my readers. [3]: Or, only slightly less accurately, it's like saying "5's days" instead of "5 days" in English. [4]: The Ruby answer to that is that writing an application is really about writing an app-specific language that lets you write the app itself a trivial single-page program, and in an OO context, that means extending types in app-specific ways. The number 5 in general doesn't know how to construct a timespan, but the number 5 in the context of a calendaring web application does. That's an arguably reasonable stance (Paul Graham has some great blog posts making the non-OO version of the same argument for Lisp/Arc), but it's blatantly obvious that it's directly opposed to the Python stance that the language should be simple enough that we can all immediately read each other's code. Python doesn't encourage indefinitely expanding core types for the same reason it doesn't encourage writing your own flow control statements.
On Thu, Mar 24, 2016 at 8:46 AM, Andrew Barnert <abarnert@yahoo.com> wrote:
The advantage of having a small set of builtins is that you know the entire set of builtins.
Python 3.6 has len(dir(builtins))==150, which is hardly small. Even removing the ones that don't begin with a lower-case letter (which removes a small number of special names like __package__, and a large number of exception classes) leaves 72. Can you name them all? ChrisA
On Mar 23, 2016, at 15:01, Chris Angelico <rosuav@gmail.com> wrote:
Can you name all 25000 words of English that you know? There are 68 builtins listed in a nice table at the top of the builtin functions chapter in the docs. Take a look at that table. They're all (except maybe format and iter[1]) functions[2] that you can immediately recognize them when reading them, and recall when they're relevant to code you're writing, including know exactly what their interface is and what they do. If you see "any()" or "int()" in code, you don't need to turn to the docs to look up what it does. If you need to convert something to a debug string representation, you know to call "repr()". The fact that you might leave out "any", "int", or "repr" in a pop quiz demanding that you list all 68 of them in alphabetical order doesn't mean you don't know them. You certainly can't say the same is true for all functions and methods in the stdlib. --- [1]: I think everyone knows iter _exists_, but not everyone knows about its two-argument form. [2]: ... or types...
On Thu, Mar 24, 2016 at 9:38 AM, Andrew Barnert <abarnert@yahoo.com> wrote:
Fair enough, naming them all is a poor definition of "know". But the flip side (knowing what one means when you meet it) is also a poor definition. The real trick is knowing, when you're faced with a problem, what the solution is; and while most people here will be able to do that, I'm not sure that every Python programmer can. Some builtins are well known, like int and len; others, not so much - zip, hash, divmod. And then there are some (compile, bytearray, callable) that, if they were elsewhere in the stdlib, would barely change people's code. How many non-experts know exactly which function decorators are builtins and which are imported from functools? If you have to look them up, (a) there's not a lot of difference between built-in and stdlib, and (b) the builtins are already a bit crowded. My conclusion: itertools.chain is just as discoverable as builtins.chain, so the only real benefit would be playing around at the interactive prompt, which you can improve on with an auto script. -0.5 on making chain a builtin. ChrisA
On 03/23/2016 04:17 PM, Michel Desmoulin wrote:
Le 24/03/2016 00:15, Chris Angelico a écrit :
You know, that's a good point! It just has one problem: I use datetime.datetime and datetime.date a heck of a lot more than chain and chain.from_iterable; and os.getuid and os.setuid! and sys.argv, and collections.OrderedDict, and sys.version_info, and sys.modules, and most of the things in logging... So all those things should be builtin first. Then we can talk about chain. Chaining-all-my-favorite-imports-ly yrs, -- ~Ethan~
On Mar 23, 2016, at 16:00, Chris Angelico <rosuav@gmail.com> wrote:
Well, readability is at least as important as writability. But yes, writability is also important, which is why I covered it: 'If you need to convert something to a debug string representation, you know to call "repr()".'
and while most people here will be able to do that, I'm not sure that every Python programmer can.
Sure, there's a certain level of competence below which you can't say people know all the builtins. There's also a certain level of competence below which you can't say people know every kind of statement in the syntax. But the point is that the level is low enough to be reasonably achievable. It's not only the hardcore experts that know almost all of the builtins. If we tripled the number of builtins, I don't think that would be true anymore. (Secondarily, the alphabetical chart in the docs is good enough at 68; at 200, we'd almost certainly need to reorganize it to make it useful.)
I don't think anyone would be horribly bothered by moving compile outside of builtins, but it has to be there for bootstrapping reasons. (Maybe that isn't true anymore, but it apparently was in 3.0.) For callable, that actually _was_ removed from builtins in 3.0, and added back pretty late in the 3.2 cycle because, as it turns out, people really did miss it more than you'd expect a priori. But anyway, the presence of 2 or 3 or even 10 functions out of those 68 that maybe shouldn't be there isn't license to add 100 more. If anything, it means we have to be a tiny bit more careful what we add there in the future.
I don't think it's _just_ as discoverable. But I do think it's discoverable _enough_. (Plus, as I already mentioned, I think chain.from_iterable is more of a problem than chain, and moving chain to builtins wouldn't help that problem enough to be worth doing.) So I'm also -0.5.
On Mon, Mar 21, 2016 at 8:06 PM, Michel Desmoulin <desmoulinmichel@gmail.com
wrote:
I like how dropwhile and takewhile could be easily integrated with list comprehensions / generator expressions: [x for x in range(10) while x < 5] # takewhile [x for x in range(10) not while x < 5] # dropwhile [x for x in range(10) from x >= 5] # forward thinking dropwhile I believe this is almost plain english without creating new keywords. Regards
On Wed, Mar 23, 2016, 5:27 PM João Bernardo <jbvsmo@gmail.com> wrote:
They read well, except for the square brackets which to me imply consuming the entire iterator. Itertools takewhile will early exit, possibly leaving some values on the iterator. If these do consume the entire iterator, what's the difference with the ``if`` clause in a comprehension?
On Wed, Mar 23, 2016, 5:55 PM Sven R. Kunze <srkunze@mail.de> wrote:
After the execution of the list comprehension using "while" if the condition triggered halfway, would there be anything left in the original iterator? If it was a generator expression, we'd say "Yes" immediately. As a list comprehension, I'm not sure.
João Bernardo On Wed, Mar 23, 2016 at 7:18 PM, Michael Selik <mike@selik.org> wrote:
Dropwhile alone would consume the entire iterator for obvious reasons. Takewhile will stop when necessary, otherwise why implement it? You could nest them and still be readable: [x for x in range(10) from x > 3 while x < 7] Still quite readable. If this were an iterator like "foo = iter(range(10))", there will be items left in the end.
On 3/21/2016 9:29 AM, Steven D'Aprano wrote:
Why is the truth a mistake?
It's not the truth.
Yes it is. When the iterator is empty, possibly after many iterations, and the iterator is tested, the else clause runs.
The else block does *not* only run if the for block doesn't run.
I have never, ever, claimed that. Initially empty and empty after all items have been processed are different.
It becomes a while loop when 'go to if-test' is added after 'a'.
Each time next(iterable) is called by the for loop machinery, executing b is the alternative to executing a. You and I view the overall looping process differently. The way I see it, as repeating if-else whenever if is true, makes 'else' perfectly sensible. You see it differently, in a way that makes 'else' less sensible. Here is an alternative way to construct 'while condition' from if-else. Imagine that we only have a loop-forever primative (implemented internally as an unconditional jump). Python currently spells this as 'while True'. Then 'while condition' could written as the following (which works today). while True: if condition: true_part() else: false_part() break The false_part is the alternative to the true_part. -- Terry Jan Reedy
On 3/22/2016 3:34 PM, Ethan Furman wrote:
On 03/22/2016 11:49 AM, Terry Reedy wrote:
from the perspective of a particular loop.
but for/else and while/else treat them the same.
Which is to say, each test is a test of current state, independent of history. It does not matter if the iterater was created empty, was emptied by previous code out or inside the current function, or emptied by the current loop. People who want to condition on history should record it. The idiom presented previously is a form of recording history. item = marker = object() # could be earlier in function body for item in iterable: ... else: if item = marker: # started loop empty ... -- Terry Jan Reedy
On 03/23/2016 03:25 PM, Terry Reedy wrote:
On 3/22/2016 3:34 PM, Ethan Furman wrote:
From the perspective of many, many (most?) people. I use for loops and while loops *alot*, and I can count the times I have wanted to use the `else` clause in the meaning of "all my tests failed, so do this block" on the fingers of one hand.
By the way, I appreciate your explanations -- they help make a little more sense out of it -- but the behaviour of `for/else` and `while/else` is definitely a learned exception to the general rule of "truthy -> this thing is something" / "falsey -> there is nothing here". -- ~Ethan~
I"ve only used for .. else a handful of times, and honestly, probably forgot to use it when it would have been the right way to do something far more times :-) but I can't think of a SINGLE time when I wanted to know that the loop didn't run because it's iterable was empty. Which is my way of saying: I think adding something would make this use case more clear and easy to do, but it's a rare enough use case that it's not worth it. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Thu, Mar 24, 2016 at 5:34 PM Chris Barker <chris.barker@noaa.gov> wrote:
More specifically, a time when the loop didn't run because the iterable was empty AND the iterable didn't have a len to make that a simple check. Which suddenly gives me an idea... class Peeking: def __init__(self, iterator): self.it = iter(iterator) self.sentinel = object() self.buf = next(self.it, self.sentinel) def __iter__(self): return self def __next__(self): if self.buf is self.sentinel: raise StopIteration value = self.buf self.buf = next(self.it, self.sentinel) return value def __bool__(self): return self.buf is not self.sentinel If you find yourself in a situation where you want to check empty as if your iterator was a sequence (ie. len(seq) == 0 means False), give this wrapper a shot just before you do the loop. >>> bool(Peeking(range(0))) False >>> bool(Peeking(range(1))) True We could even make it a context manager to monkey-patch a buffering inside the context for the bool check, then undo the patch on leaving the context to stop buffering.
On 03/24/2016 03:14 PM, Michael Selik wrote:
You mean like this answer? http://stackoverflow.com/a/7976272/208880 -- ~Ethan~
On Tue, Mar 22, 2016 at 02:49:44PM -0400, Terry Reedy wrote:
I just realised I typo-ed that. I meant "iterable", not iterator. For-loops can iterate over any iterable (sequences, sets, dicts etc) not just iterators (as I'm sure you know). I may have inadvertently given you the wrong impression, in which case, I apologise.
I can only guess that you thought I was referring to the invisible, implementation-dependent iterator which is used internally by the for-loop. (See byte code below for what I mean.) That's fine, but it isn't what the topic of this discussion is about. It's irrelevent. Nobody should care about whether *that* iterator is exhausted or not, because that iterator is unreachable from Python code. What we might care about is the iterable "seq" used in (for example) "for x in seq". Sometimes people want to run a block of code *only* if the body of the for loop never runs, and some people mistakenly thought, or think, that the "else" clause does that. I know this for a fact because I was one of those people, and I have seen others make the same mistake. I can prove it is a mistake by running this code: seq = [2, 4] for x in seq: print("seq is not empty") # this is correct else: print("seq is empty") # this is wrong assert list(seq) == [] The assertion won't *always* fail, e.g. try passing "seq = []") but the fact that it *sometimes* fails proves that it is a mistake to interpret the "else" clause as "run only when the for loop doesn't run". Take this for-loop: def f(): for x in seq: print(x) else: print(1) In Python 2.7, dis.dis(f) gives: 2 0 SETUP_LOOP 24 (to 27) 3 LOAD_GLOBAL 0 (seq) 6 GET_ITER >> 7 FOR_ITER 11 (to 21) 10 STORE_FAST 0 (x) 3 13 LOAD_FAST 0 (x) 16 PRINT_ITEM 17 PRINT_NEWLINE 18 JUMP_ABSOLUTE 7 >> 21 POP_BLOCK 5 22 LOAD_CONST 1 (1) 25 PRINT_ITEM 26 PRINT_NEWLINE >> 27 LOAD_CONST 0 (None) 30 RETURN_VALUE My guess is that you thought I was referring to the iterator which is created by the call to GET_ITER, and iterated over by FOR_ITER. Mea culpa -- I was not. I meant to say iterable, not iterator, and in this example I would mean "seq". If you look at the byte code, you will see that there is no test for whether this internal iterator, or seq for that matter, is empty. When the for loop, the code unconditionally executes the else block. If the for loop contains a break (or a return, or a raise) execution may transfer to outside of the for...else altogether, skipping the else block, but there is still no test for emptiness. It's an unconditional jump: if you reach the break, you jump past the end of the for...else.
Then I hope that we now are on the same page, and agree that: (1) "else" does NOT mean "execute this block only if the for block doesn't run", even if some people mistakenly did, or do, think so. (2) Some people want such a statement, and that's the topic for this discussion. (3) As for...else now exists, there is no testing whether or not the for-loop ITERABLE is empty, or whether the internal (inaccessible) ITERATOR is empty; furthermore, other implementations or versions may not even use an internal ITERATOR (e.g. Python 1.5 certainly doesn't). (4) "break" doesn't set a flag, it just jumps to the outside of for...else (although in principle this is implementation dependent, there is nothing in the semantics of for...else that *requires* a flag be set), or the iterable be tested for emptiness. -- Steve
2016-03-21 2:32 GMT+01:00, Steven D'Aprano <steve@pearwood.info>: [...]
A. Just a little analyze about this theoretical possibility for x in seq: if good(x):break # (1) then: all_bad() # (2) else: empty() # (3) (3) seems to be alternative to (2) like "if then else" , but is not , it is alternative to (1) "union" (2) Also empty is not contradictory to all_bad! 'else' seems to be not ideal but yes we have what we have B. if we don't want to define new keywords and want to map all (?) possibilities (empty vs non empty, break vs no break) and don't want to break backward compatibility then we could analyze this: for i in seq: if cond(i): print('non empty & break') # (1) break and: print('non empty & all') # (2) or: print('no elements') # (3) else: print('no break') # (4) (1) is exclusive but (4) could be with (2) or<exclusive> (3) (2) and (3) are contradictory I have not opinion about order of and, or, else in this moment.
On Sun, Mar 20, 2016 at 07:12:58PM +0100, Sven R. Kunze wrote:
"empty" is a poor choice, as I expect that it will break a lot of code that already uses it as a variable. if empty: ...
I'm not too keen on the look of "or" for this: for x in iterable: block or: alternative but I think it's the least worst of the proposed keywords.
4) except -> as alternative if a new keyword is too much
I think that "except" completely fails to describe what the keyword does, and it will give some people the misunderstanding that it catches errors in the body of the for-loop. -- Steve
On 21.03.2016 02:44, Steven D'Aprano wrote:
So, "empty" is off the table I suppose.
Got it but I am open for suggestions. :-)
Now, that you mention it. This might give birth to a completely different idea. What about? for item in collection: # do for item except EmptyCollection: # do if collection is empty except StopIteration: # do after the loop Which basically merges try and for? Best, Sven
On Tue, Mar 22, 2016 at 01:20:05PM +0100, Sven R. Kunze wrote:
From your statement 'merges try and for', it would seem as you now want
What worries me about this is that the next question is going to be: Why not have this on `with` suites, `if` suites, and so on. I've already seen remarks symmilar to that as replies. Especially because of your remark 'Which basically merges try and for', which adds to my worries. Consider the following: for item in [0, 1, 2]: raise StopIteration except StopIteration: # do after the loop # Also after a `raise` inside the loop body? the exception to be caught in this case as well. Now, maybe you did not intend it as such, and really intended to only catch the exceptions raised by the iterable itself. But that was not quite clear to begin with. Because of this (possible) confusion, I'd really suggest slapping on another keyword to the `for` statement over allowing `except` clauses with limited use. Another reason is that for item in [0, 1, 2]: # do something except StopIteration: pass would do exactly the same as the `for` loop without the `except` clause, (due to the `pass`). Now this does not break the principle of 'one obvious way to do it', because for Pythonistas with experience, the `except` is superfluous. But would a novice know? (Assuming he started not by reading the Python docs, but by working on code written by another programmer)? Yes, RTFM, but more often you learn by reading other peoples code.
On 24.03.2016 07:54, Sjoerd Job Postmus wrote:
My initial motivation is to support the "empty" use-case. I am not attached to any proposal as long as that is achieved. For one reason, I dislike to explain how to emulate the missing feature to others (especially when Django and jinja2 supports it) and for another I found usage for it for myself recently. [offtopic] I am curious now. What is the problem with providing exception handling to more than just try? Is it just because it complicates the language which you are used to? Or is there something really problematic I miss here?
Especially because of your remark 'Which basically merges try and for', which adds to my worries.
This was to make people understand it more easily. :-)
That's interesting. Yes, that could be confusing. But what can we do? I cannot believe that we run out of ideas just for something that some simpler languages can handle easily. :-/
Noted but I wouldn't give too much weight to this issue. As others noted "else" is equally confusing. Best, Sven
On 20/03/2016 18:12, Sven R. Kunze wrote:
I don't do hypothetical. Practicality beats purity. So having followed this thread I do not believe that language changes are needed. So why not add a paragraph to this https://docs.python.org/3/tutorial/controlflow.html#break-and-continue-state... giving the cleanest solution from this thread for the way to write this Pythonically. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence
Am 20.03.2016 um 19:12 schrieb Sven R. Kunze:
Yes, this looks good. What about this? {{{ for item in my_iterator: # do per item on empty: # this code gets executed if iterator was empty on break: # this code gets executed if the iteration was left by a "break" on notempty: # ... }}} Is there a case which I have forgotten? ... I have no clue how to implement this :-) Regards, Thomas -- Thomas Guettler http://www.thomas-guettler.de/
On 24.03.2016 14:54, Thomas Güttler wrote:
Hmm, interesting. "on" would indeed be a distinguishing keyword to "except". So, "except" handles exceptions and "on" handles internal control flow (the iter protocol). Nice idea actually.
}}}
Is there a case which I have forgotten?
"on notbreak" I suppose. I think the downside here is that we would need to register another keyword "on". Is this used as a normal variable often? Not sure if "on empty" can be registered as a single keyword since it contains whitespace. Best, Sven
Am 24.03.2016 um 18:44 schrieb Sven R. Kunze:
I thank you, because you had the idea to extend the loop syntax :-) But I guess it is impossible to get "on empty" implemented. Regards, Thomas Güttler -- Thomas Guettler http://www.thomas-guettler.de/
What about:
The empty marker would be set to True but the for loop and set to False when iterating on the loop. The perf cost of this is only added if you explicitly requires it with the "as" keyword. You don't handle break, which is handle by else anyway. And maybe having access to the iterator could open the door to more features. Le 06/04/2016 16:16, Thomas Güttler a écrit :
participants (29)
-
Alexander Walters -
Andrew Barnert -
Brendan Barnwell -
Chris Angelico -
Chris Barker -
Ethan Furman -
Franklin? Lee -
Graham Gower -
Henshaw, Andy -
Joao S. O. Bueno -
João Bernardo -
Koos Zevenhoven -
Mark Lawrence -
Michael Selik -
Michel Desmoulin -
Nick Coghlan -
Paul Moore -
Pavol Lisy -
Rob Cliffe -
Robert Collins -
Sjoerd Job Postmus -
Stephen J. Turnbull -
Steve Dower -
Steven D'Aprano -
Sven R. Kunze -
Terry Reedy -
Thomas Güttler -
Vito De Tullio -
Émanuel Barry