Hello all, First time post, so go gentle. :)
During a twitter conversation with @simeonvisser, an idea emerged that he suggested I share here.
The conversation began with me complaining that I'd like a third mode of explicit flow control in Python for-loops; the ability to repeat a loop iteration in whole. The reason for this was that I was parsing data where a datapoint indicated the end of a preceding sub-group, so the datapoint was both a data structure indicator *and* data in its own right. So I'd like to have iterated over the line once, branching into the flow control management part of an if/else, and then again to branch into the data management part of the same if/else.
Yes, this is unnecessary and just a convenience for parsing that I'd like to see.
During the discussion though, a much more versatile solution presented itself: repurposing the `continue` keyword to `continue with`, similarly to the repurposing of `yield` with `yield from`. This would avoid keyword bloat, but it would be explicit and intuitive to use, and would allow a pretty interesting extension of iteration with the for-loop.
The idea is that `continue with` would allow injection of any datatype or variable, which becomes the next iterated item. So, saying `continue with 5` means that the next *x* in a loop structured as `for x in..` would be 5.
This would satisfy my original, niche-y desire to re-iterate over something, but would also more broadly allow dynamic injection of *any* value into a for-loop iteration.
As an extension to the language, it would present new and exciting ways to iterate infinitely by accident. It would also present new and exciting ways for people to trip up on mutable data; one way to misuse this is to iterate over data, modifying the data, then iterating over it again. The intention, rather, is that the repeated iteration allows re-iteration over unaltered data in response to a mid-iteration change in state (i.e., in my case, parsing a token, changing parser state, and then parsing the token again because it also carries informational content).
However, bad data hygiene has always been part of Python, because it's a natural consequence of the pan-mutability that makes it such a great language. So, given that the same practices that an experienced Python dev learns to use would apply in this case, I don't think it adds anything that an experienced dev would trip over.
As a language extension, I've never seen something like "continue with" in another language. You can mimic it with recursive functions and `cons`-ing the injection, but as a flow-control system for an imperative language, nope..and Python doesn't encourage or facilitate recursion anyway (no TCE).
I'd love thoughts on this. It's unnecessary; but then, once something's declared turing complete you can accuse any additional feature of being unnecessary. I feel, more to the point, that it's in keeping with the Python philosophy and would add a novel twist to the language not seen elsewhere.
best, Cathal
Can't we achieve the same effect with code like the following. Maybe slightly less elegant, but only slightly and hence not worth adding to the language.
class Continuation(object):
def __init__(self, val):
self.val = val
# Other stuff, including creating the iteratable
x = None
while True:
if isinstance(x, Continuation):
x = x.val
else:
try:
x = it.next()
except StopIteration:
break
if something(x):
do_things(x)
else:
x = Continuation(special_value)
continue
On Wed, Sep 24, 2014 at 3:09 PM, Cathal Garvey <cathalgarvey@cathalgarvey.me
wrote:
Hello all, First time post, so go gentle. :)
During a twitter conversation with @simeonvisser, an idea emerged that he suggested I share here.
The conversation began with me complaining that I'd like a third mode of explicit flow control in Python for-loops; the ability to repeat a loop iteration in whole. The reason for this was that I was parsing data where a datapoint indicated the end of a preceding sub-group, so the datapoint was both a data structure indicator *and* data in its own right. So I'd like to have iterated over the line once, branching into the flow control management part of an if/else, and then again to branch into the data management part of the same if/else.
Yes, this is unnecessary and just a convenience for parsing that I'd like to see.
During the discussion though, a much more versatile solution presented itself: repurposing the `continue` keyword to `continue with`, similarly to the repurposing of `yield` with `yield from`. This would avoid keyword bloat, but it would be explicit and intuitive to use, and would allow a pretty interesting extension of iteration with the for-loop.
The idea is that `continue with` would allow injection of any datatype or variable, which becomes the next iterated item. So, saying `continue with 5` means that the next *x* in a loop structured as `for x in..` would be 5.
This would satisfy my original, niche-y desire to re-iterate over something, but would also more broadly allow dynamic injection of *any* value into a for-loop iteration.
As an extension to the language, it would present new and exciting ways to iterate infinitely by accident. It would also present new and exciting ways for people to trip up on mutable data; one way to misuse this is to iterate over data, modifying the data, then iterating over it again. The intention, rather, is that the repeated iteration allows re-iteration over unaltered data in response to a mid-iteration change in state (i.e., in my case, parsing a token, changing parser state, and then parsing the token again because it also carries informational content).
However, bad data hygiene has always been part of Python, because it's a natural consequence of the pan-mutability that makes it such a great language. So, given that the same practices that an experienced Python dev learns to use would apply in this case, I don't think it adds anything that an experienced dev would trip over.
As a language extension, I've never seen something like "continue with" in another language. You can mimic it with recursive functions and `cons`-ing the injection, but as a flow-control system for an imperative language, nope..and Python doesn't encourage or facilitate recursion anyway (no TCE).
I'd love thoughts on this. It's unnecessary; but then, once something's declared turing complete you can accuse any additional feature of being unnecessary. I feel, more to the point, that it's in keeping with the Python philosophy and would add a novel twist to the language not seen elsewhere.
best, Cathal
-- Twitter: @onetruecathal, @formabiolabs Phone: +353876363185 Blog: http://indiebiotech.com miniLock.io: JjmYYngs7akLZUjkvFkuYdsZ3PyPHSZRBKNm6qTYKZfAM
Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On 25 September 2014 08:09, Cathal Garvey cathalgarvey@cathalgarvey.me wrote:
The idea is that `continue with` would allow injection of any datatype or variable, which becomes the next iterated item. So, saying `continue with 5` means that the next *x* in a loop structured as `for x in..` would be 5.
You can effectively do this just with generators.
def process(iterable): for e in iterable: yield e yield -e
for e in process([1, 2, 3]): print(e)
If you need to get more complex, investigate the send() method of generators.
Tim Delaney
On Sep 24, 2014, at 15:09, Cathal Garvey cathalgarvey@cathalgarvey.me wrote:
Hello all, First time post, so go gentle. :)
During a twitter conversation with @simeonvisser, an idea emerged that he suggested I share here.
The conversation began with me complaining that I'd like a third mode of explicit flow control in Python for-loops; the ability to repeat a loop iteration in whole. The reason for this was that I was parsing data where a datapoint indicated the end of a preceding sub-group, so the datapoint was both a data structure indicator *and* data in its own right. So I'd like to have iterated over the line once, branching into the flow control management part of an if/else, and then again to branch into the data management part of the same if/else.
Yes, this is unnecessary and just a convenience for parsing that I'd like to see.
It would really help to have specific use cases, so we can look at how much the syntactic sugar helps readability vs. what we can write today. Otherwise all anyone can say is, "Well, it sounds like it might be nice, but I can't tell if it would be nice enough to be worth a language change", or try to invent their own use cases that might not be as nice as yours and then unfairly dismiss it as unnecessary.
During the discussion though, a much more versatile solution presented itself: repurposing the `continue` keyword to `continue with`, similarly to the repurposing of `yield` with `yield from`. This would avoid keyword bloat, but it would be explicit and intuitive to use, and would allow a pretty interesting extension of iteration with the for-loop.
The idea is that `continue with` would allow injection of any datatype or variable, which becomes the next iterated item. So, saying `continue with 5` means that the next *x* in a loop structured as `for x in..` would be 5.
This would satisfy my original, niche-y desire to re-iterate over something, but would also more broadly allow dynamic injection of *any* value into a for-loop iteration.
As an extension to the language, it would present new and exciting ways to iterate infinitely by accident. It would also present new and exciting ways for people to trip up on mutable data; one way to misuse this is to iterate over data, modifying the data, then iterating over it again. The intention, rather, is that the repeated iteration allows re-iteration over unaltered data in response to a mid-iteration change in state (i.e., in my case, parsing a token, changing parser state, and then parsing the token again because it also carries informational content).
However, bad data hygiene has always been part of Python, because it's a natural consequence of the pan-mutability that makes it such a great language. So, given that the same practices that an experienced Python dev learns to use would apply in this case, I don't think it adds anything that an experienced dev would trip over.
As a language extension, I've never seen something like "continue with" in another language. You can mimic it with recursive functions and `cons`-ing the injection, but as a flow-control system for an imperative language, nope..and Python doesn't encourage or facilitate recursion anyway (no TCE).
I'd love thoughts on this. It's unnecessary; but then, once something's declared turing complete you can accuse any additional feature of being unnecessary. I feel, more to the point, that it's in keeping with the Python philosophy and would add a novel twist to the language not seen elsewhere.
best, Cathal
-- Twitter: @onetruecathal, @formabiolabs Phone: +353876363185 Blog: http://indiebiotech.com miniLock.io: JjmYYngs7akLZUjkvFkuYdsZ3PyPHSZRBKNm6qTYKZfAM <0x988B9099.asc> _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On 25 Sep 2014 02:09, "Andrew Barnert" abarnert@yahoo.com.dmarc.invalid wrote:
On Sep 24, 2014, at 15:09, Cathal Garvey cathalgarvey@cathalgarvey.me
wrote:
The conversation began with me complaining that I'd like a third mode of explicit flow control in Python for-loops; the ability to repeat a loop iteration in whole. The reason for this was that I was parsing data where a datapoint indicated the end of a preceding sub-group, so the datapoint was both a data structure indicator *and* data in its own right. So I'd like to have iterated over the line once, branching into the flow control management part of an if/else, and then again to branch into the data management part of the same if/else.
Yes, this is unnecessary and just a convenience for parsing that I'd like to see.
It would really help to have specific use cases, so we can look at how
much the syntactic sugar helps readability vs. what we can write today. Otherwise all anyone can say is, "Well, it sounds like it might be nice, but I can't tell if it would be nice enough to be worth a language change", or try to invent their own use cases that might not be as nice as yours and then unfairly dismiss it as unnecessary.
The way I would describe this is, the proposal is to add single-item pushback support to all for loops. Tokenizers are a common case that needs pushback ("if we are in the IDENTIFIER state and the next character is not alphanumeric, then set state to NEW_TOKEN and process it again").
I don't know how common such cases are in the grand scheme of things, but they are somewhat cumbersome to handle when they come up.
The most elegant solution I know is:
class PushbackAdaptor: def __init__(self, iterable): self.base = iter(iterable) self.stack = []
def next(self): if self.stack: return self.stack.pop() else: return self.base.next()
def pushback(self, obj): self.stack.append(obj)
it = iter(character_source) for char in it: ... if state is IDENTIFIER and char not in IDENT_CHARS: state = NEW_TOKEN it.push_back(char) continue ...
In modern python, I think the natural meaning for 'continue with' wouldn't be to special-case something like this. Instead, where 'continue' triggers a call to 'it.next()', I'd expect 'continue with x' to trigger a call to 'it.send(x)'. I suspect this might enable some nice idioms in coroutiney code, though I'm not very familiar with such.
-n
On Thu, Sep 25, 2014 at 2:50 AM, Nathaniel Smith njs@pobox.com wrote:
The most elegant solution I know is:
class PushbackAdaptor: def __init__(self, iterable): self.base = iter(iterable) self.stack = []
def next(self): if self.stack: return self.stack.pop() else: return self.base.next() def pushback(self, obj): self.stack.append(obj)it = iter(character_source) for char in it: ... if state is IDENTIFIER and char not in IDENT_CHARS: state = NEW_TOKEN it.push_back(char) continue ...
In modern python, I think the natural meaning for 'continue with' wouldn't be to special-case something like this. Instead, where 'continue' triggers a call to 'it.next()', I'd expect 'continue with x' to trigger a call to 'it.send(x)'. I suspect this might enable some nice idioms in coroutiney code, though I'm not very familiar with such.
In fact, given the 'send' definition of 'continue with x', the above tokenization code would become simply:
def redoable(iterable): for obj in iterable: while yield obj == "redo": pass
for char in redoable(character_source): ... if state is IDENTIFIER and char not in IDENT_CHARS: state = NEW_TOKEN continue with "redo" ...
which I have to admit is fairly sexy.
I appreciate the comparison to coroutines, it helps frame some of the use-cases. However (forgive me for saying), I find that Python's coroutine model isn't intuitive, certainly for newcomers. I often find it hard to envisage use-cases for Python coroutines that wouldn't be better served with a class, for the sake of readability; Python's overall philosophy leans toward readability, so that suggests it's leaning away from coroutines.
Now, that's an aside, so I don't want to fall off-topic; I also realise Python is drifting into more uniform and useful coroutine based code, based on asyncio. So, if `continue with` were useful to building better or more intuitive coroutines, then by all means that's a valid application.
What I had more in mind was to have it as something that would stand independent of the custom-class spaghetti normally required to implement "clean" alternative looping or coroutines. That is, everyone can accept that it's not necessary; but does it clean up real-world code?
So, for clarity, this is the kind of thing I had in mind, which I've encountered in various forms and which I think `continue with` would help with, at its most basic:
Ingredient (unit) (unit/100g): Protein g, 5 Carbohydrate g, 50 Fibre g 10 Insoluble Fibre g 5 Soluble Fibre g 5 Starches g 20 Sugars g 20 Sucrose g 10 Glucose g 5 Fructose g 5 Vitamins mg 100 Ascorbic Acid mg 50 Niacin mg 50
The above is invented, but I was actually parsing an ingredient/nutrition list when the idea occurred to me. As you can see, there are "totals" followed by sub-categories, some of which are subtotals which form their own category. When parsed, I might want (pseudo-json):
{ Protein: 5g Carbohydrates: { total: 50g fibre: { total: 10g, soluble: 5 insoluble: 5 } <...> }
To parse this, I create code like this, with some obviously-named functions that aren't given. With recursive subtables, obviously this isn't going to work as-is, but it illustrates the point:
``` table = {} subtable = '' for line in raw_table: name, unit, quant = line.strip().rsplit(None, 2) if subtable: if is_valid_subtable_element(subtable, name): table[subtable][name] = quant + unit else: subtable = '' table[name] = quant + unit # DRY! else: if is_subtable_leader(name): subtable = name table[subtable] = {'total': quant_unit} else: table[name] = quant + unit # DRY! ```
Now, if I have to maintain this code, which will quickly become nontrivial for enough branches, I have several locations that need parallel fixes and modifications.
One solution is to functionalise this and build functions to which the container (table) and the tokens are passed; changes are then made in the functions, and the repeated calls in different code branches become more maintainable. Another is to make an object instead of a dict-table, and the object performs some magic to handle things correctly.
However, with `continue with` the solution is more straightforward. Because the problem, essentially, is that some tokens cause a state-change in addition to presenting data in their own right, by using `continue with` you can handle them in one code branch first, then repeat to handle them in the other branch:
``` table = {} subtable = '' for line in raw_table.splitlines(): name, unit, quant = line.rsplit(None, 2) if subtable: if is_valid_subtable_element(subtable, name): table[subtable][name] = quant + unit else: subtable = '' continue with line else: if is_subtable_leader(name): subtable = name table[subtable] = {} continue with 'total {} {}'.format(quant, unit) else: table[name] = quant + unit ```
The result is a single table entry per branch; one for subtables, one for base table. The handling of tokens that present flow-control issues, like titles of subtables or values that indicate the subtable should end (like a vitamin, when we were parsing carbohydrates), is handled first as flow-control issues and then again as data. (in this case, assume that the function is_valid_subtable_element accepts "total" as a valid subtable element always, and judges the rest according to predefined valid items for categories like "carbohydrates", "fibres", "vitamins", etcetera).
The flow control is cleaner, more intuitive to read IMO, and there is less call for the definition of special flow-control classes, functions or coroutines. In my opinion, anything that removes the need for custom classes and functions *for the purpose of flow control and readability* is an improvement to the language.
Now, as indicated above, `continue with` does not merely repeat the current iteration; you can dynamically generate the next iteration cycle. In the above example, that changed a line like "Carbohydrates g 50" into "total g 50" for use in the subtable iteration. More creative uses of dynamic iterable injection will surely present themselves with further thought.
Sorry for the poor clarity last night, and perhaps today; I'm recovering from illness and distracted by various other things. :) Thanks for your feedback and thoughts!
Cathal
On 25/09/14 04:51, Nathaniel Smith wrote:
On Thu, Sep 25, 2014 at 2:50 AM, Nathaniel Smith njs@pobox.com wrote:
The most elegant solution I know is:
class PushbackAdaptor: def __init__(self, iterable): self.base = iter(iterable) self.stack = []
def next(self): if self.stack: return self.stack.pop() else: return self.base.next() def pushback(self, obj): self.stack.append(obj)it = iter(character_source) for char in it: ... if state is IDENTIFIER and char not in IDENT_CHARS: state = NEW_TOKEN it.push_back(char) continue ...
In modern python, I think the natural meaning for 'continue with' wouldn't be to special-case something like this. Instead, where 'continue' triggers a call to 'it.next()', I'd expect 'continue with x' to trigger a call to 'it.send(x)'. I suspect this might enable some nice idioms in coroutiney code, though I'm not very familiar with such.
In fact, given the 'send' definition of 'continue with x', the above tokenization code would become simply:
def redoable(iterable): for obj in iterable: while yield obj == "redo": pass
for char in redoable(character_source): ... if state is IDENTIFIER and char not in IDENT_CHARS: state = NEW_TOKEN continue with "redo" ...
which I have to admit is fairly sexy.
There are a couple minor errors in Nathaniels (presumably untested) code. But I think it looks quite elegant overall, actually:
#!/usr/bin/env python3 from string import ascii_lowercase from random import random
class PushbackAdaptor(object): def __init__(self, iterable): self.base = iter(iterable) self.stack = []
def __next__(self): if self.stack: return self.stack.pop() else: return next(self.base)
def pushback(self, obj): self.stack.append(obj)
def __iter__(self): return self
def repeat_some(it): it = PushbackAdaptor(it) for x in it: print(x, end='') if random() > 0.5: it.pushback(x) continue print()
repeat_some(ascii_lowercase) repeat_some(range(10))
On Wed, Sep 24, 2014 at 6:50 PM, Nathaniel Smith njs@pobox.com wrote:
On 25 Sep 2014 02:09, "Andrew Barnert" abarnert@yahoo.com.dmarc.invalid wrote:
On Sep 24, 2014, at 15:09, Cathal Garvey cathalgarvey@cathalgarvey.me
wrote:
The conversation began with me complaining that I'd like a third mode
of
explicit flow control in Python for-loops; the ability to repeat a loop iteration in whole. The reason for this was that I was parsing data where a datapoint indicated the end of a preceding sub-group, so the datapoint was both a data structure indicator *and* data in its own right. So I'd like to have iterated over the line once, branching into the flow control management part of an if/else, and then again to
branch
into the data management part of the same if/else.
Yes, this is unnecessary and just a convenience for parsing that I'd like to see.
It would really help to have specific use cases, so we can look at how
much the syntactic sugar helps readability vs. what we can write today. Otherwise all anyone can say is, "Well, it sounds like it might be nice, but I can't tell if it would be nice enough to be worth a language change", or try to invent their own use cases that might not be as nice as yours and then unfairly dismiss it as unnecessary.
The way I would describe this is, the proposal is to add single-item pushback support to all for loops. Tokenizers are a common case that needs pushback ("if we are in the IDENTIFIER state and the next character is not alphanumeric, then set state to NEW_TOKEN and process it again").
I don't know how common such cases are in the grand scheme of things, but they are somewhat cumbersome to handle when they come up.
The most elegant solution I know is:
class PushbackAdaptor: def __init__(self, iterable): self.base = iter(iterable) self.stack = []
def next(self): if self.stack: return self.stack.pop() else: return self.base.next() def pushback(self, obj): self.stack.append(obj)it = iter(character_source) for char in it: ... if state is IDENTIFIER and char not in IDENT_CHARS: state = NEW_TOKEN it.push_back(char) continue ...
In modern python, I think the natural meaning for 'continue with' wouldn't be to special-case something like this. Instead, where 'continue' triggers a call to 'it.next()', I'd expect 'continue with x' to trigger a call to 'it.send(x)'. I suspect this might enable some nice idioms in coroutiney code, though I'm not very familiar with such.
-n
Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On 24Sep2014 23:09, Cathal Garvey cathalgarvey@cathalgarvey.me wrote:
The conversation began with me complaining that I'd like a third mode of explicit flow control in Python for-loops; the ability to repeat a loop iteration in whole. The reason for this was that I was parsing data where a datapoint indicated the end of a preceding sub-group, so the datapoint was both a data structure indicator *and* data in its own right. So I'd like to have iterated over the line once, branching into the flow control management part of an if/else, and then again to branch into the data management part of the same if/else.
Sounds a bit like Perl's "redo" statement. Example (pretending Python syntax):
for x in iterable_thing: if special_circumstance_here: x = 9 redo
Perl lets you redo to a special label, too (good practice actually - all sorts of horrible subtle bugs can come in with bare redos).
In Python you'd be better off writing a special iterator with "push back". You used to find this kind of thing on I/O streams and parsers.
Example use:
I2 = PushableIter(iterable_thing) for x in I2: if special_circumstance_here: I2.push_back(9) else: rest of loop ...
Then "PushableIter" is a special iterator that keeps a "pushed back" variable or stack. Simple untested example:
class PushableIter:
def __init__(self, iterable): self.pushed = [] self.iterator = iter(iterable)
def __next__(self): if self.pushed: return self.pushed.pop() return next(self.iterator)
def push_back(self, value): self.pushed.append(value)
Should slot into the above example directly. Keep it around in your convenience library.
This doesn't require modifying the language to insert is special purpose and error prone contruct. Instead, you're just making a spceial iterator. Which also leaves you free to make _other_ special iterators for other needs.
Cheers, Cameron Simpson cs@zip.com.au
[...] post-block actions should be allowed everywhere, not just on subroutines. The ALWAYS keyword was agreed upon as a good way of doing this, although POST was also suggested. This lead to the semi-inevitable rehash of the try- catch exception handling debate. According to John Porter, "There is no try, there is only do. :-)" - from the perl6 development discussion
-
Andrew Barnert -
Cameron Simpson -
Cathal Garvey -
David Mertz -
Nathaniel Smith -
Tim Delaney