Iterable function calls and unpacking [was: PEP for issue2292, "Missing *-unpacking generalizations"]
I've moved this to a different thread, as I agree with Guido that it's
a different PEP.
On 15 July 2013 21:06, Guido van Rossum
On Mon, Jul 15, 2013 at 1:01 PM, Oscar Benjamin
wrote: On 15 July 2013 12:08, Joshua Landau
wrote: On 15 July 2013 11:40, Oscar Benjamin
wrote: In fact, I'd much like it if there was an iterable "unpacking" method for functions, too, so "chain.from_iterable()" could use the same interface as "chain" (and str.format with str.format_map, etc.). I feel we already have a good deal of redundancy due to this.
I've also considered this before. I don't know what a good spelling would be but lets say that it uses *args* so that you have a function signature like:
def chain(*iterables*): for iterable in iterables: yield from iterable
Personally some form of decorator would be simpler: @lazy_unpack() def chain(*iterables): ... (and it could theoretically work for mappings too, by just "bundling" them; useful for circumstances where the mapping is left untouched and just passed to the next function in line.)
And then if the function is called with
for line in chain(first_line, *inputfile): # do stuff
then iterables would be bound to a lazy generator that chains [first_line] and inputfile. Then you could create the unpacking iterator I wanted by just using chain e.g.:
chain(prepend, *iterable, append)
But how could you do this without generating different code depending on how the function you are calling is declared? Python's compiler doesn't have access to that information.
You could simply make the code such that if has an unpack inside the call it does a run-time check. Whilst this will be slower for the false-positives, the number of times *args is pass-through (and thus you save a redundant copy of the argument tuple) and *args is a simple loop-once construct makes it plausible that those losses would be outweighed. It doesn't even reduce efficiency that much, too, as the worst case scenario is immediately falling back after checking a single C-level attribute of the function, and the function doesn't need to be fetched again or anything suchlike. Then again, I'm guessing. You'd also need to add a call to exhaust the iterator at the end of every function utilising this (transparently, probably) to make this have no obvious externally-visible effects. There would still be a call-order change, but that's much more minor.
On 15 July 2013 21:32, Joshua Landau
I've moved this to a different thread, as I agree with Guido that it's a different PEP.
On 15 July 2013 21:06, Guido van Rossum
wrote: On Mon, Jul 15, 2013 at 1:01 PM, Oscar Benjamin
wrote: On 15 July 2013 12:08, Joshua Landau
wrote: On 15 July 2013 11:40, Oscar Benjamin
wrote: In fact, I'd much like it if there was an iterable "unpacking" method for functions, too, so "chain.from_iterable()" could use the same interface as "chain" (and str.format with str.format_map, etc.). I feel we already have a good deal of redundancy due to this.
I've also considered this before. I don't know what a good spelling would be but lets say that it uses *args* so that you have a function signature like:
def chain(*iterables*): for iterable in iterables: yield from iterable
Personally some form of decorator would be simpler:
@lazy_unpack() def chain(*iterables): ...
How would the above decorator work? It would need to exploit some new capability since this requires unpacking everything: def lazy_unpack(func): @wraps(func) def wrapper(*args **kwargs): # The line above has already expanded *args return func(*args, **kwargs) return wrapper @lazy_unpack def chain(*iterables): ...
(and it could theoretically work for mappings too, by just "bundling" them; useful for circumstances where the mapping is left untouched and just passed to the next function in line.)
I don't understand. Do you mean to use it somehow for **kwargs?
And then if the function is called with
for line in chain(first_line, *inputfile): # do stuff
then iterables would be bound to a lazy generator that chains [first_line] and inputfile. Then you could create the unpacking iterator I wanted by just using chain e.g.:
chain(prepend, *iterable, append)
But how could you do this without generating different code depending on how the function you are calling is declared? Python's compiler doesn't have access to that information.
You could simply make the code such that if has an unpack inside the call it does a run-time check. Whilst this will be slower for the false-positives, the number of times *args is pass-through (and thus you save a redundant copy of the argument tuple) and *args is a simple loop-once construct makes it plausible that those losses would be outweighed.
It probably would be better to have a specific syntax at the calling site since you probably want to know when you look at f(*infinite_iterator) whether or not infinite_iterator is going to be expanded. Oscar
On 15 July 2013 21:43, Oscar Benjamin
On 15 July 2013 21:32, Joshua Landau
wrote: I've moved this to a different thread, as I agree with Guido that it's a different PEP.
On 15 July 2013 21:06, Guido van Rossum
wrote: On Mon, Jul 15, 2013 at 1:01 PM, Oscar Benjamin
wrote: On 15 July 2013 12:08, Joshua Landau
wrote: On 15 July 2013 11:40, Oscar Benjamin
wrote: In fact, I'd much like it if there was an iterable "unpacking" method for functions, too, so "chain.from_iterable()" could use the same interface as "chain" (and str.format with str.format_map, etc.). I feel we already have a good deal of redundancy due to this.
I've also considered this before. I don't know what a good spelling would be but lets say that it uses *args* so that you have a function signature like:
def chain(*iterables*): for iterable in iterables: yield from iterable
Personally some form of decorator would be simpler:
@lazy_unpack() def chain(*iterables): ...
How would the above decorator work? It would need to exploit some new capability since this requires unpacking everything:
Yeah, it would just set an attribute on the function that tells Python to special-case it. It's new functionality, just without new syntax. My way also makes it so you can change old-style unpackers into new-style iter-packers by doing "lazy_version = lazy_unpack(original)".
(and it could theoretically work for mappings too, by just "bundling" them; useful for circumstances where the mapping is left untouched and just passed to the next function in line.)
I don't understand. Do you mean to use it somehow for **kwargs?
Yup. A "lazy_kwargs" version that lets you do nothing more than pass it along or convert to dict. In fact, for str.format you'd want a "frozen non-copy" that lets you access elements of the original dicts and kwargs without changing them too. Say you have: def foo(*args, **kwargs): return bar(*modified_args, **kwargs, possibly=more_keywords) there's little point in copying kwargs twice, is there? Same idea with: "{foo}".format(**very_many_things)
And then if the function is called with
for line in chain(first_line, *inputfile): # do stuff
But how could you do this without generating different code depending on how the function you are calling is declared? Python's compiler doesn't have access to that information.
You could simply make the code such that if has an unpack inside the call it does a run-time check. Whilst this will be slower for the false-positives, the number of times *args is pass-through (and thus you save a redundant copy of the argument tuple) and *args is a simple loop-once construct makes it plausible that those losses would be outweighed.
It probably would be better to have a specific syntax at the calling site since you probably want to know when you look at f(*infinite_iterator) whether or not infinite_iterator is going to be expanded.
True. "*args*" and "**kwargs**" are actually quite reasonable; *args* for chain.from_iterable and **kwargs** for collections.ChainMap. Some of the silly things you could do: (item for item in *iter_a*, *iter_b*) == itertools.chain(iter_a, iter_b) {default: foo, **mydict**}[default] === mydict.get({default: foo, **mydict**}) === mydict.get(default, foo) "{SPARKLE}{HOUSE}{TANABATA TREE}".format(**unicode**) === "{SPARKLE}{HOUSE}{TANABATA TREE}".format_map(unicode) print("Hey, look!", *lines*, sep="\n") === print("Hey, look!", "\n".join(lines), sep="\n") next((*iterable*, default)) == next(iterable, default)¹ Next PEP anyone? :D ¹ What's the point? Well, we wouldn't have *needed* the default argument if it was that easy in the first place. Same with dict.get. I also have changed my mind where I said:
You'd also need to add a call to exhaust the iterator at the end of every function utilising this (transparently, probably) to make this have no obvious externally-visible effects. There would still be a call-order change, but that's much more minor.
because I obviously wasn't thinking when I said it.
participants (2)
-
Joshua Landau
-
Oscar Benjamin