Addition to operator module: starcaller
Hello, I have run into a use case where I want to create a wrapper object that will star-unpack a sequence of arguments to pass to a function. Currently one way to construct this is to create a function: def starcaller(f): def wrapper(args): return f(*args) return wrapper Such a function feels simple enough, and specific enough to the language, that it would fit in well in the operator module of the standard library. We already have a similar representation of this functionality in itertools.starmap. Whereas the nested function implementation above will produce unpickleable objects (due to the closure), a straightforward c implementation will produce pickleable ones, making them useful for parallel applications. Thanks, Dan Spitz
Hello, You’re presumably doing something like ‘star = starcaller(f); star((“foo”, “bar”, “baz”))’ – how is it different from ‘f(*(“foo”, “bar”, “baz”))’ ? I don’t see any difference… -Emanuel From: Python-ideas [mailto:python-ideas-bounces+vgr255=live.ca@python.org] On Behalf Of Daniel Spitz Sent: Thursday, July 21, 2016 10:07 AM To: python-ideas@python.org Subject: [Python-ideas] Addition to operator module: starcaller Hello, I have run into a use case where I want to create a wrapper object that will star-unpack a sequence of arguments to pass to a function. Currently one way to construct this is to create a function: def starcaller(f): def wrapper(args): return f(*args) return wrapper Such a function feels simple enough, and specific enough to the language, that it would fit in well in the operator module of the standard library. We already have a similar representation of this functionality in itertools.starmap. Whereas the nested function implementation above will produce unpickleable objects (due to the closure), a straightforward c implementation will produce pickleable ones, making them useful for parallel applications. Thanks, Dan Spitz
On 22 July 2016 at 00:19, Emanuel Barry <vgr255@live.ca> wrote:
Hello,
You’re presumably doing something like ‘star = starcaller(f); star((“foo”, “bar”, “baz”))’ – how is it different from ‘f(*(“foo”, “bar”, “baz”))’ ? I don’t see any difference…
Similar to functools.partial, it lets you more easily separate the process of changing a function's signature from actually calling it. Compare: def f(a, b): print(a, b) accepts_only_b = functools.partial(f, "predefined_a") accepts_args_as_tuple = operator.starcaller(f)
From the point of view of subsequent code, "args_as_tuple" now works as a callable that accepts a 2-tuple and prints the elements.
However, the similarity to partial does make me wonder whether we might be better off resurrecting the old apply() builtin as a functools function: https://docs.python.org/2/library/functions.html#apply Then the requested operation would just be: accepts_args_as_tuple = functools.partial(functools.apply, f) (with the added bonus of also accepting an optional keyword dict) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
From: Nick Coghlan Sent: Thursday, July 21, 2016 11:21 AM
On 22 July 2016 at 00:19, Emanuel Barry <vgr255@live.ca> wrote:
Hello,
You’re presumably doing something like ‘star = starcaller(f); star((“foo”, “bar”, “baz”))’ – how is it different from ‘f(*(“foo”, “bar”, “baz”))’ ? I don’t see any difference…
Similar to functools.partial, it lets you more easily separate the process of changing a function's signature from actually calling it.
Compare:
def f(a, b): print(a, b)
accepts_only_b = functools.partial(f, "predefined_a") accepts_args_as_tuple = operator.starcaller(f)
Ah, of course - I didn't think of the multiple arguments case, thanks :) -Emanuel
Well, partial + apply seems like the most powerful solution (supports **keywords, opens up the possibility of using apply in other ways). Perhaps adding apply back to functools rather than as a builtin would evade some of the antipatterns it used to encourage (a la reduce). I'm not sure and don't have a strong opinion about the original removal of apply. Alternatively a form of "starcaller" or "unpackcaller" defined roughly as partial(apply) could be added to the operator module in the capacity I proposed above. The only possible advantage here is aesthetic; it fits conceptually with the rest of the module as a callable capturing a repeatable operation that is normally only possible with special syntax. It also disallows using apply in "non-curried" contexts. Thanks, Dan PS. Emanuel, my use case is one where I am creating data transformations by composing chains of function calls. The functions are executed in parallel across multiple processes. Specifically I'm using the Dask Distributed <https://distributed.readthedocs.io/en/latest/> library. On Thu, Jul 21, 2016 at 11:51 AM Emanuel Barry <vgr255@live.ca> wrote:
From: Nick Coghlan Sent: Thursday, July 21, 2016 11:21 AM
On 22 July 2016 at 00:19, Emanuel Barry <vgr255@live.ca> wrote:
Hello,
You’re presumably doing something like ‘star = starcaller(f); star((“foo”, “bar”, “baz”))’ – how is it different from ‘f(*(“foo”, “bar”, “baz”))’ ? I don’t see any difference…
Similar to functools.partial, it lets you more easily separate the process of changing a function's signature from actually calling it.
Compare:
def f(a, b): print(a, b)
accepts_only_b = functools.partial(f, "predefined_a") accepts_args_as_tuple = operator.starcaller(f)
Ah, of course - I didn't think of the multiple arguments case, thanks :)
-Emanuel
On Thu, Jul 21, 2016 at 12:37 PM Daniel Spitz <spitz.dan.l@gmail.com> wrote:
Alternatively a form of "starcaller" or "unpackcaller" defined roughly as partial(apply) could be added to the operator module in the capacity I proposed above. The only possible advantage here is aesthetic; it fits conceptually with the rest of the module as a callable capturing a repeatable operation that is normally only possible with special syntax. It also disallows using apply in "non-curried" contexts.
It does seem to fit with the theme of the operator module. One major use of the operator module is providing an alternative to lambdas. In this case, the clear alternative is ``g = lambda args, kwargs: f(*args, **kwargs)``
Daniel Spitz wrote:
I'm not sure and don't have a strong opinion about the original removal of apply.
I expect it was removed simply because the * and ** calling syntax makes it unecessary (before that was introduced, apply was the only way of getting that funcionality). If it's to be reintroduced, the operator module would seem to be the right place for it. -- Greg
On 22 July 2016 at 09:45, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Daniel Spitz wrote:
I'm not sure and don't have a strong opinion about the original removal of apply.
I expect it was removed simply because the * and ** calling syntax makes it unecessary (before that was introduced, apply was the only way of getting that funcionality).
If it's to be reintroduced, the operator module would seem to be the right place for it.
I guess folks doing higher order functions are already used to importing the operator module for access to operator.add (etc), and this is a pretty thin wrapper around the __call__ protocol (unlike reduce(), which has no syntactic counterpart). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, Jul 22, 2016 at 12:52 AM Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Daniel Spitz wrote:
I'm not sure and don't have a strong opinion about the original removal of apply.
I expect it was removed simply because the * and ** calling syntax makes it unecessary (before that was introduced, apply was the only way of getting that funcionality).
If it's to be reintroduced, the operator module would seem to be the right place for it.
If it goes in the operator module, then the name ``operator.starcall`` seems better than ``starcaller``. I'd say ``operator.unpack`` would be better, except for the confusion between unpacking assignment and unpacking as arguments. I suppose ``operator.apply`` would be OK, too. Is there a better vocabulary word for unpack-as-args?
On 2016-07-29 19:59, Michael Selik wrote:
On Fri, Jul 22, 2016 at 12:52 AM Greg Ewing <greg.ewing@canterbury.ac.nz <mailto:greg.ewing@canterbury.ac.nz>> wrote:
Daniel Spitz wrote: > I'm > not sure and don't have a strong opinion about the original removal of > apply.
I expect it was removed simply because the * and ** calling syntax makes it unecessary (before that was introduced, apply was the only way of getting that funcionality).
If it's to be reintroduced, the operator module would seem to be the right place for it.
If it goes in the operator module, then the name ``operator.starcall`` seems better than ``starcaller``. I'd say ``operator.unpack`` would be better, except for the confusion between unpacking assignment and unpacking as arguments. I suppose ``operator.apply`` would be OK, too. Is there a better vocabulary word for unpack-as-args?
Why not just operator.call? I suppose actually operator.caller would be more consistent with the existing attrgetter and methodcaller? -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Jul 29, 2016 11:00 PM, "Michael Selik" <michael.selik@gmail.com> wrote:
If it goes in the operator module, then the name ``operator.starcall`` seems better than ``starcaller``. I'd say ``operator.unpack`` would be better, except for the confusion between unpacking assignment and unpacking as arguments. I suppose ``operator.apply`` would be OK, too. Is there a better vocabulary word for unpack-as-args?
Not sure if "splat" is better. `operator.starcall(f, *args, **kwargs)` isn't obvious, unless you know `itertools.starmap` and `map`. `operator.unpack(...)` can be obvious, and the obvious meanings for many are probably wrong. That's bad. "unpack[ed]call" is ugly, long, and three syllables. On Jul 30, 2016 1:57 AM, "Brendan Barnwell" <brenbarn@brenbarn.net> wrote:
Why not just operator.call? I suppose actually operator.caller would be more consistent with the existing attrgetter and methodcaller?
The desired meaning of `somename(x, y)` is `x(*y)`. `call(x,y)` looks like it should be equivalent to `x.__call__(y)`, which means `x(y)`. `attrgetter(x)(y)` means `getattr(y,x)` (in the simple case), and `methodcaller(m,x)(y)` means `getattr(y,m)(x)`, so `caller(x)(y)` might mean `y(x)`. `operator.thinger` is for reverse currying: `thingdoer(x)` returns a new callable, and `thingdoer(x)(y)` means `dothing(y,x)` (conceptually). (I'm not sure if the original proposal of a wrapper would be called `starrer(f)(args)` or `unstarrer(f)(args)`,and that bugs me.)
On 22 July 2016 at 02:36, Daniel Spitz <spitz.dan.l@gmail.com> wrote:
Well, partial + apply seems like the most powerful solution (supports **keywords, opens up the possibility of using apply in other ways). Perhaps adding apply back to functools rather than as a builtin would evade some of the antipatterns it used to encourage (a la reduce). I'm not sure and don't have a strong opinion about the original removal of apply.
Alternatively a form of "starcaller" or "unpackcaller" defined roughly as partial(apply) could be added to the operator module in the capacity I proposed above. The only possible advantage here is aesthetic; it fits conceptually with the rest of the module as a callable capturing a repeatable operation that is normally only possible with special syntax. It also disallows using apply in "non-curried" contexts.
apply was removed primarily for reasons of redundancy rather than there being anything particularly objectionable about the function itself, so I wouldn't anticipate any major objections to giving it the reduce treatment. The answer to "Which approach should I use when?" is pretty straightforward: "use the builtin syntax in most situations, but use functools.apply instead if you specifically need a higher-order function". Where I've personally seen that need come up is in distributed task processing where you normally want the actual computation to happen *somewhere else* (e.g. on another machine, in another process or thread, or even as a callback within the current thread), but the mechanism for actually invoking the callback is itself a callback that accepts a "(callable, args, kwds)" triple. "functools.apply" then slots in neatly as a lowest common denominator "just call it immediately" baseline implementation. There's also an interesting potential synergy with PEP 484 type annotations when it comes to manipulating higher order functions: at the moment, there's no way to explicitly type callables that accept keyword arguments, but you *can* type a callable that accepts a tuple and a dict with particular contents as positional arguments. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (7)
-
Brendan Barnwell -
Daniel Spitz -
Emanuel Barry -
Franklin? Lee -
Greg Ewing -
Michael Selik -
Nick Coghlan