Add a "partial with placeholders" function to the functools module
Hello, currently, regarding positional arguments, `partial` gives us the option to partialize functions from the left. There's been some interest about partializing functions from the right instead (e.g. [SO post, 9k views, 39 upvotes](https://stackoverflow.com/q/7811247/3767239)), especially w.r.t. the various `str` methods. I propose adding a function to `functools` that works with placeholders and thus offers even greater flexibility. The Ellipsis literal `...` seems a intuitive choice for that task. When eventually calling such a "partial placeholder" object, it would fill in placeholders from the left and add remaining `args` to the right. In terms of implementation this can be realized as a subclass of `partial` itself. ## Implementation from functools import partial from reprlib import recursive_repr class partial_placehold(partial): placeholder = Ellipsis def __call__(self, /, *args, **keywords): args = iter(args) try: old_args = [x if x is not self.placeholder else next(args) for x in self.args] except StopIteration: raise TypeError('too few arguments were supplied') from None keywords = {**self.keywords, **keywords} return self.func(*old_args, *args, **keywords) @recursive_repr() def __repr__(self): qualname = type(self).__qualname__ args = [repr(self.func)] args.extend(repr(x) if x is not self.placeholder else '...' for x in self.args) # Only this line deviates from `partial.__repr__`; could also factor that out into a separate method. args.extend(f"{k}={v!r}" for (k, v) in self.keywords.items()) if type(self).__module__ == "functools": return f"functools.{qualname}({', '.join(args)})" return f"{qualname}({', '.join(args)})" # Would need to add something for compatibility with `partial`, i.e. for partializing a placeholder function. ## Example This allows for example the following usage: replace_dots_with_underscore = partial_placehold(str.replace, ..., '.', '_') replace_dots_with_underscore('foo.bar.baz') ## Relevance Sure we could also use a `lambda` instead ([as discussed here](https://mail.python.org/archives/list/python-ideas@python.org/message/YD5OQE...)) but there was a reason `partial` was introduced and I think the same arguments apply here too. Though most functions allow partializing via keyword arguments and this is undoubtedly a cleaner way, some might not and for example built-ins' methods won't allow it. Especially Python 3.8's introduction of positional-only parameters (PEP 570) might give rise to cases where `partial` is not sufficient. In case inspection is desired a `lambda` does not provide much information (sure you could always dig deeper with `inspect` for example but that's not the point). Consider the following example of a pre-defined sequence of default postprocessing steps and the user might add their own or remove existing ones, as appropriate: postprocessing_steps = [ lambda s: s.replace('foo', 'bar'), ] print(postprocessing_steps[0]) # <function <lambda> at 0x7f94a850dd30> This doesn't give a lot of information about what the lambda actually does (and thus whether the user should remove it or not). Using the `partial_placehold` instead, it's clear what is happening: postprocessing_steps = [ partial_placehold(str.replace, ..., 'foo', 'bar'), ] print(postprocessing_steps[0]) # partial_placehold(<method 'replace' of 'str' objects>, ..., 'foo', 'bar') ## Compatibility The proposed solution works with the current syntax and the usage of Ellipsis as a placeholder object is likely not to collide with actually used values (in any case the user might still reassign the `.placeholder` attribute). Because the direction of partializing is unchanged (still left to right) this doesn't introduce ambiguities which might come with a "right partial" function. Creating a placeholder function from a `partial` object is possible without any changes, the opposite way requires an additional check to result in a placeholder object again. ## Possible confusion Regarding the usage of Ellipsis right now, in `numpy` or `typing` for example, it always represents a placeholder for multiple "things", not a single one: array[..., None] # All the dimensions of `array` plus a new one. typing.Tuple[str, ...] # Any number of str objects. So the expectations might be biased in that sense. For example: def foo(a, b, c, d): pass p_foo = partial_placehold(foo, ..., 1, 2) p_foo(3, 4) Someone else reviewing the code might now assume that the `...` means to act as a placeholder for all arguments except the last two (and hence `p_foo(3, 4)` would be equivalent to `foo(3, 4, 1, 2)` while it actually is equivalent to `foo(3, 1, 2, 4)`). But this would be again some kind of "right partial" function and also the function name implies something else; documentation might clarify as well, of course. ## Conclusion Adding a "partial with placeholders" function to `functools` allows for covering use cases where the standard `partial` is not sufficient. No new syntax is required and the implementation is fairly straightforward given the inheritance from `partial`. Ellipsis `...` seems an intuitive choice for acting as a placeholder (concerning both, conflicts with actual partial values and code readability). There are uses cases where such a function would provide a clean solution and there is an interest in the community (https://stackoverflow.com/q/7811247/3767239, https://stackoverflow.com/q/19701775/3767239 for example). Especially with the introduction of positional-only parameters new use cases are likely to arise. ----- **Related threads:** * https://mail.python.org/archives/list/python-ideas@python.org/message/TVNCM7... - Mentions essentially a similar idea. The original [PEP 309 -- Partial Function Application](https://www.python.org/dev/peps/pep-0309/) also mentions:
Partially applying arguments from the right, or inserting arguments at arbitrary positions creates its own problems, but pending discovery of a good implementation and non-confusing semantics, I don't think it should be ruled out.
On 27 Jul 2019, at 14:47, Dominik Vilsmeier <dominik.vilsmeier@gmx.de> wrote:
Hello,
currently, regarding positional arguments, `partial` gives us the option to partialize functions from the left. There's been some interest about partializing functions from the right instead (e.g. [SO post, 9k views, 39 upvotes](https://stackoverflow.com/q/7811247/3767239)), especially w.r.t. the various `str` methods.
Do you have other examples? That (and most likely similar) examples are just that the standard library contains methods and functions that could be fixed to accept keyword arguments. This would be less confusing and more coherent. / Anders
On Jul 27, 2019, at 08:04, Anders Hovmöller <boxed@killingar.net> wrote:
On 27 Jul 2019, at 14:47, Dominik Vilsmeier <dominik.vilsmeier@gmx.de> wrote:
currently, regarding positional arguments, `partial` gives us the option to partialize functions from the left. There's been some interest about partializing functions from the right instead (e.g. [SO post, 9k views, 39 upvotes](https://stackoverflow.com/q/7811247/3767239)), especially w.r.t. the various `str` methods.
Do you have other examples? That (and most likely similar) examples are just that the standard library contains methods and functions that could be fixed to accept keyword arguments. This would be less confusing and more coherent.
Many of the compelling examples for PEP 570 are good examples here. The following are all impossible intentionally, and for different reasons: skipper = partial(range, step=n) powbase = partial(pow, mod=base) clscheck = partial(isinstance, class_or_tuple=cls) Plus, there’s also one very general example: anything that semantically has to take *args can’t be changed to use keywords: partial(format, 2=x) partial(map, 1=x, 2=y) partial(executor.submit, 1=arg) And similarly for itertools.product, min/max, and everything that acts as a proxy like submit. Also, even for cases like the OP’s where there’s no semantics reason the argument couldn’t have a keyword, there may still be other reasons. When argclinic was added in PEP 436, it was specifically argued that it shouldn’t be used as an opportunity to phase out positional-only params, in part because for many functions the performance cost of keyword processing outweighs the uncommon potential use of the keywords; IIRC, the OP’s specific method was even one of the core examples. And, as PEP 570 points out, METH_FASTCALL makes that potentially true for pure Python functions as well, to the extent that some other stdlib functions might actually want to lose their keyword args. All that being said, this proposal seems like something that could easily be put on PyPI to see how much uptake it gets, instead of put immediately into the stdlib. I think people will find it useful. But maybe, say, funcy/toolz/etc. will borrow the idea and it’ll turn out that almost everyone who wants it already wants one of those libs anyway. Or maybe the bikeshedding potential will be higher than expected, and a variety of different modules that all handle the placeholders differently will compete, and it’ll turn out that everyone loves some different design that allows both single-arg and multi-arg placeholders
On 27 Jul 2019, at 21:10, Andrew Barnert <abarnert@yahoo.com> wrote:
On Jul 27, 2019, at 08:04, Anders Hovmöller <boxed@killingar.net> wrote:
On 27 Jul 2019, at 14:47, Dominik Vilsmeier <dominik.vilsmeier@gmx.de> wrote:
currently, regarding positional arguments, `partial` gives us the option to partialize functions from the left. There's been some interest about partializing functions from the right instead (e.g. [SO post, 9k views, 39 upvotes](https://stackoverflow.com/q/7811247/3767239)), especially w.r.t. the various `str` methods.
Do you have other examples? That (and most likely similar) examples are just that the standard library contains methods and functions that could be fixed to accept keyword arguments. This would be less confusing and more coherent.
Many of the compelling examples for PEP 570 are good examples here. The following are all impossible intentionally, and for different reasons:
skipper = partial(range, step=n) powbase = partial(pow, mod=base) clscheck = partial(isinstance, class_or_tuple=cls)
Those seem like great cases for changing the standard library to use normal parameters! range and pow are both nicer with keyword arguments imo.
Plus, there’s also one very general example: anything that semantically has to take *args can’t be changed to use keywords:
partial(format, 2=x) partial(map, 1=x, 2=y) partial(executor.submit, 1=arg)
Ok now THAT is a compelling argument!
And similarly for itertools.product, min/max, and everything that acts as a proxy like submit.
Also, even for cases like the OP’s where there’s no semantics reason the argument couldn’t have a keyword, there may still be other reasons. When argclinic was added in PEP 436, it was specifically argued that it shouldn’t be used as an opportunity to phase out positional-only params, in part because for many functions the performance cost of keyword processing outweighs the uncommon potential use of the keywords; IIRC, the OP’s specific method was even one of the core examples. And, as PEP 570 points out, METH_FASTCALL makes that potentially true for pure Python functions as well, to the extent that some other stdlib functions might actually want to lose their keyword args.
Well that sounds pretty terrible to me. I’ve tried to write calls with keywords of many many functions in the standard library because it just isn’t really readable with positional :( / Anders
On Jul 27, 2019, at 12:47, Anders Hovmöller <boxed@killingar.net> wrote:
On 27 Jul 2019, at 21:10, Andrew Barnert <abarnert@yahoo.com> wrote:
Many of the compelling examples for PEP 570 are good examples here. The following are all impossible intentionally, and for different reasons:
skipper = partial(range, step=n) powbase = partial(pow, mod=base) clscheck = partial(isinstance, class_or_tuple=cls)
Those seem like great cases for changing the standard library to use normal parameters! range and pow are both nicer with keyword arguments imo.
Look at the signatures for range and isinstance: range takes an optional argument on the left (or, if you prefer, it’s an overload of two different signatures), which can’t even be expressed in Python syntax; isinstance takes a class or a tuple of classes in the same parameter, so there’s no good name for it. There are a few other functions just like range, and a bunch like isinstance, and some other classes of functions with different kinds of weird signatures (although pow isn’t one of them). You could argue that these are just bad designs, but better designs weren’t possible when they were added to Python. For example, anything designed to take 1 or many classes since 2.2 would just use *classes instead of a single argument, but isinstance-style functions predate 2.2, so there was no *args. Changing those existing functions would break a lot of existing code. And if that wasn’t acceptable for 2.2 or even 3.0, it’s probably never going to be acceptable.
Also, even for cases like the OP’s where there’s no semantics reason the argument couldn’t have a keyword, there may still be other reasons. When argclinic was added in PEP 436, it was specifically argued that it shouldn’t be used as an opportunity to phase out positional-only params, in part because for many functions the performance cost of keyword processing outweighs the uncommon potential use of the keywords; IIRC, the OP’s specific method was even one of the core examples. And, as PEP 570 points out, METH_FASTCALL makes that potentially true for pure Python functions as well, to the extent that some other stdlib functions might actually want to lose their keyword args.
Well that sounds pretty terrible to me. I’ve tried to write calls with keywords of many many functions in the standard library because it just isn’t really readable with positional :(
I think many of the functions in the stdlib do get it wrong, for legacy reasons (nobody thought about the tradeoff consciously until the argclinic discussion, and also it was a pain to add keywords in C before argclinic—and many of the functions you use most often date back even farther, to before keywords existing at all), but there is an actual tradeoff, so it shouldn’t be 100% of functions take keywords. The two PEPs both make the case for that. And nobody’s going to sweep through the stdlib evaluating every function. If you’ve got a group of functions that particularly annoys you, and can make the case that the arg parsing performance isn’t important for that function and the keywords are useful, you can file it on b.p.o. (ideally with a patch) and I’m sure someone will take a look. Meanwhile, there are going to be functions that take positional arguments, whether for semantic reasons, for performance reasons, or for pure legacy issues. As long as any of those cases exist, the OP’s proposal makes sense, at least to me.
On 2019-07-27 20:10, Andrew Barnert via Python-ideas wrote:
On Jul 27, 2019, at 08:04, Anders Hovmöller <boxed@killingar.net> wrote:
On 27 Jul 2019, at 14:47, Dominik Vilsmeier <dominik.vilsmeier@gmx.de> wrote:
currently, regarding positional arguments, `partial` gives us the option to partialize functions from the left. There's been some interest about partializing functions from the right instead (e.g. [SO post, 9k views, 39 upvotes](https://stackoverflow.com/q/7811247/3767239)), especially w.r.t. the various `str` methods.
Do you have other examples? That (and most likely similar) examples are just that the standard library contains methods and functions that could be fixed to accept keyword arguments. This would be less confusing and more coherent.
Many of the compelling examples for PEP 570 are good examples here. The following are all impossible intentionally, and for different reasons:
skipper = partial(range, step=n) powbase = partial(pow, mod=base) clscheck = partial(isinstance, class_or_tuple=cls)
Plus, there’s also one very general example: anything that semantically has to take *args can’t be changed to use keywords:
partial(format, 2=x) partial(map, 1=x, 2=y) partial(executor.submit, 1=arg)
And similarly for itertools.product, min/max, and everything that acts as a proxy like submit.
[snip] I was thinking that 'partial' is like a function definition, so why not have a variation on the function definition: def powbase(x, y) is pow(x, y, base) def clscheck(obj) is isinstance(obj, cls) It would capture the current reference of any name that's not passed via the parameter list, e.g. base and cls in the above examples.
Before we go too far down this route, let's consider whether these can't be solved with lambda rather than introducing new special cases to partial(). -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Indeed it is important to not only consider the potential use cases for such a placeholder function, but to consider the cases that go beyond of what a lambda can do. A lambda is always a good option if it is used as a "single-serving" function, i.e. one that is only relevant locally where it is coded (and not stored somewhere). If on the other hand the function is to be reused or users are expected to interact with it, more clarity is needed (e.g. a descriptive `repr`). Of course in such cases a developer could always go with a proper function definition and even supply it with a doc string; it's just that for simple cases this feels a bit heavy. In the end it would provide a concise solution for cases where (a) partializing via keyword is not possible (positional-only parameters now being official might give rise to new use cases as well) and (b) the function is not "single-service", i.e. it is retained in some way (where PEP 8 says "do not assign a lambda"). As an example say there's a 2D array (semantics like numpy) and we want to convert a flat index to 2D-index (numpy contains such functionality but let's say we're doing something more lightweight): a = array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) convert_flat_index = placeholder(divmod, ..., a.shape[1]) assert a[convert_flat_index(4)] == 5 assert a[convert_flat_index(8)] == 9 The other option would be a full-fledged function definition (which seems a bit heavy): def generate_index_converter(n): def convert_flat_index(x): return divmod(x, n) return convert_flat_index
On Sat, Jul 27, 2019 at 12:47:39PM -0000, Dominik Vilsmeier wrote:
I propose adding a function to `functools` that works with placeholders and thus offers even greater flexibility. The Ellipsis literal `...` seems a intuitive choice for that task. When eventually calling such a "partial placeholder" object, it would fill in placeholders from the left and add remaining `args` to the right. In terms of implementation this can be realized as a subclass of `partial` itself.
If you're going to add support for a placeholder, there is no need to create a new kind of partial function? Support for a placeholder is completely backwards compatible, if you use a brand new special sentinel that doesn't yet exist. # May want to give it a nicer repr, but this is the # minimal version that will work. SKIP = object() There's only one public function, ``partial``, regardless of whether the caller uses the sentinel or not: def function(a, b, c, d): pass # Existing use is unchanged: from functools import partial partial(function, 1, 2) # Fill in a, b. # New functionality is triggered by the use of the sentinel: from functools import partial, SKIP partial(function, SKIP, 1, SKIP, 2) # Fill in b, d. If you don't like the use of a named sentinel, we could still use Ellipsis, but that will break anyone's partial functions that happen to already use Ellipsis as a value. That will probably require a warning in 3.9 and the new functionality only gets added in 3.10.
Sure we could also use a `lambda` instead
I believe that partial is faster than lambda. [steve@ando cpython]$ ./python -m timeit \ -s "from functools import partial" \ -s "f = partial(lambda x,y: x+y, 1)" \ "f(100)" 100000 loops, best of 5: 1.98 usec per loop [steve@ando cpython]$ ./python -m timeit \ -s "g = lambda x,y: x+y" \ -s "f = lambda a: g(1, a)" \ "f(100)" 100000 loops, best of 5: 2.44 usec per loop To avoid the name lookup of "g", I tried this, but it was even slower: [steve@ando cpython]$ ./python -m timeit \ -s "f = lambda a: (lambda x,y: x+y)(1, a)" \ "f(100)" 100000 loops, best of 5: 3.4 usec per loop (Not surprising, as we've changed a name lookup into creating a new function object.) Obviously this example is simple enough that we don't need partial at all, but its just an illustration demonstrating that partial seems to have lower overhead than normal function objects. -- Steven
On Sat, Jul 27, 2019 at 9:28 PM Steven D'Aprano <steve@pearwood.info> wrote:
If you're going to add support for a placeholder, there is no need to create a new kind of partial function?
Exactly! This is just an enhanced functools.partial. An enhancement that has already been done in several 3rd party libraries, in fact. I like the ellipsis, which most 3rd parties use, but it could be configurable what the sentinel is. Maybe a default brand new `functools.SKIP` object, but with the docs explaining the way to use the ellipsis for those 99% of users for whom that wouldn't break anything. Someone posted an example implementation that used a configurable placeholder like this.
Sure we could also use a `lambda` instead I believe that partial is faster than lambda.
I don't really care about faster. I think the "partial with placeholder" is simply nicer to read for many cases. Sure this is a toy, but:
lastname = 'Mertz' greet_david = partial(print, ..., 'David', lastname) # In Python 3.9 greet_david('Verwelkoming') Verwelkoming David Mertz
vs.
greet_david = lambda greeting: print(greeting, "David", lastname) greet_david('Cześć') Cześć David Mertz
OK, that's a bad example because we are using it for the side effect too, but I think it illustrates how the more flexible partial() would look better than lambda. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
An extension of `partial` itself would indeed be a clean solution (+ backwards compatibility), my concern was just that when most people are still using it in the "traditional" way (for example partializing via keyword), that the additional checks for `SKIP` introduce an unnecessary overhead for those cases. However by checking if `args` contain any placeholder in `__new__` and storing that information in a separate variable it should be possible to keep it to a minimum. Regarding third-party implementations one aspect to consider is compatibility with `partial` itself, i.e. what should happen when further `partial`'izing a previously defined placeholder function (it would need to be a placeholder function again, otherwise the `Ellipsis` or `SKIP` doesn't make much sense).
participants (7)
-
Anders Hovmöller -
Andrew Barnert -
David Mertz -
Dominik Vilsmeier -
Guido van Rossum -
MRAB -
Steven D'Aprano