PEP 443 - Single-dispatch generic functions
Hello, I would like to submit the following PEP for discussion and evaluation. PEP: 443 Title: Single-dispatch generic functions Version: $Revision$ Last-Modified: $Date$ Author: Łukasz Langa <lukasz@langa.pl> Discussions-To: Python-Dev <python-dev@python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 22-May-2013 Post-History: 22-May-2013 Replaces: 245, 246, 3124 Abstract ======== This PEP proposes a new mechanism in the ``functools`` standard library module that provides a simple form of generic programming known as single-dispatch generic functions. A **generic function** is composed of multiple functions sharing the same name. Which form should be used during a call is determined by the dispatch algorithm. When the implementation is chosen based on the type of a single argument, this is known as **single dispatch**. Rationale and Goals =================== Python has always provided a variety of built-in and standard-library generic functions, such as ``len()``, ``iter()``, ``pprint.pprint()``, ``copy.copy()``, and most of the functions in the ``operator`` module. However, it currently: 1. does not have a simple or straightforward way for developers to create new generic functions, 2. does not have a standard way for methods to be added to existing generic functions (i.e., some are added using registration functions, others require defining ``__special__`` methods, possibly by monkeypatching). In addition, it is currently a common anti-pattern for Python code to inspect the types of received arguments, in order to decide what to do with the objects. For example, code may wish to accept either an object of some type, or a sequence of objects of that type. Currently, the "obvious way" to do this is by type inspection, but this is brittle and closed to extension. Abstract Base Classes make it easier to discover present behaviour, but don't help adding new behaviour. A developer using an already-written library may be unable to change how their objects are treated by such code, especially if the objects they are using were created by a third party. Therefore, this PEP proposes a uniform API to address dynamic overloading using decorators. User API ======== To define a generic function, decorate it with the ``@singledispatch`` decorator. Note that the dispatch happens on the type of the first argument, create your function accordingly: .. code-block:: pycon
from functools import singledispatch @singledispatch ... def fun(arg, verbose=False): ... if verbose: ... print("Let me just say,", end=" ") ... print(arg)
To add overloaded implementations to the function, use the ``register()`` attribute of the generic function. It takes a type parameter: .. code-block:: pycon
@fun.register(int) ... def _(arg, verbose=False): ... if verbose: ... print("Strength in numbers, eh?", end=" ") ... print(arg) ... @fun.register(list) ... def _(arg, verbose=False): ... if verbose: ... print("Enumerate this:") ... for i, elem in enumerate(arg): ... print(i, elem)
To enable registering lambdas and pre-existing functions, the ``register()`` attribute can be used in a functional form: .. code-block:: pycon
def nothing(arg, verbose=False): ... print("Nothing.") ... fun.register(type(None), nothing)
When called, the function dispatches on the first argument: .. code-block:: pycon
fun("Hello, world.") Hello, world. fun("test.", verbose=True) Let me just say, test. fun(42, verbose=True) Strength in numbers, eh? 42 fun(['spam', 'spam', 'eggs', 'spam'], verbose=True) Enumerate this: 0 spam 1 spam 2 eggs 3 spam fun(None) Nothing.
The proposed API is intentionally limited and opinionated, as to ensure it is easy to explain and use, as well as to maintain consistency with existing members in the ``functools`` module. Implementation Notes ==================== The functionality described in this PEP is already implemented in the ``pkgutil`` standard library module as ``simplegeneric``. Because this implementation is mature, the goal is to move it largely as-is. Several open issues remain: * the current implementation relies on ``__mro__`` alone, making it incompatible with Abstract Base Classes' ``register()``/``unregister()`` functionality. A possible solution has been proposed by PJE on the original issue for exposing ``pkgutil.simplegeneric`` as part of the ``functools`` API [#issue-5135]_. * the dispatch type is currently specified as a decorator argument. The implementation could allow a form using argument annotations. This usage pattern is out of scope for the standard library [#pep-0008]_. However, whether this registration form would be acceptable for general usage, is up to debate. Based on the current ``pkgutil.simplegeneric`` implementation and following the convention on registering virtual subclasses on Abstract Base Classes, the dispatch registry will not be thread-safe. Usage Patterns ============== This PEP proposes extending behaviour only of functions specifically marked as generic. Just as a base class method may be overridden by a subclass, so too may a function be overloaded to provide custom functionality for a given type. Universal overloading does not equal *arbitrary* overloading, in the sense that we need not expect people to randomly redefine the behavior of existing functions in unpredictable ways. To the contrary, generic function usage in actual programs tends to follow very predictable patterns and overloads are highly-discoverable in the common case. If a module is defining a new generic operation, it will usually also define any required overloads for existing types in the same place. Likewise, if a module is defining a new type, then it will usually define overloads there for any generic functions that it knows or cares about. As a result, the vast majority of overloads can be found adjacent to either the function being overloaded, or to a newly-defined type for which the overload is adding support. It is only in rather infrequent cases that one will have overloads in a module that contains neither the function nor the type(s) for which the overload is added. In the absence of incompetence or deliberate intention to be obscure, the few overloads that are not adjacent to the relevant type(s) or function(s), will generally not need to be understood or known about outside the scope where those overloads are defined. (Except in the "support modules" case, where best practice suggests naming them accordingly.) As mentioned earlier, single-dispatch generics are already prolific throughout the standard library. A clean, standard way of doing them provides a way forward to refactor those custom implementations to use a common one, opening them up for user extensibility at the same time. Alternative approaches ====================== In PEP 3124 [#pep-3124]_ Phillip J. Eby proposes a full-grown solution with overloading based on arbitrary rule sets (with the default implementation dispatching on argument types), as well as interfaces, adaptation and method combining. PEAK-Rules [#peak-rules]_ is a reference implementation of the concepts described in PJE's PEP. Such a broad approach is inherently complex, which makes reaching a consensus hard. In contrast, this PEP focuses on a single piece of functionality that is simple to reason about. It's important to note this does not preclude the use of other approaches now or in the future. In a 2005 article on Artima [#artima2005]_ Guido van Rossum presents a generic function implementation that dispatches on types of all arguments on a function. The same approach was chosen in Andrey Popp's ``generic`` package available on PyPI [#pypi-generic]_, as well as David Mertz's ``gnosis.magic.multimethods`` [#gnosis-multimethods]_. While this seems desirable at first, I agree with Fredrik Lundh's comment that "if you design APIs with pages of logic just to sort out what code a function should execute, you should probably hand over the API design to someone else". In other words, the single argument approach proposed in this PEP is not only easier to implement but also clearly communicates that dispatching on a more complex state is an anti-pattern. It also has the virtue of corresponding directly with the familiar method dispatch mechanism in object oriented programming. The only difference is whether the custom implementation is associated more closely with the data (object-oriented methods) or the algorithm (single-dispatch overloading). Acknowledgements ================ Apart from Phillip J. Eby's work on PEP 3124 [#pep-3124]_ and PEAK-Rules, influences include Paul Moore's original issue [#issue-5135]_ that proposed exposing ``pkgutil.simplegeneric`` as part of the ``functools`` API, Guido van Rossum's article on multimethods [#artima2005]_, and discussions with Raymond Hettinger on a general pprint rewrite. References ========== .. [#issue-5135] http://bugs.python.org/issue5135 .. [#pep-0008] PEP 8 states in the "Programming Recommendations" section that "the Python standard library will not use function annotations as that would result in a premature commitment to a particular annotation style". (http://www.python.org/dev/peps/pep-0008) .. [#pep-3124] http://www.python.org/dev/peps/pep-3124/ .. [#peak-rules] http://peak.telecommunity.com/DevCenter/PEAK_2dRules .. [#artima2005] http://www.artima.com/weblogs/viewpost.jsp?thread=101605 .. [#pypi-generic] http://pypi.python.org/pypi/generic .. [#gnosis-multimethods] http://gnosis.cx/publish/programming/charming_python_b12.html Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev
I like the general idea. Does you have any specific stdlib use cases in mind? I thought of pprint, which at some point dispatches on dict versus set/sequence, but overall it seems more complicated than mere arg type dispatch. Unittest.TestCase.assertEqual mostly (but not completely) uses first arg dispatch based on an instance-specific dict, and it has an custom instance registration method addTypeEqualityFunc. (Since each test_xxx runs in a new instance, a registration for multiple methods has to be done either in a setup method or repeated in each test_method.) Terry
On 23 maj 2013, at 01:16, Terry Jan Reedy <tjreedy@udel.edu> wrote:
I like the general idea. Does you have any specific stdlib use cases in mind?
I thought of pprint, which at some point dispatches on dict versus set/sequence, but overall it seems more complicated than mere arg type dispatch.
I want to make pprint extensible for 3.4 and PEP 443 started out as an idea to introduce a uniform API for the boilerplate I'm going to need anyway. It turned out the idea has been around for years.
Unittest.TestCase.assertEqual mostly (but not completely) uses first arg dispatch based on an instance-specific dict, and it has an custom instance registration method addTypeEqualityFunc. (Since each test_xxx runs in a new instance, a registration for multiple methods has to be done either in a setup method or repeated in each test_method.)
If a registration mechanism is already in place, it will probably need to stay (backwards compatibility). The feasability of refactoring to @singledispatch will have to be considered on a case-by-case basis. On a more general note, I'm sure that @singledispatch won't cover every use case. Still, PJE implemented both pkgutil.simplegeneric and PEAK-Rules because the former is the proverbial 20% that gets you 80% there. For those use cases the simplicity and transparency provided by a basic solution are a virtue. This is what PEP 443 targets. If @singledispatch turns out so successful that we'll find ourselves longing for multiple dispatch or predicate-based dispatch in the future, I'm sure there's still going to be enough PEP numbers free. The @singledispatch name has been chosen to ensure there's no name clash in that case (thanks Nick for suggesting that!). -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev
On 5/22/2013 3:33 PM, Łukasz Langa wrote:
2. does not have a standard way for methods to be added to existing generic functions (i.e., some are added using registration functions, others require defining ``__special__`` methods, possibly by monkeypatching).
I assume you are talking about things like __add__, for operator overloading. And later you mention:
To define a generic function, decorate it with the ``@singledispatch`` decorator. Note that the dispatch happens on the type of the first argument, create your function accordingly:
Yet about half of the operator overloads would be incomplete if there were not corresponding __r*__ methods (__radd__, __rsub__, etc.) because the second parameter is as key to the dispatch as the first. While unary operators, and one argument functions would be fully covered by single dispatch, it is clear that single dispatch doesn't cover a large collection of useful cases for operator overloading. It would seem appropriate to me for the PEP to explain why single dispatch is sufficient, in the presence of a large collection of operations for which it has been demonstrably shown to be insufficient... while the solution is already in place for such operations, single dispatch could clearly not be used as a replacement solution for those operations, opening the door to the thought that maybe single dispatch is an insufficiently useful mechanism, and that perhaps at least two arguments should be used for dispatch (when they exist). On the other hand, when using function call notation instead of operator notation, maybe single dispatch is sufficient... still, non-commutative operations (subtract, divide, etc.) can be difficult to express without resorting to function names like "backwardsSubtract" (__rsub__). But even with commutative operations between unlike objects, it may be that only one of the objects knows how to perform the operations and must be the one that controls the dispatch... Granted, there are few ternary (or n-ary) operators that are not expressed using functional notation anyway, but certainly there is a case to be made for dispatch to happen based on types of all arguments. While that doesn't necessarily detract from the benefits of a single dispatch system, it does raise the question about whether single dispatch is sufficient, especially in the presence of a large collection of (binary) operations for which it is already known to be insufficient.
On Wed, May 22, 2013 at 5:14 PM, Glenn Linderman <v+python@g.nevcal.com> wrote:
Yet about half of the operator overloads would be incomplete if there were not corresponding __r*__ methods (__radd__, __rsub__, etc.) because the second parameter is as key to the dispatch as the first.
This (and your subsequent argument) sounds like a typical case of "perfection is the enemy of the good." Łukasz already pointed out that for dispatch on multiple arguments, consensus has been elusive, and there are some strong statements in opposition. While this does not exclude the possibility that it might be easier to get consensus on dual-argument dispatch, I think the case for dual-argument dispatch is still much weaker than that for single-argument dispatch. The binary operations, which you use as the primary example, are already special because they correspond to syntactic forms. Python intentionally does not have a generalized syntax to invoke arbitrary binary operations, but only supports a small number of predefined binary operators -- code in other languages (like Haskell) that uses "unconventional" binary operators is usually hard to read except for mathematicians. Since the language already offers a way to do dual-argument dispatch for the predefined operations, your proposed dual-argument dispatch wouldn't be particularly useful for those. (And retrofitting it would be a very tricky business, given the many subtleties in the existing binary operator dispatch -- for example, did you know that there's a scenario where __radd__ is tried *before* __add__?) For standard function calls, it would be very odd if dual-dispatch were supported but multiple-dispatch weren't. In general, 0, 1 and infinity are fair game for special treatment, but treating 2 special as well usually smells. So I'd say that Łukasz's single-dispatch proposal covers a fairly important patch of new ground, while dual-dispatch is both much harder and less useful. Ergo, Łukasz has made the right trade-off. -- --Guido van Rossum (python.org/~guido)
On 5/22/2013 5:55 PM, Guido van Rossum wrote:
Yet about half of the operator overloads would be incomplete if there were not corresponding __r*__ methods (__radd__, __rsub__, etc.) because the second parameter is as key to the dispatch as the first. This (and your subsequent argument) sounds like a typical case of "perfection is the enemy of the good." Łukasz already pointed out that for dispatch on multiple arguments, consensus has been elusive, and
On Wed, May 22, 2013 at 5:14 PM, Glenn Linderman <v+python@g.nevcal.com> wrote: there are some strong statements in opposition. While this does not exclude the possibility that it might be easier to get consensus on dual-argument dispatch, I think the case for dual-argument dispatch is still much weaker than that for single-argument dispatch.
The binary operations, which you use as the primary example, are already special because they correspond to syntactic forms. Python intentionally does not have a generalized syntax to invoke arbitrary binary operations, but only supports a small number of predefined binary operators -- code in other languages (like Haskell) that uses "unconventional" binary operators is usually hard to read except for mathematicians.
Since the language already offers a way to do dual-argument dispatch for the predefined operations, your proposed dual-argument dispatch wouldn't be particularly useful for those. (And retrofitting it would be a very tricky business, given the many subtleties in the existing binary operator dispatch -- for example, did you know that there's a scenario where __radd__ is tried *before* __add__?)
For standard function calls, it would be very odd if dual-dispatch were supported but multiple-dispatch weren't. In general, 0, 1 and infinity are fair game for special treatment, but treating 2 special as well usually smells. So I'd say that Łukasz's single-dispatch proposal covers a fairly important patch of new ground, while dual-dispatch is both much harder and less useful. Ergo, Łukasz has made the right trade-off.
Yep. The above, plus a recap of the arguments in opposition to multiple argument dispatch, would make the PEP stronger, which is all I was asking for. I sort of agree with his quote of Frederick Lundh, regarding the complexity of multiple argument dispatch, and multiple argument dispatch/overloading is one of the most complex things to understand and use in C++.
On Thu, May 23, 2013 at 10:14 AM, Glenn Linderman <v+python@g.nevcal.com> wrote:
Yet about half of the operator overloads would be incomplete if there were not corresponding __r*__ methods (__radd__, __rsub__, etc.) because the second parameter is as key to the dispatch as the first.
While unary operators, and one argument functions would be fully covered by single dispatch, it is clear that single dispatch doesn't cover a large collection of useful cases for operator overloading.
The binary operators can be more accurately said to use a complicated single-dispatch dance rather than supporting native dual-dispatch. As you say, the PEP would be strengthened by pointing this out as an argument in favour of staying *away* from a multi-dispatch system (because it isn't obvious how to build a comprehensible one that would even support our existing NotImplemented based dual dispatch system for the binary operators). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Funny. I thought that the PEP was quite strong enough already in its desire to stay away from multi-dispatch. But sure, I don't mind making it stronger. :-) On Wed, May 22, 2013 at 7:12 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Thu, May 23, 2013 at 10:14 AM, Glenn Linderman <v+python@g.nevcal.com> wrote:
Yet about half of the operator overloads would be incomplete if there were not corresponding __r*__ methods (__radd__, __rsub__, etc.) because the second parameter is as key to the dispatch as the first.
While unary operators, and one argument functions would be fully covered by single dispatch, it is clear that single dispatch doesn't cover a large collection of useful cases for operator overloading.
The binary operators can be more accurately said to use a complicated single-dispatch dance rather than supporting native dual-dispatch. As you say, the PEP would be strengthened by pointing this out as an argument in favour of staying *away* from a multi-dispatch system (because it isn't obvious how to build a comprehensible one that would even support our existing NotImplemented based dual dispatch system for the binary operators).
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
On Thu, 23 May 2013 12:12:26 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
On Thu, May 23, 2013 at 10:14 AM, Glenn Linderman <v+python@g.nevcal.com> wrote:
Yet about half of the operator overloads would be incomplete if there were not corresponding __r*__ methods (__radd__, __rsub__, etc.) because the second parameter is as key to the dispatch as the first.
While unary operators, and one argument functions would be fully covered by single dispatch, it is clear that single dispatch doesn't cover a large collection of useful cases for operator overloading.
The binary operators can be more accurately said to use a complicated single-dispatch dance rather than supporting native dual-dispatch.
Not one based on the type of a single argument, though. I guess you can also reduce every function of several arguments to a function accepting a single tuple of several items, but that doesn't sound very interesting. Regards Antoine.
On Thu, May 23, 2013 at 2:04 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Thu, 23 May 2013 12:12:26 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
The binary operators can be more accurately said to use a complicated single-dispatch dance rather than supporting native dual-dispatch.
Not one based on the type of a single argument, though.
Why not? I'd expect it to look something like this: @singledispatch def ladd(left, right): return NotImplemented @singledispatch def radd(right, left): return NotImplemented def add(left, right): x = ladd(left, right) if x is not NotImplemented: return x x = radd(right, left) if x is not NotImplemented: return x raise TypeError Then instead of defining __add__ you define an overloaded implementation of ladd, and instead of defining __radd__ you define an overloaded implementation of radd. -- Devin
On Thu, 23 May 2013 02:33:57 -0400 Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
On Thu, May 23, 2013 at 2:04 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Thu, 23 May 2013 12:12:26 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
The binary operators can be more accurately said to use a complicated single-dispatch dance rather than supporting native dual-dispatch.
Not one based on the type of a single argument, though.
Why not?
I'd expect it to look something like this:
@singledispatch def ladd(left, right): return NotImplemented
@singledispatch def radd(right, left): return NotImplemented
def add(left, right): x = ladd(left, right) if x is not NotImplemented: return x x = radd(right, left) if x is not NotImplemented: return x raise TypeError
Then instead of defining __add__ you define an overloaded implementation of ladd, and instead of defining __radd__ you define an overloaded implementation of radd.
Well, I don't think you can say add() dispatches based on the type of a single argument. But that may be a question of how you like to think about decomposed problems. Regards Antoine.
On 5/23/2013 12:14 AM, Antoine Pitrou wrote:
On Thu, 23 May 2013 02:33:57 -0400 Devin Jeanpierre<jeanpierreda@gmail.com> wrote:
On Thu, May 23, 2013 at 2:04 AM, Antoine Pitrou<solipsis@pitrou.net> wrote:
On Thu, 23 May 2013 12:12:26 +1000 Nick Coghlan<ncoghlan@gmail.com> wrote:
>The binary operators can be more accurately said to use a complicated >single-dispatch dance rather than supporting native dual-dispatch.
Not one based on the type of a single argument, though.
Why not?
I'd expect it to look something like this:
@singledispatch def ladd(left, right): return NotImplemented
@singledispatch def radd(right, left): return NotImplemented
def add(left, right): x = ladd(left, right) if x is not NotImplemented: return x x = radd(right, left) if x is not NotImplemented: return x raise TypeError
Then instead of defining __add__ you define an overloaded implementation of ladd, and instead of defining __radd__ you define an overloaded implementation of radd. Well, I don't think you can say add() dispatches based on the type of a single argument. But that may be a question of how you like to think about decomposed problems.
I suspect the point was not that add can be described as doing single dispatch (it can't), but rather that add could possibly be implemented in terms of lower-level functions doing single dispatch. If that was the point, perhaps the next level point is trying to be that single dispatch is a sufficient mechanism that can be augmented (as above) to handle more complex cases. Whether the above (which I think would need to use raise and try instead of return and if) is sufficient to handle such cases is not yet proven. The case Guido mention where radd is tried before add would seem to require a bit more complex logic than the above.
Le Thu, 23 May 2013 00:31:38 -0700, Glenn Linderman <v+python@g.nevcal.com> a écrit :
I suspect the point was not that add can be described as doing single dispatch (it can't), but rather that add could possibly be implemented in terms of lower-level functions doing single dispatch. If that was the point, perhaps the next level point is trying to be that single dispatch is a sufficient mechanism that can be augmented (as above) to handle more complex cases.
This is true, but as it is of everything Turing-complete. Generic functions don't add anything that you can't already do manually (for example with custom registries) :-) Regardless, I also agree that single-dispatch is much easier to reason about, and good enough for now. Regards Antoine.
On 23 May 2013 16:37, "Devin Jeanpierre" <jeanpierreda@gmail.com> wrote:
On Thu, May 23, 2013 at 2:04 AM, Antoine Pitrou <solipsis@pitrou.net>
wrote:
On Thu, 23 May 2013 12:12:26 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
The binary operators can be more accurately said to use a complicated single-dispatch dance rather than supporting native dual-dispatch.
Not one based on the type of a single argument, though.
Why not?
I'd expect it to look something like this:
@singledispatch def ladd(left, right): return NotImplemented
@singledispatch def radd(right, left): return NotImplemented
def add(left, right): x = ladd(left, right) if x is not NotImplemented: return x x = radd(right, left) if x is not NotImplemented: return x raise TypeError
Then instead of defining __add__ you define an overloaded implementation of ladd, and instead of defining __radd__ you define an overloaded implementation of radd.
That's the basic idea, but there's the extra complication that if type(right) is a strict subclass of type(left), you try radd first. Cheers, Nick.
-- Devin _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
Hi, On Thu, May 23, 2013 at 12:33 AM, Łukasz Langa <lukasz@langa.pl> wrote:
Alternative approaches ======================
You could also mention "pairtype", used in PyPy: https://bitbucket.org/pypy/pypy/raw/default/rpython/tool/pairtype.py (very short code). It's originally about adding double-dispatch, but the usage that grew out of it is for generic single-dispatch functions that are bound to some common "state" object as follows (Python 2 syntax): class MyRepr(object): ...state of my repr... class __extend__(pairtype(MyRepr, int)): def show((myrepr, x), y): print "hi, I'm the integer %d, arg is %s" % (x, y) class __extend__(pairtype(MyRepr, list)): def show((myrepr, x), y): print "hi, I'm a list" ...use myrepr to control the state... pair(MyRepr(), [2,3,4]).show(42) - Armin
On 23 maj 2013, at 09:33, Armin Rigo <arigo@tunes.org> wrote:
Hi,
On Thu, May 23, 2013 at 12:33 AM, Łukasz Langa <lukasz@langa.pl> wrote:
Alternative approaches ======================
You could also mention "pairtype", used in PyPy:
Thanks for pointing that out. Information on it added in http://hg.python.org/peps/rev/b7979219f3cc#l1.7 +PyPy's RPython offers ``extendabletype`` [#pairtype]_, a metaclass which +enables classes to be externally extended. In combination with +``pairtype()`` and ``pair()`` factories, this offers a form of +single-dispatch generics. +.. [#pairtype] + https://bitbucket.org/pypy/pypy/raw/default/rpython/tool/pairtype.py -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev
Łukasz, are there any open issues? Otherwise I'm ready to accept the PEP. -- --Guido van Rossum (python.org/~guido)
On 23 maj 2013, at 16:49, Guido van Rossum <guido@python.org> wrote:
Łukasz, are there any open issues? Otherwise I'm ready to accept the PEP.
There's one. Quoting the PEP: "The dispatch type is currently specified as a decorator argument. The implementation could allow a form using argument annotations. This usage pattern is out of scope for the standard library (per PEP 8). However, whether this registration form would be acceptable for general usage, is up to debate." I feel that the PEP should explicitly allow or disallow for the implementation to accept dispatch on annotations, e.g.: @func.register def _(arg: int): ... versus @func.register(int) def _(arg): ... -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev
On 05/23/2013 07:58 AM, Łukasz Langa wrote:
On 23 maj 2013, at 16:49, Guido van Rossum <guido@python.org> wrote:
Łukasz, are there any open issues? Otherwise I'm ready to accept the PEP.
There's one. Quoting the PEP:
"The dispatch type is currently specified as a decorator argument. The implementation could allow a form using argument annotations. This usage pattern is out of scope for the standard library (per PEP 8). However, whether this registration form would be acceptable for general usage, is up to debate."
I feel that the PEP should explicitly allow or disallow for the implementation to accept dispatch on annotations, e.g.:
@func.register def _(arg: int): ...
versus
@func.register(int) def _(arg): ...
If the stdlib is still staying out of the annotation business, then it should not be allowed. -- ~Ethan~
On 24/05/13 01:04, Ethan Furman wrote:
On 05/23/2013 07:58 AM, Łukasz Langa wrote:
I feel that the PEP should explicitly allow or disallow for the implementation to accept dispatch on annotations, e.g.:
@func.register def _(arg: int): ...
versus
@func.register(int) def _(arg): ...
If the stdlib is still staying out of the annotation business, then it should not be allowed.
Perhaps it is time to relax that ruling? The standard library acts as a guide to best practice in Python, and I think that uptake of annotations has been hurt due to the lack of good examples. Also, anyone with the conceit that their library or module may someday be in the standard library cannot afford to use annotations at all. So I'm tentatively +1 on allowing the annotation form in addition to the decorator argument form. -- Steven
On May 23, 2013 4:37 PM, "Steven D'Aprano" <steve@pearwood.info> wrote:
On 24/05/13 01:04, Ethan Furman wrote:
If the stdlib is still staying out of the annotation business, then it
should not be allowed.
Perhaps it is time to relax that ruling? The standard library acts as a
guide to best practice in Python, and I think that uptake of annotations has been hurt due to the lack of good examples. Also, anyone with the conceit that their library or module may someday be in the standard library cannot afford to use annotations at all. The idea that decorators determine the meaning of annotations (i.e. they have no meaning without a decorator) really appeals to me. I don't see the imperative for this PEP though, but I'm not opposed. If there were more discussion and consensus on annotations + decorators I'd be more convinced. -eric
On Fri, May 24, 2013 at 11:45 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
On Thu, May 23, 2013 at 7:30 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
If there were more discussion and consensus on annotations + decorators I'd be more convinced.
However, this PEP should not be gated on any such discussion.
Right, I think the latest update makes the right call by saying "maybe someday, but not for now". Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 23 May 2013 15:58, Łukasz Langa <lukasz@langa.pl> wrote:
On 23 maj 2013, at 16:49, Guido van Rossum <guido@python.org> wrote:
Łukasz, are there any open issues? Otherwise I'm ready to accept the PEP.
There's one. Quoting the PEP:
"The dispatch type is currently specified as a decorator argument. The implementation could allow a form using argument annotations. This usage pattern is out of scope for the standard library (per PEP 8). However, whether this registration form would be acceptable for general usage, is up to debate."
I feel that the PEP should explicitly allow or disallow for the implementation to accept dispatch on annotations, e.g.:
@func.register def _(arg: int): ...
versus
@func.register(int) def _(arg): ...
Personally, I think the register(int) form seems more natural. But that may well be because there are no uses of annotations in the wild (at least not in code I'm familiar with) and having this as an example of how annotations can be used would help with adoption. I'm not 100% sure what the options are. 1. Only support the register(int) form 2. Only support the annotation form 3. Support both annotation and argument forms Is the debate between 1 and 2, or 1 and 3? Is it even possible to implement 3 without having 2 different names for "register"? If the debate is between 1 and 2, I'd prefer 1. But if it's between 1 and 3, I'm less sure - having the *option* to try annotations for this in my own code sounds useful. Paul.
On Thu, May 23, 2013 at 11:11 AM, Paul Moore <p.f.moore@gmail.com> wrote:
Is the debate between 1 and 2, or 1 and 3? Is it even possible to implement 3 without having 2 different names for "register"?
Yes. You could do it as either: @func.register def doit(foo: int): ... by checking for the first argument to register() being a function, or: @func.register() def doit(foo: int): ... by using a default None first argument. In either case, you would then raise a TypeError if there wasn't an annotation. As to the ability to do multiple types registration, you could support it only in type annotations, e.g.: @func.register def doit(foo: [int, float]): ... without it being confused with being multiple dispatch. One other thing about the register API that's currently unspecified in the PEP: what does it return, exactly? I generally lean towards returning the undecorated function, so that if you say: @func.register def do_int(foo: int): ... You still have the option of calling it explicitly. OTOH, some may prefer to treat it like an overload and call it 'func' every time, in which case register should return the generic function. Some guidance as to what should be the official One Obvious Way would be helpful here. (Personally, I usually name my methods explicitly because in debugging it's a fast clue as to which piece of code I should be looking at.)
On Thu, May 23, 2013 at 2:59 PM, PJ Eby <pje@telecommunity.com> wrote:
I generally lean towards returning the undecorated function, so that if you say:
@func.register def do_int(foo: int): ...
Oops, forgot to mention: one other advantage to returning the undecorated function is that you can do this: @func.register(int) @func.register(float) def do_num(foo): ... Which neatly solves the multiple registration problem, even without argument annotations.
On 23 maj 2013, at 20:59, PJ Eby <pje@telecommunity.com> wrote:
As to the ability to do multiple types registration, you could support it only in type annotations, e.g.:
@func.register def doit(foo: [int, float]): ...
Initially I thought so, too. But it seems other people might think this means "a sequence with the first element being an integer, and the second a float". The BDFL seems to have yet a different idea: http://mail.python.org/pipermail/python-ideas/2012-December/018129.html This is clearly material for a separate PEP, wink wink, nudge nudge. To the point though. Based on this, and the fact PEP 8 currently disallows annotations within the standard library, I came to the conclusion that currently we should not include the annotation-driven form.
I generally lean towards returning the undecorated function, so that if you say:
@func.register def do_int(foo: int): ...
Me too. The PEP has been updated to specify that explicitly. -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev
On 05/23/2013 01:10 PM, Łukasz Langa wrote:
On 23 maj 2013, at 20:59, PJ Eby <pje@telecommunity.com> wrote:
As to the ability to do multiple types registration, you could support it only in type annotations, e.g.:
@func.register def doit(foo: [int, float]): ...
Initially I thought so, too. But it seems other people might think this means "a sequence with the first element being an integer, and the second a float". The BDFL seems to have yet a different idea:
http://mail.python.org/pipermail/python-ideas/2012-December/018129.html
This is clearly material for a separate PEP, wink wink, nudge nudge.
To the point though. Based on this, and the fact PEP 8 currently disallows annotations within the standard library, I came to the conclusion that currently we should not include the annotation-driven form.
I generally lean towards returning the undecorated function, so that if you say:
@func.register def do_int(foo: int): ...
Me too. The PEP has been updated to specify that explicitly.
So with this decision made, are there any open issues left? Or can we invite Guido back to the discussion? ;) -- ~Ethan~
Ok, happy bikeshedding. I'm outta here until that's settled. :-) On Thu, May 23, 2013 at 7:58 AM, Łukasz Langa <lukasz@langa.pl> wrote:
On 23 maj 2013, at 16:49, Guido van Rossum <guido@python.org> wrote:
Łukasz, are there any open issues? Otherwise I'm ready to accept the PEP.
There's one. Quoting the PEP:
"The dispatch type is currently specified as a decorator argument. The implementation could allow a form using argument annotations. This usage pattern is out of scope for the standard library (per PEP 8). However, whether this registration form would be acceptable for general usage, is up to debate."
I feel that the PEP should explicitly allow or disallow for the implementation to accept dispatch on annotations, e.g.:
@func.register def _(arg: int): ...
versus
@func.register(int) def _(arg): ...
-- Best regards, Łukasz Langa
WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev
-- --Guido van Rossum (python.org/~guido)
On 23.05.13 00:33, Łukasz Langa wrote:
Hello, I would like to submit the following PEP for discussion and evaluation.
PEP: 443 Title: Single-dispatch generic functions [...]
@fun.register(int) ... def _(arg, verbose=False): ... if verbose: ... print("Strength in numbers, eh?", end=" ") ... print(arg) ...
Should it be possible to register multiple types for the generic function with one register() call, i.e. should: @fun.register(int, float) def _(arg, verbose=False): ... be allowed as a synonym for @fun.register(int) @fun.register(float) def _(arg, verbose=False): ... Servus, Walter
On 23 May 2013 17:00, Walter Dörwald <walter@livinglogic.de> wrote:
Should it be possible to register multiple types for the generic function with one register() call, i.e. should:
@fun.register(int, float) def _(arg, verbose=False): ...
be allowed as a synonym for
@fun.register(int) @fun.register(float) def _(arg, verbose=False):
No, because people will misread register(int, float) as meaning first argument int, second float. The double decorator is explicit as to what is going on, and isn't too hard to read or write. Paul
On 24/05/13 02:56, Paul Moore wrote:
On 23 May 2013 17:00, Walter Dörwald <walter@livinglogic.de> wrote:
Should it be possible to register multiple types for the generic function with one register() call, i.e. should:
@fun.register(int, float) def _(arg, verbose=False): ...
be allowed as a synonym for
@fun.register(int) @fun.register(float) def _(arg, verbose=False):
No, because people will misread register(int, float) as meaning first argument int, second float. The double decorator is explicit as to what is going on, and isn't too hard to read or write.
I don't think that they will. Being able to register multiple types with a single call reads very naturally to me, while multiple decorators still looks weird. Even after many years of seeing them, I still get a momentary "What the hell...?" moment when I see two decorators on one function. That's only going to be increased when both decorators are the same (apart from the argument). The double decorator form above looks to me as weird as: x = func(a) x = func(b) would. I have to stop and think about what is going on, and whether or not it is a mistake. So I am a strong +1 on allowing multiple types to be registered in one call. -- Steven
So I am a strong +1 on allowing multiple types to be registered in one call.
Yeah, agreed. It also fits the pattern set by isinstance(), which allows a tuple of types, like isinstance(x, (int, str)). That said, I'm +0 on this PEP itself. It seems no one has provided decent use-case examples (apart from contrived ones), from the stdlib for example. In the fairly large codebase I work on, it'd only be used in one place, and even there the PEP's approach is arguably too simple for what we're doing. It seems to me for the few times this would be used, direct and simple use of isinstance() would be clearer. But maybe that's just our particular codebase. -Ben
On Thu, May 23, 2013 at 6:58 PM, Ben Hoyt <benhoyt@gmail.com> wrote:
It seems no one has provided decent use-case examples (apart from contrived ones)
Um, copy.copy(), pprint.pprint(), a bunch of functions in pkgutil which are actually *based on this implementation already* and have been since Python 2.5... I don't see how any of those are contrived examples. If we'd had this in already, all the registration-based functions for copying, pickling, etc. would likely have been implemented this way, and the motivating example for the PEP is the coming refactoring of pprint.pprint.
On Fri, May 24, 2013 at 10:31 AM, PJ Eby <pje@telecommunity.com> wrote:
On Thu, May 23, 2013 at 6:58 PM, Ben Hoyt <benhoyt@gmail.com> wrote:
It seems no one has provided decent use-case examples (apart from contrived ones)
Um, copy.copy(), pprint.pprint(), a bunch of functions in pkgutil which are actually *based on this implementation already* and have been since Python 2.5... I don't see how any of those are contrived examples. If we'd had this in already, all the registration-based functions for copying, pickling, etc. would likely have been implemented this way, and the motivating example for the PEP is the coming refactoring of pprint.pprint.
We should be able to use it to help deal with the "every growing importer API" problem, too. I know that's technically what pkgutil already uses it for, but elevating this from "pkgutil implementation detail" to "official stdlib functionality" should make it easier to document properly :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, May 23, 2013 at 11:57 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
We should be able to use it to help deal with the "every growing importer API" problem, too. I know that's technically what pkgutil already uses it for, but elevating this from "pkgutil implementation detail" to "official stdlib functionality" should make it easier to document properly :)
Oh, that reminds me. pprint() is actually an instance of a general pattern that single dispatch GF's are good for: "visitor pattern" algorithms. There's a pretty good write-up on the general issues with doing visitor pattern stuff in Python, and how single-dispatch GF's can solve that problem, here: http://peak.telecommunity.com/DevCenter/VisitorRevisited The code samples use a somewhat different API from the PEP, but it's pretty close. The main issues solved are eliminating monkeypatching and fixing the inheritance problems that occur when you use 'visit_foo' methods. One of the samples actually comes from the old 'compiler' package in the stdlib... which tells you how long ago I did the write-up. ;-)
On Fri, May 24, 2013 at 8:40 AM, Steven D'Aprano <steve@pearwood.info> wrote:
I don't think that they will. Being able to register multiple types with a single call reads very naturally to me, while multiple decorators still looks weird. Even after many years of seeing them, I still get a momentary "What the hell...?" moment when I see two decorators on one function. That's only going to be increased when both decorators are the same (apart from the argument). The double decorator form above looks to me as weird as:
x = func(a) x = func(b)
would. I have to stop and think about what is going on, and whether or not it is a mistake.
The difference is that this idiom quickly becomes familiar and unexceptional: @fun.register(float) @fun.register(Decimal) def fun_floating_point(arg1, arg2): ... "Oh, OK, 'fun' is a generic function, and we're registering this as the implementation for floats and Decimals" By contrast, the following are *always* ambiguous at the point of definition, as it depends on how fun is defined: @fun.register(float, Decimal) def fun_floating_point(arg1, arg2): ... @fun.register([float, Decimal]) def fun_floating_point(arg1, arg2): ... Is that multiple dispatch? Or is it registering for single dispatch on multiple different types? Sure, we could pick the latter meaning for the standard library, but existing generic function implementations (cited in the PEP) use the tuple-of-types notation for multiple dispatch. By opting for stacking decorators in the PEP and hence the stdlib, we leave the way clear for 3rd party multi-dispatch libraries to use multiple type arguments without introducing any ambiguity.
So I am a strong +1 on allowing multiple types to be registered in one call.
Whereas I'm a strong -1, as the ambiguity problem it would create is persistent and irreversible, while stacking registration decorators is just a new idiom to become accustomed to. Cheers, Nick.
On 24/05/13 15:09, Nick Coghlan wrote:
On Fri, May 24, 2013 at 8:40 AM, Steven D'Aprano <steve@pearwood.info> wrote:
I don't think that they will. Being able to register multiple types with a single call reads very naturally to me, while multiple decorators still looks weird. Even after many years of seeing them, I still get a momentary "What the hell...?" moment when I see two decorators on one function. That's only going to be increased when both decorators are the same (apart from the argument). The double decorator form above looks to me as weird as:
x = func(a) x = func(b)
would. I have to stop and think about what is going on, and whether or not it is a mistake.
The difference is that this idiom quickly becomes familiar and unexceptional:
@fun.register(float) @fun.register(Decimal) def fun_floating_point(arg1, arg2): ...
I initially wrote a reply about the nature of ambiguity, why register(float, Decimal) should not be considered ambiguous, why stacked decorators that are near-duplicates are a code smell, blah blah blah. But for the sake of brevity I'm going to skip it. The important point that you make is here:
Is that multiple dispatch? Or is it registering for single dispatch on multiple different types?
Sure, we could pick the latter meaning for the standard library, but existing generic function implementations (cited in the PEP) use the tuple-of-types notation for multiple dispatch.
This is an excellent point I had not considered. By the way, it seems to me that Guido's multimethod implementation referenced in the PEP actually uses a single decorator argument per function argument, not a tuple-of-types: @multimethod(int, int) def foo(a, b): ...code for two ints... http://www.artima.com/weblogs/viewpost.jsp?thread=101605 You have convinced me: ambiguous or not, for the sake of future expansion I agree that multiple positional arguments to the register method should be left for some hypothetical multiple-dispatch generics: @fun.register(float, Decimal) # not yet supported, but maybe someday would mean "first argument is a float, second argument is a Decimal". But that still leaves open how to specify single dispatch on more than one type:
stacking registration decorators is just a new idiom to become accustomed to.
Python built-ins and the standard library already have a standard idiom for specifying multiple values at once. A tuple of types is the One Obvious Way to do this: @fun.register((float, Decimal)) which matches the same standard idiom that should be familiar to most people: isinstance(obj, (float, Decimal)) issubclass(cls, (float, Decimal)) And of course it is forward-compatible with our hypothetical future multiple-generics. -- Steven
On Fri, May 24, 2013 at 8:53 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Python built-ins and the standard library already have a standard idiom for specifying multiple values at once. A tuple of types is the One Obvious Way to do this:
@fun.register((float, Decimal))
It's not obvious, it's ambiguous - some third party libraries use that notation for multi-method dispatch, and they always will, no matter what notation we choose for the standard library. We have three available notations to register the same function for multiple types: stacked decorators, tuple-of-types and multiple arguments. Of those, the first we *cannot avoid* supporting, since we want to return the undecorated function regardless for pickle support and ease of testing. The second two are both used as notations by existing third party multiple dispatch libraries. Thus, your request is that we add a second way to do it that is *known* to conflict with existing third party practices. There is no practical gain on offer, it merely aligns with your current sense of aesthetics slightly better than stacked decorators do. While you're entitled to that aesthetic preference, it isn't a valid justification for adding an unneeded alternate spelling. Furthermore, the proposed registration syntax in the PEP is identical to the syntax which already exists for ABC registration as a class decorator (http://docs.python.org/3/library/abc#abc.ABCMeta.register). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, May 24, 2013 at 9:41 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Furthermore, the proposed registration syntax in the PEP is identical to the syntax which already exists for ABC registration as a class decorator (http://docs.python.org/3/library/abc#abc.ABCMeta.register).
Sorry, I withdraw that observation - it's wrong. ABC registration obviously doesn't need to provide arguments to the decorator at all, since the only necessary info is the ABC itself, and that's providing through the method binding. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, May 23, 2013 at 6:40 PM, Steven D'Aprano <steve@pearwood.info> wrote:
I don't think that they will. Being able to register multiple types with a single call reads very naturally to me, while multiple decorators still looks weird. Even after many years of seeing them, I still get a momentary "What the hell...?" moment when I see two decorators on one function. That's only going to be increased when both decorators are the same (apart from the argument). The double decorator form above looks to me as weird as:
x = func(a) x = func(b)
would. I have to stop and think about what is going on, and whether or not it is a mistake.
That's absurd. The above is not comparable to double decorators, the following is: x = func(a) x = func(x) And this is clearly not something anyone has to stop and think about. (more literally, obviously it's actually def x(...): ... ; x = func(a)(x); x = func(b)(x)) There is nothing remotely wrong or distasteful about using multiple decorators. It's a natural thing to want to compose multiple functions together; for example, @functools.lru_cache with @fun.register or @staticmethod or [...]. And it's even natural to want to apply the same decorator with different arguments multiple times to the same thing, if it happens to do something different when given different arguments. -- Devin
User API ========
To define a generic function, decorate it with the ``@singledispatch`` decorator. Note that the dispatch happens on the type of the first argument, create your function accordingly:
.. code-block:: pycon
from functools import singledispatch @singledispatch ... def fun(arg, verbose=False): ... if verbose: ... print("Let me just say,", end=" ") ... print(arg)
To add overloaded implementations to the function, use the ``register()`` attribute of the generic function. It takes a type parameter:
.. code-block:: pycon
@fun.register(int) ... def _(arg, verbose=False): ... if verbose: ... print("Strength in numbers, eh?", end=" ") ... print(arg) ... @fun.register(list) ... def _(arg, verbose=False): ... if verbose: ... print("Enumerate this:") ... for i, elem in enumerate(arg): ... print(i, elem)
To enable registering lambdas and pre-existing functions, the ``register()`` attribute can be used in a functional form:
.. code-block:: pycon
def nothing(arg, verbose=False): ... print("Nothing.") ... fun.register(type(None), nothing)
So to have a generic `mapping` function that worked on dicts, namedtuples, user-defined record types, etc., would look something like: --> from functools import singledispatch --> @singledispatch --> def mapping(d): ... new_d = {} ... new_d.update(d) ... return new_d ... --> @mapping.register(tuple) ... def _(t): ... names = getattr(t, '_fields', ['f%d' % n for n in range(len(t))]) ... values = list(t) ... return dict(zip(names, values)) ... --> @mapping.register(user_class): ... def _(uc): ... blah blah ... return dict(more blah) ... Very cool. I'm looking forward to it! Oh, the tuple example above is intended primarily for named tuples, but since there is no common base class besides tuple I had to also handle the case where a plain tuple is passed in, and personally I'd rather have generic field names than raise an exception. -- ~Ethan~
Hi, Thanks for writing this PEP. Blessing one implementation for the stdlib and one official backport will make programmers’ lives a bit easier :)
@fun.register(int) ... def _(arg, verbose=False): ... if verbose: ... print("Strength in numbers, eh?", end=" ") ... print(arg) ...
Does this work if the implementation function is called like the first decorated function? (I don’t know the proper terminology) e.g. >>> @fun.register(int) ... def fun(arg, verbose=False): ... if verbose: ... print("Strength in numbers, eh?", end=" ") ... print(arg) The precedent is 2.6+ properties, where prop.setter mutates and returns the property object, which then overwrites the previous name in the class dictionary.
* the current implementation relies on ``__mro__`` alone, making it incompatible with Abstract Base Classes' ``register()``/``unregister()`` functionality. A possible solution has been proposed by PJE on the original issue for exposing ``pkgutil.simplegeneric`` as part of the ``functools`` API [#issue-5135]_.
Making generic functions work with ABCs sounds like a requirement to me, as ABCs are baked into the language (isinstance). ABCs and interfaces (i.e. zope.interface) are really neat and powerful.
* the dispatch type is currently specified as a decorator argument. The implementation could allow a form using argument annotations. This usage pattern is out of scope for the standard library [#pep-0008]_. However, whether this registration form would be acceptable for general usage, is up to debate.
+1 to passing the type as argument to the decorator and not supporting annotations. It’s simple and works. Question: what happens if two functions (say in two different modules) are registered for the same type?
On 05/23/2013 11:13 AM, Éric Araujo wrote:
Thanks for writing this PEP. Blessing one implementation for the stdlib and one official backport will make programmers’ lives a bit easier :)
@fun.register(int) ... def _(arg, verbose=False): ... if verbose: ... print("Strength in numbers, eh?", end=" ") ... print(arg) ...
Does this work if the implementation function is called like the first decorated function? (I don’t know the proper terminology) e.g.
>>> @fun.register(int) ... def fun(arg, verbose=False): ... if verbose: ... print("Strength in numbers, eh?", end=" ") ... print(arg)
The precedent is 2.6+ properties, where prop.setter mutates and returns the property object, which then overwrites the previous name in the class dictionary.
Actually, properties return new instances: --> class Test(object): ... _temp = 'fleeting' ... @property ... def temp(self): ... return self._temp ... @temp.setter ... def new_temp(self, value): ... self._temp = value ... --> id(Test.temp) 30245384 --> id(Test.new_temp) 30246352 --> Test.temp is Test.new_temp False -- ~Ethan~
On 23 maj 2013, at 20:13, Éric Araujo <merwok@netwok.org> wrote:
Does this work if the implementation function is called like the first decorated function?
No, the ``register()`` attribute returns the undecorated function which enables decorator stacking, as well as creating unit tests for each variant independently.
Making generic functions work with ABCs sounds like a requirement to me
Yes, I will implement that.
Question: what happens if two functions (say in two different modules) are registered for the same type?
Last one wins. Just like with assigning names in a scope, defining methods in a class or overriding them in a subclass. -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev
2013/5/23 Łukasz Langa <lukasz@langa.pl>
On 23 maj 2013, at 20:13, Éric Araujo <merwok@netwok.org> wrote:
Question: what happens if two functions (say in two different modules) are registered for the same type?
Last one wins. Just like with assigning names in a scope, defining methods in a class or overriding them in a subclass.
This is a serious annoyance, considering that there are several places where a large library can reasonably define the implementations (i.e. with the class, with the function, or in some utility module). Note that in contrast with the case of functions in a module or methods in a class, linting tools cannot be expected to detect a duplication between functions with different names defined in different modules. Another thing missing from the PEP is the ability to access the implementation function when you know the generic function and the class. A major use case for this is to define the implementation for a subclass by reusing its parent's implementation, e.g. : @some_generic.register(my_int) def _(arg): print("Hello from my_int!") return some_generic[int](arg) -- Ronan Lamy
On 05/23/2013 02:02 PM, Ronan Lamy wrote:
2013/5/23 Łukasz Langa <lukasz@langa.pl <mailto:lukasz@langa.pl>>
On 23 maj 2013, at 20:13, Éric Araujo <merwok@netwok.org <mailto:merwok@netwok.org>> wrote:
> Question: what happens if two functions (say in two different modules) > are registered for the same type?
Last one wins. Just like with assigning names in a scope, defining methods in a class or overriding them in a subclass.
This is a serious annoyance, considering that there are several places where a large library can reasonably define the implementations (i.e. with the class, with the function, or in some utility module). Note that in contrast with the case of functions in a module or methods in a class, linting tools cannot be expected to detect a duplication between functions with different names defined in different modules.
What would you suggest happen in this case? -- ~Ethan~
2013/5/24 Ethan Furman <ethan@stoneleaf.us>
On 05/23/2013 02:02 PM, Ronan Lamy wrote:
2013/5/23 Łukasz Langa <lukasz@langa.pl <mailto:lukasz@langa.pl>>
On 23 maj 2013, at 20:13, Éric Araujo <merwok@netwok.org <mailto: merwok@netwok.org>> wrote:
> Question: what happens if two functions (say in two different modules) > are registered for the same type?
Last one wins. Just like with assigning names in a scope, defining methods in a class or overriding them in a subclass.
This is a serious annoyance, considering that there are several places where a large library can reasonably define the implementations (i.e. with the class, with the function, or in some utility module). Note that in contrast with the case of functions in a module or methods in a class, linting tools cannot be expected to detect a duplication between functions with different names defined in different modules.
What would you suggest happen in this case?
Raise a ValueError, maybe? In that case, there needs to be a way to force the overriding when it is explicitly desired. One way would be to allow unregistering implementations: overriding is then done by unregistering the old implementation before defining the new one. This is a bit cumbersome, which IMHO is a good thing for an operation that is just as disruptive as monkey-patching a class or a module.
On Fri, May 24, 2013 at 10:22 PM, Ronan Lamy <ronan.lamy@gmail.com> wrote:
Raise a ValueError, maybe? In that case, there needs to be a way to force the overriding when it is explicitly desired. One way would be to allow unregistering implementations: overriding is then done by unregistering the old implementation before defining the new one. This is a bit cumbersome, which IMHO is a good thing for an operation that is just as disruptive as monkey-patching a class or a module.
If you're registering an implementation for a type you didn't define on a generic function you didn't define, it's *exactly* as disruptive as monkey-patching. Note that the PEP proposes giving exactly as much of a runtime warning about overwriting a registration as we do about monkeypatching: none. The two cases are exactly analagous: you can do it, you don't get a warning if you do it, but it you do it implicitly as a side effect of import then you will have developers cursing your name. So don't do that, put it in a function that people can call if they want to register your implementations (along the lines of gevent.monkey). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 24 maj 2013, at 14:22, Ronan Lamy <ronan.lamy@gmail.com> wrote:
2013/5/24 Ethan Furman <ethan@stoneleaf.us> What would you suggest happen in this case? Raise a ValueError, maybe? In that case, there needs to be a way to force the overriding when it is explicitly desired. One way would be to allow unregistering implementations: overriding is then done by unregistering the old implementation before defining the new one.
Unfortunately this isn't going to work because the order of imports might change during the life cycle of a program. Especially if you wish to expose registering to users, the order of imports cannot be guaranteed. I recognize the need for such behaviour to be discoverable. This is important for debugging purposes. This is why I'm going to let users inspect registered overloads, as well as provide their own mapping for the registry. I'm working on the reference implementation now, stay tuned. -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev
On 23 May 2013 22:02, Ronan Lamy <ronan.lamy@gmail.com> wrote:
2013/5/23 Łukasz Langa <lukasz@langa.pl>
Last one wins. Just like with assigning names in a scope, defining methods in a class or overriding them in a subclass.
This is a serious annoyance, considering that there are several places where a large library can reasonably define the implementations (i.e. with the class, with the function, or in some utility module). Note that in contrast with the case of functions in a module or methods in a class, linting tools cannot be expected to detect a duplication between functions with different names defined in different modules.
But isn't it much much worse than names in scope, as with assigning names in a scope it is only your scope that is affected : from os.path import join def join(wibble): 'overloads join in this module only' any other module is unaffected, os.path.join still calls os.path.join however with this all scopes globally are affected by the last one wins rule. -----default.py------- from pkgutil import simplegeneric @simplegeneric def fun(x): print 'default impl' -------a.py-------- from default import fun @fun.register(int) def impl_a(x): print 'impl_a' def f(): fun(0) # expect this to call impl_a -------b.py------ from default import fun @fun.register(int) def impl_b(x): print 'impl_b' def f(): fun(0) # expect this to call impl_b --------
import a, b a.f() impl_b b.f() impl_b
import b, a a.f() impl_a b.f() impl_a exit()
That is rather worrying. It is more analagous in the above example to sys.modules['os.path'].join = myjoin I don't have a solution mind though. Sam
On Fri, May 24, 2013 at 7:54 PM, Sam Partington <sam.partington@gmail.com> wrote:
But isn't it much much worse than names in scope, as with assigning names in a scope it is only your scope that is affected :
from os.path import join def join(wibble): 'overloads join in this module only'
any other module is unaffected, os.path.join still calls os.path.join
however with this all scopes globally are affected by the last one wins rule.
Indeed, as with any modification of process global state, generic implementation registration is something to be approached with care. ABC registration is similar. There's actually three kinds of registration that can happen, and only two of them are appropriate for libraries to do implicitly, while the last should only be explicitly triggered from main: * registering a class your library defines with a stdlib or third party generic function * registering a stdlib or third party class with a generic function your library defines * registering a stdlib or third party class with a stdlib or third party generic function The first two cases? Those are just part of defining class behaviour or function behaviour. That's entirely up to the library developer and an entirely sensible thing for them to be doing. That third case? It's the moral equivalent of monkey patching, and it's the strict prerogative of application integrators. The core assumption is that on import, you're just providing one component of an application, and you don't know what that application is or what it's needs are. By contrast, when you're explicitly called from main, then you can assume that this is an explicit request from the *integrated* application that wants you to modify the global state. One of the best examples of a project that gets this distinction right is gevent - you can general import gevent without any side effects on the process global state. However, the gevent.monkey module exposes monkeypatching that the *application* developer can request. You know you have a well written library if someone else could import every single one of your modules into their application and it would have *zero* effect on them until they call a function. This is often the tipping point that pushes things over from being libraries to being frameworks: the frameworks have side effects on import that mean they don't play well with others (implicitly configuring the logging system is a common example - the changes to the logging module's default behaviour in 3.2 were designed to make it easier for library developers to *stop doing that*, because it causes spurious incompatibilities. Messing too much with the import system as a side effect of import is also framework-like behaviour). Making *any* changes outside your module scope as a side effect of import can be problematic, since even if it doesn't conflict with another library, it has a tendency to break module reloading. One of the minor reasons that ABC registration, and the proposed single dispatch registration, permit silent overwriting is that being too aggressive about enforcing "once and only once" can make module reloading even more fragile than it is already. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le 23/05/2013 16:10, Łukasz Langa a écrit :
Does this work if the implementation function is called like the first decorated function? No, the ``register()`` attribute returns the undecorated function which enables decorator stacking, as well as creating unit tests for each variant independently.
Perfect. My web framework of choice uses decorators that register things and return the function as is and I love it. I guess the common pattern will be to use variants of the generic function name, e.g. func is implemented by func_int, func_str and co, which also helps debugging.
Making generic functions work with ABCs sounds like a requirement to me Yes, I will implement that.
Great!
Question: what happens if two functions (say in two different modules) are registered for the same type? Last one wins. Just like with assigning names in a scope, defining methods in a class or overriding them in a subclass.
Works for me. Cheers
participants (18)
-
Antoine Pitrou -
Armin Rigo -
Ben Hoyt -
Devin Jeanpierre -
Eric Snow -
Ethan Furman -
Glenn Linderman -
Guido van Rossum -
Nick Coghlan -
Paul Moore -
PJ Eby -
Ronan Lamy -
Sam Partington -
Steven D'Aprano -
Terry Jan Reedy -
Walter Dörwald -
Éric Araujo -
Łukasz Langa