Mailman 3 May 2013 - Python-Dev

PEP 443 - Single-dispatch generic functions
by Łukasz Langa May 24, 2013

May 24, 2013

Hello, I would like to submit the following PEP for discussion and evaluation. PEP: 443 Title: Single-dispatch generic functions Version: $Revision$ Last-Modified: $Date$ Author: Łukasz Langa <lukasz(a)langa.pl> Discussions-To: Python-Dev <python-dev(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 22-May-2013 Post-History: 22-May-2013 Replaces: 245, 246, 3124 Abstract ======== This PEP proposes a new mechanism in the ``functools`` standard library module that provides a simple form of generic programming known as single-dispatch generic functions. A **generic function** is composed of multiple functions sharing the same name. Which form should be used during a call is determined by the dispatch algorithm. When the implementation is chosen based on the type of a single argument, this is known as **single dispatch**. Rationale and Goals =================== Python has always provided a variety of built-in and standard-library generic functions, such as ``len()``, ``iter()``, ``pprint.pprint()``, ``copy.copy()``, and most of the functions in the ``operator`` module. However, it currently: 1. does not have a simple or straightforward way for developers to create new generic functions, 2. does not have a standard way for methods to be added to existing generic functions (i.e., some are added using registration functions, others require defining ``__special__`` methods, possibly by monkeypatching). In addition, it is currently a common anti-pattern for Python code to inspect the types of received arguments, in order to decide what to do with the objects. For example, code may wish to accept either an object of some type, or a sequence of objects of that type. Currently, the "obvious way" to do this is by type inspection, but this is brittle and closed to extension. Abstract Base Classes make it easier to discover present behaviour, but don't help adding new behaviour. A developer using an already-written library may be unable to change how their objects are treated by such code, especially if the objects they are using were created by a third party. Therefore, this PEP proposes a uniform API to address dynamic overloading using decorators. User API ======== To define a generic function, decorate it with the ``@singledispatch`` decorator. Note that the dispatch happens on the type of the first argument, create your function accordingly: .. code-block:: pycon >>> from functools import singledispatch >>> @singledispatch ... def fun(arg, verbose=False): ... if verbose: ... print("Let me just say,", end=" ") ... print(arg) To add overloaded implementations to the function, use the ``register()`` attribute of the generic function. It takes a type parameter: .. code-block:: pycon >>> @fun.register(int) ... def _(arg, verbose=False): ... if verbose: ... print("Strength in numbers, eh?", end=" ") ... print(arg) ... >>> @fun.register(list) ... def _(arg, verbose=False): ... if verbose: ... print("Enumerate this:") ... for i, elem in enumerate(arg): ... print(i, elem) To enable registering lambdas and pre-existing functions, the ``register()`` attribute can be used in a functional form: .. code-block:: pycon >>> def nothing(arg, verbose=False): ... print("Nothing.") ... >>> fun.register(type(None), nothing) When called, the function dispatches on the first argument: .. code-block:: pycon >>> fun("Hello, world.") Hello, world. >>> fun("test.", verbose=True) Let me just say, test. >>> fun(42, verbose=True) Strength in numbers, eh? 42 >>> fun(['spam', 'spam', 'eggs', 'spam'], verbose=True) Enumerate this: 0 spam 1 spam 2 eggs 3 spam >>> fun(None) Nothing. The proposed API is intentionally limited and opinionated, as to ensure it is easy to explain and use, as well as to maintain consistency with existing members in the ``functools`` module. Implementation Notes ==================== The functionality described in this PEP is already implemented in the ``pkgutil`` standard library module as ``simplegeneric``. Because this implementation is mature, the goal is to move it largely as-is. Several open issues remain: * the current implementation relies on ``__mro__`` alone, making it incompatible with Abstract Base Classes' ``register()``/``unregister()`` functionality. A possible solution has been proposed by PJE on the original issue for exposing ``pkgutil.simplegeneric`` as part of the ``functools`` API [#issue-5135]_. * the dispatch type is currently specified as a decorator argument. The implementation could allow a form using argument annotations. This usage pattern is out of scope for the standard library [#pep-0008]_. However, whether this registration form would be acceptable for general usage, is up to debate. Based on the current ``pkgutil.simplegeneric`` implementation and following the convention on registering virtual subclasses on Abstract Base Classes, the dispatch registry will not be thread-safe. Usage Patterns ============== This PEP proposes extending behaviour only of functions specifically marked as generic. Just as a base class method may be overridden by a subclass, so too may a function be overloaded to provide custom functionality for a given type. Universal overloading does not equal *arbitrary* overloading, in the sense that we need not expect people to randomly redefine the behavior of existing functions in unpredictable ways. To the contrary, generic function usage in actual programs tends to follow very predictable patterns and overloads are highly-discoverable in the common case. If a module is defining a new generic operation, it will usually also define any required overloads for existing types in the same place. Likewise, if a module is defining a new type, then it will usually define overloads there for any generic functions that it knows or cares about. As a result, the vast majority of overloads can be found adjacent to either the function being overloaded, or to a newly-defined type for which the overload is adding support. It is only in rather infrequent cases that one will have overloads in a module that contains neither the function nor the type(s) for which the overload is added. In the absence of incompetence or deliberate intention to be obscure, the few overloads that are not adjacent to the relevant type(s) or function(s), will generally not need to be understood or known about outside the scope where those overloads are defined. (Except in the "support modules" case, where best practice suggests naming them accordingly.) As mentioned earlier, single-dispatch generics are already prolific throughout the standard library. A clean, standard way of doing them provides a way forward to refactor those custom implementations to use a common one, opening them up for user extensibility at the same time. Alternative approaches ====================== In PEP 3124 [#pep-3124]_ Phillip J. Eby proposes a full-grown solution with overloading based on arbitrary rule sets (with the default implementation dispatching on argument types), as well as interfaces, adaptation and method combining. PEAK-Rules [#peak-rules]_ is a reference implementation of the concepts described in PJE's PEP. Such a broad approach is inherently complex, which makes reaching a consensus hard. In contrast, this PEP focuses on a single piece of functionality that is simple to reason about. It's important to note this does not preclude the use of other approaches now or in the future. In a 2005 article on Artima [#artima2005]_ Guido van Rossum presents a generic function implementation that dispatches on types of all arguments on a function. The same approach was chosen in Andrey Popp's ``generic`` package available on PyPI [#pypi-generic]_, as well as David Mertz's ``gnosis.magic.multimethods`` [#gnosis-multimethods]_. While this seems desirable at first, I agree with Fredrik Lundh's comment that "if you design APIs with pages of logic just to sort out what code a function should execute, you should probably hand over the API design to someone else". In other words, the single argument approach proposed in this PEP is not only easier to implement but also clearly communicates that dispatching on a more complex state is an anti-pattern. It also has the virtue of corresponding directly with the familiar method dispatch mechanism in object oriented programming. The only difference is whether the custom implementation is associated more closely with the data (object-oriented methods) or the algorithm (single-dispatch overloading). Acknowledgements ================ Apart from Phillip J. Eby's work on PEP 3124 [#pep-3124]_ and PEAK-Rules, influences include Paul Moore's original issue [#issue-5135]_ that proposed exposing ``pkgutil.simplegeneric`` as part of the ``functools`` API, Guido van Rossum's article on multimethods [#artima2005]_, and discussions with Raymond Hettinger on a general pprint rewrite. References ========== .. [#issue-5135] http://bugs.python.org/issue5135 .. [#pep-0008] PEP 8 states in the "Programming Recommendations" section that "the Python standard library will not use function annotations as that would result in a premature commitment to a particular annotation style". (http://www.python.org/dev/peps/pep-0008) .. [#pep-3124] http://www.python.org/dev/peps/pep-3124/ .. [#peak-rules] http://peak.telecommunity.com/DevCenter/PEAK_2dRules .. [#artima2005] http://www.artima.com/weblogs/viewpost.jsp?thread=101605 .. [#pypi-generic] http://pypi.python.org/pypi/generic .. [#gnosis-multimethods] http://gnosis.cx/publish/programming/charming_python_b12.html Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev

18 52

What if we didn't have repr?
by Guido van Rossum May 23, 2013

May 23, 2013

On Sun, May 19, 2013 at 4:27 PM, Gregory P. Smith <greg(a)krypto.org> wrote: > Now you've got me wondering what Python would be like if repr, `` and > __repr__ never existed as language features. Upon first thoughts, I actually > don't see much downside (no, i'm not advocating making that change). > Something to ponder. I have pondered it many times, although usually in the form "Why do we need both str and repr?" Unfortunately I always come back to the same issue: I really want print() of a string to write just the characters of the string (without quotes), but I also really want the >>> prompt to write the string with quotes (and escapes). Moreover, these are just two examples of the different use cases -- repr() is more useful whenever you are writing a value for a debugging purpose (e.g. when logging), and str() is more useful when writing a value as "output" of a program. One could argume that the only type for which it makes sense to distinguish between the two is str itself -- indeed I rarely define different __repr__ and __str__ functions for new classes I create (but I do note that PEP 435 does define them differently for enum members). But for the str type, it's pretty important to have str(s) equal to s, and it's also pretty important to have a way to produce a string literal from a string value. And it would be annoying if we only had str()/__str__() as a general protocol and repr() was just a string method -- imagine the number of times people would be calling s.repr() in order to have unambiguous debug output only to get a traceback because s is None or some other non-str object... So it looks like we really need both str(x) and repr(x). But maybe we only need the __repr__ protocol? str(x) could just special-case its own type, and use repr(x) (or x.__repr__(), which is the same) in all other cases. The __repr__() method on the string type would do just what it does today. But there would not be a __str__() protocol at all. That would reduce some confusion and make the language spec a tiny bit smaller, and it might stop people from proposing/requesting that str() of a list should return something different than its repr(). It also would make it a little harder for some classes (like enums) to do something nicer when printed. But IIRC there are almost no builtin types that use this feature. Personally I think that "Color.red" still looks like debug output, intended for the developer of the program, not for its user. If I wanted to show a Color to an end user of my program, I'd be printing x.name anyway. And for debugging, "Color.red" suits me fine as well -- it even fulfills the (always informal!) rule of thumb that the repr() of an object should resemble an expression that has that object as a value better than the repr() currently defined in the PEP. After all, if I really wanted to know what was inside, I could always print x.value... Please don't see this as a proposal to change the language. Like Greg, I'm not advocating, just pondering. (With the exception that if I was allowed to use the time machine to go back a couple of weeks, I'd adjust PEP 435 to define str(x) as x.name, e.g. "red", and repr(x) as what is currently defined as str(x), e.g. "Color.red".) -- --Guido van Rossum (python.org/~guido)

7 7

PEP 442 delegate
by Antoine Pitrou May 23, 2013

May 23, 2013

Hello, I would like to nominate Benjamin as BDFL-Delegate for PEP 442. Please tell me if you would like to object :) Regards Antoine.

6 6

_PyString_InsertThousandsGrouping()
by Carlos Nepomuceno May 22, 2013

May 22, 2013

Hi guys! Can someone explain to me where in the CPython 2.7.5 source code is _PyString_InsertThousandsGrouping() implemented? I've found the following declaration in 'Objects/stringobject.c' but it just defines _Py_InsertThousandsGrouping() as _PyString_InsertThousandsGrouping(): "#define _Py_InsertThousandsGrouping _PyString_InsertThousandsGrouping" I'm looking for the opposite! I don't even know how that doesn't cause an error! What's the trick? Besides that I've found a lot of code inside some header files, such as 'Objects/stringlib/formatter.h'. Why did you chose that way? Thanks in advance. Carlos

2 3

Best practices for Enum
by Raymond Hettinger May 21, 2013

May 21, 2013

After the long design effort for the enum module, I'm sure there will be a forthcoming effort to apply them pervasively throughout the standard library. I would like to ask for a little restraint and for there to be individual cost/benefit evaluations for each case. On the plus-side, the new integer-enums have a better repr than plain integers. For internal constants such as those in idlelib and regex, the user won't see any benefit at all. But there will be a cost in terms of code churn, risk of introducing errors in stable code, modestly slowing-down the code, making it more difficult to apply bug fixes across multiple versions of Python, and increased code verbosity (i.e. changing "if direction=LEFT: ..." to "if direction is Direction.LEFT: ...") For external constants, some thought needs to be given to: * is the current API working just fine (i.e. decimal's ROUND_DOWN) * will enums break doctests or any existing user code * will it complicate users converting from Python 2 * do users now have to learn an additional concept * does it complicate the module in any way I'm hoping that enums get used only in cases where they clearly improve the public API (i.e. cases such as sockets that have a large number of integer constants) rather than having a frenzy of every constant, everywhere getting turned into an enum. I would like to see enums used as tool for managing complexity, rather than becoming a cause of added complexity by being used for every problem, the tall and small, even where it is not needed at all. my-two-cents-ly yours, Raymond

23 55

Is thread-safe smtpd desired/possible?
by Sorin Stelian May 21, 2013

May 21, 2013

Hi, I am posting this here since I could find no active maintainer of the smtpd module. In my work as a test engineer for Axis (www.axis.com) I encountered the need of having thread-safe SMTP servers. I know the use case of several SMTP servers running in concurrent threads might seem odd, but it can actually be quite useful for testing purposes. I have implemented (for my own use) a possible solution which basically means that every SMTP channel has its own socket map, instead of using asyncore's global socket map. It would not involve any change in asyncore. Looking at the disucssion from http://bugs.python.org/issue11959 it seems to me that such a solution would not be accepted. Do you think that modifying asyncore is more feasible? If not, is this something that might be looked at? I can provide code if needed, but I would first like to know your thoughts about this. Best regards, Sorin

2 1

Re: [Python-Dev] [Python-checkins] cpython (3.3): #17973: Add FAQ entry for ([], )[0] += [1] both extending and raising.
by Nick Coghlan May 21, 2013

May 21, 2013

On Tue, May 21, 2013 at 12:35 AM, r.david.murray <python-checkins(a)python.org> wrote: Yay for having this in the FAQ, but... > +If you wrote:: > + > + >>> a_tuple = (1, 2) > + >>> a_tuple[0] += 1 > + Traceback (most recent call last): > + ... > + TypeError: 'tuple' object does not support item assignment > + > +The reason for the exception should be immediately clear: ``1`` is added to the > +object ``a_tuple[0]`` points to (``1``), producing the result object, ``2``, > +but when we attempt to assign the result of the computation, ``2``, to element > +``0`` of the tuple, we get an error because we can't change what an element of > +a tuple points to. > + > +Under the covers, what this augmented assignment statement is doing is > +approximately this:: > + > + >>> result = a_tuple[0].__iadd__(1) > + >>> a_tuple[0] = result > + Traceback (most recent call last): > + ... > + TypeError: 'tuple' object does not support item assignment For the immutable case, this expansion is incorrect: >>> hasattr(0, "__iadd__") False With immutable objects, += almost always expands to: >>> result = a_tuple[0] + 1 >>> a_tuple[0] = result (For containers that support binary operators, the presence of the relevant __i*__ methods is actually a reasonable heuristic for distinguishing the mutable and immutable versions) Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

PEP 435 - ref impl disc 2
by Glenn Linderman May 21, 2013

May 21, 2013

So I have a class based on Nick's Named Values, that has been extended to propagate names into expressions, so that if you have named values 'x' and 'y', when you x + y, the result is a named value whose name is '(x + y)'. Seems pretty awkward to integrate this with Enum. Maybe I'm missing something. Here's carved down code with just one operator defined for brevity. The third item from each print statement should be the same, as far as I understand... but isn't. class NamedInt( int ): _count = 0 def __new__( cls, *args, **kwds ): name, *args = args if len( args ) == 0: args = [ cls._count ] cls._count += 1 self = super().__new__( cls, *args, **kwds ) self._name = name return self def __init__( self, *args, **kwds ): name, *args = args super().__init__() @property def __name__( self ): return self._name def __repr__( self ): # repr() is updated to include the name and type info return "{}({!r}, {})".format(type(self).__name__, self.__name__, super().__repr__()) def __str__( self ): # str() is unchanged, even if it relies on the repr() fallback base = super() base_str = base.__str__ if base_str.__objclass__ is object: return base.__repr__() return base_str() # for simplicity, we only define one operator that propagates expressions def __add__(self, other): temp = int( self ) + int( other ) if isinstance( self, NamedInt ) and isinstance( other, NamedInt ): return NamedInt( '({0} + {1})'.format(self.__name__, other.__name__), temp ) else: return temp x = NamedInt('the-x', 1 ) y = NamedInt('the-y', 2 ) # demonstrate that NamedInt propagates the names into an expression syntax print( repr( x ), repr( y ), repr( x+y )) from ref435 import Enum # requires redundant names, but loses names in the expression class NEI( NamedInt, Enum ): x = NamedInt('the-x', 1 ) y = NamedInt('the-y', 2 ) print( repr( NEI( 1 )), repr( NEI( 2 )), repr( NEI(1) + NEI(2))) # looks redundant, and still loses the names in the expression class NEI2( NamedInt, Enum ): x = x y = y print( repr( NEI2( x )), repr( NEI2( x )), repr( NEI2(x) + NEI2(y)))

4 31

Re: [Python-Dev] Purpose of Doctests [Was: Best practices for Enum]
by Olemis Lang May 20, 2013

May 20, 2013

---------- Forwarded message ---------- From: Olemis Lang <olemis(a)gmail.com> Date: Mon, 20 May 2013 11:33:42 -0500 Subject: Re: [Python-Dev] Purpose of Doctests [Was: Best practices for Enum] To: Antoine Pitrou <solipsis(a)pitrou.net> On 5/20/13, Antoine Pitrou <solipsis(a)pitrou.net> wrote: > On Sat, 18 May 2013 23:41:59 -0700 > Raymond Hettinger <raymond.hettinger(a)gmail.com> wrote: >> >> We should continue to encourage users to make thorough unit tests >> and to leave doctests for documentation. That said, it should be >> recognized that some testing is better than no testing. And doctests >> may be attractive in that regard because it is almost effortless to >> cut-and-paste a snippet from the interactive prompt. That isn't a >> best practice, but it isn't a worst practice either. > > There are other reasons to hate doctest, such as the obnoxious > error reporting. Having to wade through ten pages of output to find > what went wrong is no fun. > +1 FWIW , while using dutest [1]_ each interactive example will be a test case and therefore the match for that particular assertion will be reported using the usual unittest output format .. [1] dutest (https://pypi.python.org/pypi/dutest) -- Regards, Olemis. Apache™ Bloodhound contributor http://issues.apache.org/bloodhound Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article:

1 0

Why is documentation not inline?
by Demian Brecht May 20, 2013

May 20, 2013

This is more out of curiosity than to spark change (although I wouldn't argue against it): Does anyone know why it was decided to document external to source files rather than inline? When rapidly digging through source, it would be much more helpful to see parameter docs than to either have to find source lines (that can easily be missed) to figure out the intention. Case in point, I've been digging through cookiejar.py and request.py to figure out their interactions. When reading through build_opener, it took me a few minutes to figure out that each element of *handlers can be either an instance /or/ a class definition (I was looking at how to define a custom cookiejar for an HTTPCookieProcessor). Yes, I'm (now) aware that there's some documentation at the top of request.py, but it would have been helpful to have it right in the definition of build_opener. It seems like external docs is standard throughout the stdlib. Is there an actual reason for this? Thanks, -- Demian Brecht http://demianbrecht.github.com

7 9